Logstash Extension Base Template

overview the logstash extension base template (logstash extension base) provides a foundation for developing logstash based extensions integration developers can use this template to create a repository and implement custom data processing pipelines several pipeline templates are available for common kyndryl use cases and can be used as a starting point for extension development architecture overview the extension performs the following operations during startup and runtime starts the entrypoint sh script extracts the keystore and truststore from the configured vault path verifies the elasticsearch certificate loads the logstash configuration files log4j2 properties pipelines yml logstash yml loads the configured pipeline definitions processes incoming data according to the selected pipeline sends output to one or more destinations console (stdout) dynatrace elasticsearch dss kafka topics configuration files file description config yaml defines extension metadata, deployment probes, and deployment time configuration variables bin/entrypoint sh prepares runtime variables and starts logstash log4j2 properties configures logging output pipelines yml defines the pipelines to run logstash yml contains logstash runtime settings configuration variables deployment configuration variables the deployment env vars section in config yaml defines the values that users configure during deployment variable purpose default value dx ext consumer group kafka consumer group id required dx ext cacf topic cacf input topic san port insights cacf kafka brokers cacf broker list environment specific dx ext kafka keystore kafka keystore vault path vault path dx ext kafka truststore kafka truststore vault path vault path dx ext keystore pwd keystore password required dx ext elastic indexname target elasticsearch index required dx ext log level extension logging level info environment variables standard environment variables the following environment variables are available to all logstash extensions through extension services variable example description dx ext internal port 5000 internal communication port dx ext consumer group kafka consumer group identifier dx ext cacf topic san port insights cacf kafka topic cacf kafka brokers environment specific kafka brokers dx ext kafka keystore vault path kafka keystore location dx ext kafka truststore vault path kafka truststore location dx ext keystore pwd vault stored password dx ext elastic indexname elasticsearch index alias dx ext kafka topic dss kafka topic dx ext log level info log level dx ext host aliases host and ip mappings dx ext settings off advanced settings toggle advanced environment variables configure these variables when polling data from cacf or for advanced performance tuning variable example description dx ext java options xmx4g xms2g override java memory settings dx ext batch size 125 pipeline batch size dx ext batch delay 50 pipeline batch delay dx ext max poll records 1000 maximum kafka records per poll number of kafka partitions 10 number of processing workers dx ext poll timeout 20000 consumer poll timeout (ms) dx ext max poll interval 400000 maximum delay between polls (ms) dx ext request timeout 700000 kafka request timeout (ms) dx ext heartbeat 200000 consumer heartbeat interval (ms) dx ext session timeout 900000 kafka session timeout (ms) dx ext max bytes 1048576 maximum payload size dx ext verify elastic full elasticsearch ssl verification mode health probes the container implements the following kubernetes health checks liveness probe verifies that the container is running readiness probe verifies that the container is ready to receive traffic build and run the extension build the image navigate to the repository directory update the env file with the required settings build the docker image docker build t is logstash start the elk stack (optional) docker compose up d configure local environment variables configure the required variables before running the container locally target=logstash base elasticsearch 1 9200 index=ext services test uname=elastic pw=test image=is logstash container name=is logstash logstash listen port=5000 network=logstash base default run the container docker run d \\ \ name "${container name}" \\ \ hostname "${container name}" \\ \ restart always \\ \ network=${network} \\ p "${logstash listen port} 5000" \\ e dx ext internal port="${logstash listen port}" \\ e dx ext destination user="${uname}" \\ e dx ext destination pw="${pw}" \\ e dx ext es url="${target}" \\ e dx ext es index="${index}" \\ ${image} use docker compose all environment variables can be defined in either \ env \ docker compose yml configure docker compose yml according to the chosen method testing and validation view container logs docker logs f is logstash access the container docker exec it is logstash /bin/bash send test data send a post request to the configured http listener example for endpoint http //127 0 0 1 5000 required header content type application/json example for payload { "test" "test", "firstfield" "extensions services", "secondfield" "is awesome", "hostname" "api na 1 kyndryl com" } podman deployment build and run using podman perform the following steps navigate to the repository directory cd /path/to/repository start the podman machine podman machine start build the image podman compose build start the container podman compose up pipeline reference all pipeline definitions are stored in the /pipelines directory as conf files available pipeline templates pipeline input processing output http elastic conf http listener field transformation and normalization elasticsearch cacf kafka dss kafka conf cacf kafka message decoding and filtering dss kafka cacf kafka elastic conf cacf kafka validation and routing elasticsearch kafka elastic http poller conf http poller message parsing elasticsearch elastic elastic conf elasticsearch data transfer elasticsearch db2 elastic conf db2 jdbc row ingestion elasticsearch common pipeline features most pipeline configurations include standard console logging dynatrace integration for failure events elasticsearch output plugins environment based credential injection logging console output all pipelines use the standard logstash stdout plugin stdout { codec => line { format => "info ou=${dx env ou key} processing data from %{filename}" } } dynatrace integration install the dynatrace plugin in the docker image run /usr/share/logstash/bin/logstash plugin install logstash output dynatrace the following environment variables are injected automatically dx env dynatrace api url dx env dynatrace api key log4j2 configuration the console format uses the following pattern appender console layout pattern = % 5p %d{iso8601} ou=${env\ dx env ou key} % 25c ${env\ dx env cat item version} %m %p %m%n known issues and limitations this template supports only logstash based integrations other endpoint integrations must be implemented as separate extensions logstash workloads can consume significant resources consider increasing the deployment size to size large or size extra large when processing high data volumes some pipeline files contain hyphenated environment variable names use underscore ( ) naming conventions where possible the sample db2 pipeline includes placeholder connection values and must be updated before production use prerequisites before deploying a logstash extension ensure that the target organizational unit (ou) is onboarded to integrated verify access to the required kafka, elasticsearch, and vault resources review the integrated onboarding documentation for onboarding requirements and configuration details