Logstash Extension Base Template

Overview

The Logstash Extension Base Template (logstash-extension-base) provides a foundation for developing Logstash-based extensions. Integration developers can use this template to create a repository and implement custom data processing pipelines.

Several pipeline templates are available for common Kyndryl use cases and can be used as a starting point for extension development.

Architecture Overview

The extension performs the following operations during startup and runtime:

Starts the entrypoint.sh script.

Extracts the keystore and truststore from the configured vault path.

Verifies the Elasticsearch certificate.

Loads the Logstash configuration files:

log4j2.properties

pipelines.yml

logstash.yml

Loads the configured pipeline definitions.

Processes incoming data according to the selected pipeline.

Sends output to one or more destinations:

Console (stdout)

Dynatrace

Elasticsearch

DSS Kafka topics

Configuration files

File	Description
config.yaml	Defines extension metadata, deployment probes, and deployment-time configuration variables.
bin/entrypoint.sh	Prepares runtime variables and starts Logstash.
log4j2.properties	Configures logging output.
pipelines.yml	Defines the pipelines to run.
logstash.yml	Contains Logstash runtime settings.

Configuration Variables

Deployment configuration variables

The deployment.env-vars section in config.yaml defines the values that users configure during deployment.

Variable	Purpose	Default Value
DX_EXT_CONSUMER_GROUP	Kafka consumer group ID	Required
DX_EXT_CACF_TOPIC	CACF input topic	san-port-insights
CACF_KAFKA_BROKERS	CACF broker list	Environment-specific
DX_EXT_KAFKA_KEYSTORE	Kafka keystore vault path	Vault path
DX_EXT_KAFKA_TRUSTSTORE	Kafka truststore vault path	Vault path
DX_EXT_KEYSTORE_PWD	Keystore password	Required
DX_EXT_ELASTIC_INDEXNAME	Target Elasticsearch index	Required
DX_EXT_LOG_LEVEL	Extension logging level	INFO

Environment Variables

Standard Environment Variables

The following environment variables are available to all Logstash extensions through extension services.

Variable	Example	Description
DX_EXT_INTERNAL_PORT	5000	Internal communication port.
DX_EXT_CONSUMER_GROUP	-	Kafka consumer group identifier.
DX_EXT_CACF_TOPIC	san-port-insights	CACF Kafka topic.
CACF_KAFKA_BROKERS	-	Environment-specific Kafka brokers.
DX_EXT_KAFKA_KEYSTORE	Vault path	Kafka keystore location.
DX_EXT_KAFKA_TRUSTSTORE	Vault path	Kafka truststore location.
DX_EXT_KEYSTORE_PWD	-	Vault-stored password.
DX_EXT_ELASTIC_INDEXNAME	-	Elasticsearch index alias.
DX_EXT_KAFKA_TOPIC	-	DSS Kafka topic.
DX_EXT_LOG_LEVEL	INFO	Log level.
DX_EXT_HOST_ALIASES	-	Host and IP mappings.
DX_EXT_SETTINGS	OFF	Advanced settings toggle.

Advanced Environment Variables

Configure these variables when polling data from CACF or for advanced performance tuning.

Variable	Example	Description
DX_EXT_JAVA_OPTIONS	-Xmx4g -Xms2g	Override Java memory settings.
DX_EXT_BATCH_SIZE	125	Pipeline batch size.
DX_EXT_BATCH_DELAY	50	Pipeline batch delay.
DX_EXT_MAX_POLL_RECORDS	1000	Maximum Kafka records per poll.
Number of Kafka Partitions	10	Number of processing workers.
DX_EXT_POLL_TIMEOUT	20000	Consumer poll timeout (ms).
DX_EXT_MAX_POLL_INTERVAL	400000	Maximum delay between polls (ms).
DX_EXT_REQUEST_TIMEOUT	700000	Kafka request timeout (ms).
DX_EXT_HEARTBEAT	200000	Consumer heartbeat interval (ms).
DX_EXT_SESSION_TIMEOUT	900000	Kafka session timeout (ms).
DX_EXT_MAX_BYTES	1048576	Maximum payload size.
DX_EXT_VERIFY_ELASTIC	full	Elasticsearch SSL verification mode.

Health Probes

The container implements the following Kubernetes health checks:

Liveness probe: Verifies that the container is running.

Readiness probe: Verifies that the container is ready to receive traffic.

Build and Run the Extension

Build the Image

Navigate to the repository directory.

Update the .env file with the required settings.

Build the Docker image.

docker build -t is-logstash.

Start the ELK Stack (Optional)

docker compose up -d

Configure local environment variables

Configure the required variables before running the container locally.

TARGET=logstash-base-elasticsearch-1:9200

INDEX=ext-services-test

UNAME=elastic

PW=test

IMAGE=is-logstash

CONTAINER_NAME=is-logstash

LOGSTASH_LISTEN_PORT=5000

NETWORK=logstash-base_default

Run the container

docker run -d \

--name "${CONTAINER_NAME}" \

--hostname "${CONTAINER_NAME}" \

--restart always \

--network=${NETWORK} \

-p "${LOGSTASH_LISTEN_PORT}:5000" \

-e DX_EXT_INTERNAL_PORT="${LOGSTASH_LISTEN_PORT}" \

-e DX_EXT_DESTINATION_USER="${UNAME}" \

-e DX_EXT_DESTINATION_PW="${PW}" \

-e DX_EXT_ES_URL="${TARGET}" \

-e DX_EXT_ES_INDEX="${INDEX}" \

${IMAGE}

Use docker compose

All environment variables can be defined in either:

- .env

- docker-compose.yml

Configure docker-compose.yml according to the chosen method.

Testing and Validation

View Container Logs

docker logs -f is-logstash

Access the container

docker exec -it is-logstash /bin/bash

Send test data

Send a POST request to the configured HTTP listener.

Example for Endpoint: http://127.0.0.1:5000

Required header

Content-Type: application/json

Example for Payload:

{

"test": "test",

"firstfield": "Extensions services",

"secondfield": "is awesome",

"hostname": "api-na-1.kyndryl.com"

}

Podman Deployment

Build and run using Podman

Perform the following steps:

Navigate to the repository directory.

cd /path/to/repository

Start the Podman machine.

podman machine start

Build the image.

podman compose build

Start the container.

podman compose up

Pipeline reference

All pipeline definitions are stored in the ./pipelines directory as .conf files.

Developers are responsible for creating and maintaining the .conf file that defines their pipeline configuration. The pipeline templates listed in this section are provided as reference examples to help understand common pipeline patterns and configurations.

For information about Logstash pipeline architecture and pipeline development, see:

How Logstash works

Creating a Logstash pipeline

Available Pipeline Templates

Pipeline	Input	Processing	Output
http-elastic.conf	HTTP listener	Field transformation and normalization	Elasticsearch
cacf_kafka-dss_kafka.conf	CACF Kafka	Message decoding and filtering	DSS Kafka
cacf_kafka-elastic.conf	CACF Kafka	Validation and routing	Elasticsearch
kafka-elastic-http_poller.conf	HTTP Poller	Message parsing	Elasticsearch
elastic-elastic.conf	Elasticsearch	Data transfer	Elasticsearch
db2-elastic.conf	DB2 JDBC	Row ingestion	Elasticsearch

Common Pipeline Features

Most pipeline configurations include:

Standard console logging

Dynatrace integration for failure events

Elasticsearch output plugins

Environment-based credential injection

Logging

Console output

All pipelines use the standard Logstash stdout plugin.

stdout { codec => line { format => "INFO ou=${{DX_ENV_OU_KEY} :: processing data from %{filename}" } }

Dynatrace integration

Install the Dynatrace plugin in the Docker image:

RUN /usr/share/logstash/bin/logstash-plugin install logstash-output-dynatrace

The following environment variables are injected automatically:

DX_ENV_DYNATRACE_API_URL

DX_ENV_DYNATRACE_API_KEY

log4j2 Configuration

The console format uses the following pattern:

appender.console.layout.pattern = %-5p %d{ISO8601} ou=${env:DX_ENV_OU_KEY} %-25c ${env:DX_ENV_CAT_ITEM_VERSION} %M %p %m%n

Known issues and limitations

This template supports only Logstash-based integrations.

Other endpoint integrations must be implemented as separate extensions.

Logstash workloads can consume significant resources.

Consider increasing the deployment size to size-large or size-extra-large when processing high data volumes.

Some pipeline files contain hyphenated environment variable names. Use underscore (_) naming conventions where possible.

The sample DB2 pipeline includes placeholder connection values and must be updated before production use.

Prerequisites

Before deploying a Logstash extension:

Ensure that the target Organizational Unit (OU) is onboarded to Integrated Artificial Intelligence for IT Operations.

Verify access to the required Kafka, Elasticsearch, and vault resources.

Review the Integrated Artificial Intelligence for IT Operations onboarding documentation for onboarding requirements and configuration details.