---
title: Logstash Extension Base Template
slug: logstash-extension-base-template
docTags: 
createdAt: 2026-09-09T11:34:59.574Z
---

# Overview

The Logstash Extension Base Template (logstash-extension-base) provides a foundation for developing Logstash-based extensions. Integration developers can use this template to create a repository and implement custom data processing pipelines.

:::hint{type="info"}
Several pipeline templates are available for common Kyndryl use cases and can be used as a starting point for extension development. 
:::

# Architecture Overview

The extension performs the following operations during startup and runtime:

- Starts the entrypoint.sh script.
- Extracts the keystore and truststore from the configured vault path.
- Verifies the Elasticsearch certificate.

Loads the Logstash configuration files:

- log4j2.properties
- pipelines.yml
- logstash.yml
- Loads the configured pipeline definitions.
- Processes incoming data according to the selected pipeline.
- Sends output to one or more destinations:
  - Console (stdout)
  - Dynatrace
  - Elasticsearch           
  - DSS Kafka topics

# Configuration files

| **File**          | **Description**                                                                             |
| ----------------- | ------------------------------------------------------------------------------------------- |
| config.yaml       | Defines extension metadata, deployment probes, and deployment-time configuration variables. |
| bin/entrypoint.sh | Prepares runtime variables and starts Logstash.                                             |
| log4j2.properties | Configures logging output.                                                                  |
| pipelines.yml     | Defines the pipelines to run.                                                               |
| logstash.yml      | Contains Logstash runtime settings.                                                         |

# Configuration Variables

## Deployment configuration variables

The deployment.env-vars section in config.yaml defines the values that users configure during deployment.

| **Variable**                | **Purpose**                 | **Default Value**    |
| --------------------------- | --------------------------- | -------------------- |
| DX\_EXT\_CONSUMER\_GROUP    | Kafka consumer group ID     | Required             |
| DX\_EXT\_CACF\_TOPIC        | CACF input topic            | san-port-insights    |
| CACF\_KAFKA\_BROKERS        | CACF broker list            | Environment-specific |
| DX\_EXT\_KAFKA\_KEYSTORE    | Kafka keystore vault path   | Vault path           |
| DX\_EXT\_KAFKA\_TRUSTSTORE  | Kafka truststore vault path | Vault path           |
| DX\_EXT\_KEYSTORE\_PWD      | Keystore password           | Required             |
| DX\_EXT\_ELASTIC\_INDEXNAME | Target Elasticsearch index  | Required             |
| DX\_EXT\_LOG\_LEVEL         | Extension logging level     | INFO                 |

## Environment Variables

### Standard Environment Variables

The following environment variables are available to all Logstash extensions through extension services.

| **Variable**                | **Example**       | **Description**                     |
| --------------------------- | ----------------- | ----------------------------------- |
| DX\_EXT\_INTERNAL\_PORT     | 5000              | Internal communication port.        |
| DX\_EXT\_CONSUMER\_GROUP    | -                 | Kafka consumer group identifier.    |
| DX\_EXT\_CACF\_TOPIC        | san-port-insights | CACF Kafka topic.                   |
| CACF\_KAFKA\_BROKERS        | -                 | Environment-specific Kafka brokers. |
| DX\_EXT\_KAFKA\_KEYSTORE    | Vault path        | Kafka keystore location.            |
| DX\_EXT\_KAFKA\_TRUSTSTORE  | Vault path        | Kafka truststore location.          |
| DX\_EXT\_KEYSTORE\_PWD      | -                 | Vault-stored password.              |
| DX\_EXT\_ELASTIC\_INDEXNAME | -                 | Elasticsearch index alias.          |
| DX\_EXT\_KAFKA\_TOPIC       | -                 | DSS Kafka topic.                    |
| DX\_EXT\_LOG\_LEVEL         | INFO              | Log level.                          |
| DX\_EXT\_HOST\_ALIASES      | -                 | Host and IP mappings.               |
| DX\_EXT\_SETTINGS           | OFF               | Advanced settings toggle.           |

### Advanced Environment Variables

Configure these variables when polling data from CACF or for advanced performance tuning.

| **Variable**                 | **Example**   | **Description**                      |
| ---------------------------- | ------------- | ------------------------------------ |
| DX\_EXT\_JAVA\_OPTIONS       | -Xmx4g -Xms2g | Override Java memory settings.       |
| DX\_EXT\_BATCH\_SIZE         | 125           | Pipeline batch size.                 |
| DX\_EXT\_BATCH\_DELAY        | 50            | Pipeline batch delay.                |
| DX\_EXT\_MAX\_POLL\_RECORDS  | 1000          | Maximum Kafka records per poll.      |
| Number of Kafka Partitions   | 10            | Number of processing workers.        |
| DX\_EXT\_POLL\_TIMEOUT       | 20000         | Consumer poll timeout (ms).          |
| DX\_EXT\_MAX\_POLL\_INTERVAL | 400000        | Maximum delay between polls (ms).    |
| DX\_EXT\_REQUEST\_TIMEOUT    | 700000        | Kafka request timeout (ms).          |
| DX\_EXT\_HEARTBEAT           | 200000        | Consumer heartbeat interval (ms).    |
| DX\_EXT\_SESSION\_TIMEOUT    | 900000        | Kafka session timeout (ms).          |
| DX\_EXT\_MAX\_BYTES          | 1048576       | Maximum payload size.                |
| DX\_EXT\_VERIFY\_ELASTIC     | full          | Elasticsearch SSL verification mode. |

# Health Probes

The container implements the following Kubernetes health checks:

- **Liveness probe:** Verifies that the container is running.
- **Readiness probe:** Verifies that the container is ready to receive traffic.

# Build and Run the Extension

**Build the Image**

1. Navigate to the repository directory.
2. Update the .env file with the required settings.
3. Build the Docker image.

:::Paragraph{indent="1"}
`docker build -t is-logstash.`
:::

**Start the ELK Stack (Optional)**

:::BlockQuote
docker compose up -d
:::

**Configure local environment variables**

Configure the required variables before running the container locally.

`TARGET=logstash-base-elasticsearch-1:9200`

`INDEX=ext-services-test`

`UNAME=elastic`

`PW=test`

`IMAGE=is-logstash`

`CONTAINER_NAME=is-logstash`

`LOGSTASH_LISTEN_PORT=5000`

`NETWORK=logstash-base_default`

**Run the container**

`docker run -d \`

`--name "${CONTAINER_NAME}" \`

`--hostname "${CONTAINER_NAME}" \`

`--restart always \`

`--network=${NETWORK} \`

`-p "${LOGSTASH_LISTEN_PORT}:5000" \`

`-e DX_EXT_INTERNAL_PORT="${LOGSTASH_LISTEN_PORT}" \`

`-e DX_EXT_DESTINATION_USER="${UNAME}" \`

`-e DX_EXT_DESTINATION_PW="${PW}" \`

`-e DX_EXT_ES_URL="${TARGET}" \`

`-e DX_EXT_ES_INDEX="${INDEX}" \`

`${IMAGE}`

**Use docker compose**

All environment variables can be defined in either:

`- .env`

`- docker-compose.yml`

:::hint{type="info"}
Configure docker-compose.yml according to the chosen method.
:::

**Testing and Validation**

View Container Logs

`docker logs -f is-logstash`

**Access the container**

`docker exec -it is-logstash /bin/bash`

**Send test data**

Send a POST request to the configured HTTP listener.

Example for Endpoint: *http\://127.0.0.1:5000*

Required header

`Content-Type: application/json`

Example for Payload:

`{`

`"test": "test",`

`"firstfield": "Extensions services",`

`"secondfield": "is awesome",`

`"hostname": "api-na-1.kyndryl.com"`

`}`

# Podman Deployment

## Build and run using Podman

Perform the following steps:

1. Navigate to the repository directory.

:::Paragraph{indent="1"}
`cd /path/to/repository`
:::

2. Start the Podman machine.

:::Paragraph{indent="1"}
`podman machine start`
:::

3. Build the image.

:::Paragraph{indent="1"}
`podman compose build`
:::

4. Start the container.

:::Paragraph{indent="1"}
`podman compose up`
:::

## Pipeline reference

All pipeline definitions are stored in the ./pipelines directory as .conf files.

Developers are responsible for creating and maintaining the .conf file that defines their pipeline configuration. The pipeline templates listed in this section are provided as reference examples to help understand common pipeline patterns and configurations.

For information about Logstash pipeline architecture and pipeline development, see:

- [How Logstash works](https://www.elastic.co/docs/reference/logstash/how-logstash-works)
- [Creating a Logstash pipeline](https://www.elastic.co/docs/reference/logstash/creating-logstash-pipeline)

### Available Pipeline Templates

| **Pipeline**                    | **Input**     | **Processing**                         | **Output**    |
| ------------------------------- | ------------- | -------------------------------------- | ------------- |
| http-elastic.conf               | HTTP listener | Field transformation and normalization | Elasticsearch |
| cacf\_kafka-dss\_kafka.conf     | CACF Kafka    | Message decoding and filtering         | DSS Kafka     |
| cacf\_kafka-elastic.conf        | CACF Kafka    | Validation and routing                 | Elasticsearch |
| kafka-elastic-http\_poller.conf | HTTP Poller   | Message parsing                        | Elasticsearch |
| elastic-elastic.conf            | Elasticsearch | Data transfer                          | Elasticsearch |
| db2-elastic.conf                | DB2 JDBC      | Row ingestion                          | Elasticsearch |

### Common Pipeline Features

Most pipeline configurations include:

- Standard console logging
- Dynatrace integration for failure events
- Elasticsearch output plugins
- Environment-based credential injection

# Logging

## Console output

All pipelines use the standard Logstash stdout plugin.

:::BlockQuote
stdout \{
codec => line \{
format => "INFO ou=$\{{DX\_ENV\_OU\_KEY} \:: processing data from %{filename}"
}
}
:::

## Dynatrace integration

Install the Dynatrace plugin in the Docker image:

- `RUN /usr/share/logstash/bin/logstash-plugin install logstash-output-dynatrace`

The following environment variables are injected automatically:

- *`DX_ENV_DYNATRACE_API_URL`*
- *`DX_ENV_DYNATRACE_API_KEY`*

## log4j2 Configuration

The console format uses the following pattern:

- `appender.console.layout.pattern = %-5p %d{ISO8601} ou=${env:DX_ENV_OU_KEY} %-25c ${env:DX_ENV_CAT_ITEM_VERSION} %M %p %m%n`

# Known issues and limitations

- This template supports only Logstash-based integrations.
- Other endpoint integrations must be implemented as separate extensions.
- Logstash workloads can consume significant resources.
- Consider increasing the deployment size to size-large or size-extra-large when processing high data volumes.
- Some pipeline files contain hyphenated environment variable names. Use underscore (\_) naming conventions where possible.
- The sample DB2 pipeline includes placeholder connection values and must be updated before production use.

# Prerequisites

Before deploying a Logstash extension:

- Ensure that the target Organizational Unit (OU) is onboarded to Integrated Artificial Intelligence for IT Operations.
- Verify access to the required Kafka, Elasticsearch, and vault resources.
- Review the Integrated Artificial Intelligence for IT Operations onboarding documentation for onboarding requirements and configuration details.