Services

Explore Kyndryl Bridge Services

Data Streaming Service and API configuration for IAM
Published On May 15, 2026 - 12:21 AM

Data Streaming Service and API configuration for IAM

Learn about Data Streaming Service (DSS), its core features, streaming architecture, security controls, and integration with Kyndryl Bridge.
The Data Streaming Service (DSS) enables secure, real-time data sharing across services within the Kyndryl Bridge platform. Acting as a central integration layer, DSS allows multiple services to access a unified data stream without requiring redundant connections or individual authentication mechanisms. Built on a Kafka backbone with a custom API layer, DSS supports both REST APIs and Server-Sent Events (SSE) for efficient, scalable message exchange. By standardizing how data flows between producers and consumers, DSS streamlines integration efforts, improves operational agility, and ensures consistent communication across the enterprise ecosystem.

Key functionalities

  1. Unified data access:
     Enables multiple services to consume shared data streams without separate authentication or reconfigurations.
  2. Publish once, consume many:
     A single data source can be accessed by multiple services, eliminating the need for additional connectivity setups.
  3. Data normalization:
     Standardized schemas avoid duplication and ensure consistency.
  4. Message transformation:
     Supports redaction, enrichment, and obfuscation to safeguard sensitive data.
  5. Versatile consumption options:
     Provides both Server-Sent Events (SSE) and REST API for flexible data interactions.
  6. Pre-built consumer and producer extensions:
     Templates for producer and consumer extensions are available to accelerate development.

DSS roles and access levels

  1. Solution designer:
     Designs and configures solutions using DSS. Responsible for registering topics, configuring schemas, managing security protocols, and ensuring successful integration with DSS.

Use cases

The Data Streaming Service (DSS) provides a secure, scalable, and efficient framework for streaming data across the Kyndryl Bridge platform. It enables services to consume data from a shared source without duplicating pipelines or reauthenticating connections, simplifying integration and improving operational agility.
  1. Unified data integration:
     Multiple services subscribe to a common data stream, eliminating the need for direct integrations with each data provider.
  2. Scalable access to high-volume data:
     DSS supports high-throughput data ingestion, processing, and analysis without bottlenecks, making it suitable for large-scale enterprise workloads.
  3. Data aggregation:
     Multiple producers publish to a single topic, enabling downstream consumers to aggregate and correlate data from various sources.
  4. Data transformation:
     Consumers can filter, enrich, or reformat messages before publishing them to a secondary topic for further processing or consumption.
  5. Integration gateway (iPaaS pattern):
     DSS acts as a centralized gateway that connects diverse systems and services, providing a consistent, unified interface for publishing and consuming data across the Kyndryl Bridge ecosystem.

Prerequisites

To integrate with the Data Streaming Service, the following prerequisites are required:
  1. Access to the Kyndryl Bridge platform:
     Partners must be registered with the Kyndryl Bridge platform.
  2. Vault agent:
     Installed as a sidecar to manage token-based authentication.
  3. Schema and topic registration:
     All new topics or schemas must be registered via the Topic Registry and Schema Registry before data streaming or consumption can begin.
  4. Pre-configured producer and consumer setup:
     Data producers and consumers must be appropriately configured to interact with specific topics, adhering to the required message format and security protocols.

How to configure DSS

Follow these steps to configure and integrate with the Data Streaming Service (DSS):
  1. Set up producer or consumer extensions:
     Work with Kyndryl to co-develop a producer or consumer extension tailored to your integration needs. Use the provided templates to configure the extension for interacting with the required topics and schemas. These extensions enable secure and standardized communication through DSS.
  2. Register topics and schemas:
    • Create a new branch in the Topic Registry and register the topic using the standard naming convention (
      context-domain-subject
      ).
    • Define the topic attributes—such as message class, type, partitions, and retention policy—using the provided topic template.
    • If needed, register a new schema in the Schema Registry following the established schema governance process.
  3. Configure topic attributes:
     Use the topic configuration file to specify topic-level settings such as partition count, cleanup policy, and retention period.
  4. Enable Vault-based authentication:
     The Vault Agent is automatically injected by the platform operator. The authentication token for DSS is mounted at
    /vault/secrets/token
     within the container. This token must be used when making API calls to DSS.
  5. Connect to the DSS API:
     Use the Vault token to authenticate and perform API operations. Producers send messages and consumers retrieve messages via the DSS REST API or Server-Sent Events (SSE) endpoints.
  6. Configure message transformation:
     If your use case involves handling sensitive or regulated data, enable message redaction, enrichment, or obfuscation as needed. These transformations are applied at the message level before consumption.

Working with DSS tokens, messaging, and roles

Obtain a token:
 Tokens are inserted into containers via Vault annotations. The token is available at
/vault/secrets/token
 and is used to authenticate API requests.
Setting up topics:
 Topics in DSS help organize streams of data. Specific configurations like partition count, retention policy, and cleanup mechanisms can be defined for topics:
  • Partitions:
     Sets the number of partitions for the topic. More partitions allow for parallelism.
  • RetentionMs:
     Specifies the message retention duration in milliseconds.
  • CleanupPolicy:
     Determines the cleanup mechanism, such as deleting old messages after the retention period.
Sending and receiving messages:
 To send messages, make a POST request to the DSS API with the target topic.
Query parameters and headers:
  • sync:
     Set to true to wait for the message delivery confirmation.
  • x-account:
     Specifies the account for encryption. Only consumers with account certificates can decrypt the message.
  • x-key:
     Enforces message ordering based on the provided key.
DSS offers two methods to receive messages:
  • Single message fetch
  • Server-Sent Events (SSE)
     for continuous streaming
Security and role-based access control (RBAC):
 DSS uses RBAC to control access to various resources. Permissions are associated with roles defined during the service onboarding process:
  • Producer role:
     Allows sending messages to topics.
  • Consumer role:
     Allows reading messages from topics.
  • Admin role:
     Grants access to manage topics, consumer groups, and configurations.
To enforce permissions:
  • Configure
    Vault annotations
     during service deployment.
  • Apply the necessary roles based on the required access type (producer, consumer, admin).
Monitoring and metrics:
 DSS provides metrics APIs to monitor the health and performance of data streams:
  • Metrics API endpoint:
     http://streams/metrics
  • Monitor
    message lag
    ,
    throughput
    , and
    consumer health
     through these metrics.
  • Set up alerts based on the metrics to ensure optimal system performance.

Message format and communication

Common Message Format (CMF):
 All messages follow a standardized structure that includes HTTP headers and a payload.
API interaction:
 Services communicate with DSS through HTTP requests using an internal URL, authenticated with a Vault-injected Bearer Token.
Example internal URL format:
http://streams/{topic}
Example message structure:

{
"headers": {},
"payload": "can be text, an embedded JSON structure, or some other encoded format",
"restricted_payload": ""
}
Example with populated headers:

{
"headers": {
  "message_class": "regular",
  "message_type": "gts.synapps.unique_type_name",
  "account": {
    "id": "ou_short_id"
  },
  "object_id": "v4_uuid",
  "transaction_id": "v4_uuid",
  "message_format": "some/mimetype",
  "encryption_key": "<base64_encoded_encrypted_string_if_payload_is_encrypted>",
  "key_version": 1,
  "restricted_data_encryption_key": "<base64_encoded_encrypted_string_if_payload_is_encrypted>",
  "restricted_key_version": 1,
  "sender_service": "",
  "action": {},
  "stream": {}
}
}

API definitions and endpoints

Messaging 101:
 Follow these steps to interact with DSS APIs:
  1. Create a topic (optional):
     Register the topic in the Topic Registry and configure its schema and attributes.
  2. Obtain an authentication token:
     Retrieve the Vault-injected token from 
    /vault/secrets/token
    . This token is required to authenticate API requests.
  3. Send and receive messages:
     Use the DSS REST API or Server-Sent Events (SSE) to publish and consume messages on registered topics.
Addressing Data Streaming Service APIs:
 DSS provides different API endpoints depending on environment and use case. Select the appropriate base URL for internal or external access.
API category
URL (within cluster)
URL (external)
Messaging
Messaging (with Consumer Group)
Admin
Metrics
Authentication:
 Use Bearer token authentication for all API requests. Tokens are injected into your container at 
/vault/secrets/token
.
For full API documentation, refer to: API keys for Bridge services and applications

Sending and receiving messages

DSS supports REST-based and Server-Sent Events (SSE) interfaces for publishing and consuming messages.
Sending messages:
 To send a message to a topic, make a POST request to the DSS API with a valid bearer token.
Example (using Vault token):

TOKEN=$(cat /vault/secrets/token)
curl -H "Authorization: Bearer $TOKEN" \
-H "Connection: close" \
-d '{"message": "testing 123"}' \
http://streams/mytopic
Query parameters and headers:
  • batch
     (query param): Send an array of JSON objects; each becomes a separate message.
  • x-account:
     Specifies the encryption context for the message.
  • x-key:
     Enforces message ordering by key.
  • x-type, x-class:
     Provide message classification metadata (producer-defined).
Receiving messages:
 DSS provides two methods for message consumption:
  1. Pull-based (single message)
    Curl example:
    
TOKEN=$(cat /vault/secrets/token)
curl -H "Authorization: Bearer $TOKEN" \
-H "Connection: close" \
http://streams/mytopic

GET <base-url>/<topic>
GET <base-url>/<topic>/<consumerGroup>
  2. Server-Sent Events (SSE) for streaming
    Curl example for SSE:
    
TOKEN=$(cat /vault/secrets/token)
curl -H "Authorization: Bearer $TOKEN" \
-N -H "Connection: close" \
http://streams/mytopic?tail=true

GET <base-url>/<topic>?tail=true
GET <base-url>/<topic>/<consumerGroup>?tail=true

Using filtering

The Data Streaming Service (DSS) supports message filtering using the 
q
query parameter. Filters can be applied to both the message payload (
data
) and the Common Message Format (CMF) headers (
headers
).
Example request with filter:
Filters can be applied to:
  • Payload properties
     ( 
    data
    )
  • CMF header properties
     ( 
    headers
    )
Sample queries:
  • q=data.testing==123
  • q=headers.message_class==mymessageclass
Using Boolean operators:
  • AND:
     
    q=data.testing==123&&data.extra.say==blah
  • OR:
     
    q=headers.message_class==mymessageclass||data.testing!=123

Obtaining a token

DSS uses
Vault Agent Injection
 for token management. Tokens are injected into the container for authentication with DSS.
Vault annotations for Kubernetes (K8s):

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/agent-inject-token: "true"
  vault.hashicorp.com/role: "my-data-streaming-service"
The token is made accessible at 
/vault/secrets/token
inside the container. This token must be used to authenticate with DSS APIs.

Admin and metrics APIs

DSS provides additional APIs for administrative operations and performance monitoring.

Topic and service management

To register a new topic in DSS, define a
Topic
 object using the configuration schema below. This configuration allows you to control how messages are retained, partitioned, and managed within the system.
Creating a topic:
 Define a Topic object using the following schema:

apiVersion: synapps.kyndryl.com/v1
kind: Topic
metadata:
annotations:
  synapps/keep: "true"
  helm.sh/resource-policy: keep
name:
spec:
partitions:
retentionMs:
cleanupPolicy:

Data model

The Data Streaming Service (DSS) follows a
publish-subscribe
 model, enabling seamless data exchange between producers and consumers. To ensure a successful integration, we recommend collaborating with the Kyndryl Bridge Integration team to review the service’s data model and validate its alignment with DSS requirements.
The key entities in DSS are:
  1. Topics:
     Named channels that serve as the entry point for streaming data.
  2. Messages:
     Payloads of data that adhere to specific schemas defined in the registry.
  3. Producers:
     Services that publish data to topics.
  4. Consumers:
     Services that subscribe to topics and process data.
  5. Schema registry:
     A repository that stores message schemas used to validate data format.

Troubleshooting common issues

When working with DSS, you might encounter the following common issues:
  1. Topic not found:
     Ensure that the topic is registered in the Topic Registry and that the name follows the correct naming conventions.
  2. Schema validation errors:
     Check that the message format matches the registered schema for the topic.
  3. Consumer group errors:
     Make sure the consumer group is correctly defined and unique to avoid conflicts.

Best practices

Follow these best practices to optimize your DSS integration and operations:
  1. Efficient partitioning:
    When setting partitions for a topic, consider the expected message load to balance performance and scalability. Proper key selection is crucial, as the key determines how messages are grouped and ensures they are ordered within a specific partition.
  2. Data retention management:
    Carefully plan retention settings based on the needs of consumers. For short-lived data, use lower retention times. If all data must be retained indefinitely, consider setting the retention to -1. This is particularly useful for static metadata lists where historical data should not be lost. Leverage compaction for such cases to reduce storage usage by retaining only the latest message per key.
  3. Schema consistency:
    Follow the Field Data Standards and ensure schema definitions are well-documented and consistently applied across producers and consumers.
Do you have two minutes for a quick survey?
Take Survey