---
title: AIOps Extensions Framework Troubleshooting Guide
slug: aiops-extensions-framework-troubleshooting-guide
docTags: 
createdAt: 2026-07-17T07:37:36.540Z
---

## Introduction

This guide is designed to help L1 and L2 Support personnel troubleshoot issues with the AIOps Extension and Dashboard problems before creating a SNOW ticket for L3 Developers. The goal is to identify and resolve issues by way of either L1 or L2 support and, failing that, enlist either Development or Platform teams as appropriate.

AIOps relies on data collection extensions to optimize the health and uptime of Network resources and are therefore an integral part of workflows and planning. Kyndryl has therefore implemented incident management and ticketing systems to notify support personnel of when data collection extensions fail.

Using this guide, support personnel can perform procedures specific to individual extensions and understand when to escalate the issue at hand.

### Key components of the manual

1. **Architecture Overview**: Visual diagrams explaining how Producer Extensions, Synapps Messaging, Consumer Extensions and Kibana Dashboards interact within the AKS environment.
2. **Dashboard Troubleshooting**: Detailed procedures for addressing: 
   - Missing data issues
   - Stale data problems
   - Data inconsistencies
   - Complete troubleshooting flow chart for efficient issue identification
3. **Producer Extensions Troubleshooting**:
   - Checking pod status and logs
   - Addressing authentication issues
   - Resolving external connectivity problems
   - Verifying successful data transmission to Synapps
4. **Consumer Extensions Troubleshooting**:
   - Diagnosing message processing errors
   - Checking Elasticsearch connectivity
   - Resolving index mapping conflicts
   - Monitoring consumer group lag
5. **Clear Escalation Guidelines**: 
   - Specific criteria for when to escalate to L3 Developers
   - When to route issues to Platform teams instead
6. **Azure Kubernetes Service Commands**: AKS-specific kubectl commands for: 
   - Pod status monitoring
   - Resource usage tracking
   - Configuration management
   - Network diagnostics
   - Specific Elasticsearch and Synapps (Kafka) troubleshooting

## Architecture overview

![](https://api.archbee.com/api/optimize/QU1TIK52ntw5Jsos0uOt5/eahQJTiolL6D9Vahh7S53_image.png)

The AIOps platform uses an Extensions framework to integrate with external systems. There are two types of Extensions:

1. **Producer Extensions**: Pull data from external systems and push it to Synapps topics
2. **Consumer Extensions**: Pull data from Synapps topics and push it to Elasticsearch indices

The data is then visualized through Kibana dashboards.

## Common issue categories

Issues can generally be categorized as:

1. **Dashboard Issues**: Problems with data visualization in Kibana
2. **Producer Extension Issues**: Problems with collecting data from external systems
3. **Consumer Extension Issues**: Problems with processing data and storing in Elasticsearch
4. **Platform Issues**: Problems with underlying platform services (Synapps, Elasticsearch, Kubernetes, IAM, Vault, etc.)

### Dashboard-related issues

Dashboard issues are often the first problems reported by users. These typically manifest as missing data, stale data, or data that does not add up correctly.

### Missing data troubleshooting flow

![](https://api.archbee.com/api/optimize/QU1TIK52ntw5Jsos0uOt5/wV1sma0vMlyZFMnWvUN2-_image.png)

:::hint{type="info"}
**Note: **In the case of a consumer extension failure, restart the consumer extension. If the problem persists, escalate to L3.
:::

**Step 1: Verify Dashboard Configuration**

1. **Check Time Range**
   - Ensure the dashboard is set to the correct time range
   - Try expanding the time range to see if data appears
2. **Check Filters**
   - Verify that no filters are inadvertently excluding data
   - Reset all filters and see if data appears
3. **Check Index Pattern**
   - Verify the dashboard is using the correct index pattern
   - Command to check available indices:

`kubectl exec -it $(kubectl get pods -n elasticsearch -l app=elasticsearch -o name | head -1) -n elasticsearch -- curl -s 'localhost:9200/_cat/indices?v'`

**Step 2: Check Elasticsearch Data**

1. **Check if data exists in the Elasticsearch index**

`kubectl exec -it $(kubectl get pods -n elasticsearch -l app=elasticsearch -o name | head -1) -n elasticsearch -- curl -s 'localhost:9200/index-name/_search?size=1&pretty'`

2. **Check index mapping**

`kubectl exec -it $(kubectl get pods -n elasticsearch -l app=elasticsearch -o name | head -1) -n elasticsearch -- curl -s 'localhost:9200/index-name/_mapping?pretty'`

3. **Check document count**

`kubectl exec -it $(kubectl get pods -n elasticsearch -l app=elasticsearch -o name | head -1) -n elasticsearch -- curl -s 'localhost:9200/index-name/_count?pretty'`

**Step 3: Stale Data Issues**

1. **Check the last updated timestamp in the index**

`kubectl exec -it $(kubectl get pods -n elasticsearch -l app=elasticsearch -o name | head -1) -n elasticsearch -- curl -s 'localhost:9200/index-name/_search?sort=@timestamp:desc&size=1&pretty'`

2. **Verify Consumer Extension schedule**: Check the consumer extension logs for scheduled runs

`kubectl logs -n extensions $(kubectl get pods -n extensions -l app=consumer-extension-name -o name | head -1) --tail=100`

**Step 4: Data Consistency Issues**

If data does not add up correctly:

1. **Check for duplicate data**
   - Look for duplicate entries in Elasticsearch
   - Check if the consumer extension is properly handling deduplication
2. **Check aggregation settings**
   - Verify that dashboard aggregations are configured correctly
   - Ensure the proper time field is being used for time-based aggregations

### Producer Extensions Issues

**Step 1: Check Producer Extension Status**

1. **Check if the producer pods are running**

`kubectl get pods -n extensions -l app=producer-extension-name`

2. **Check the producer extension logs**

`kubectl logs -n extensions $(kubectl get pods -n extensions -l app=producer-extension-name -o name | head -1) --tail=100`

3. **Check pod resource usage**

`kubectl top pod -n extensions $(kubectl get pods -n extensions -l app=producer-extension-name -o name | cut -d/ -f2)`

**Step 2: Common Producer Extension Issues**

1. **Authentication Issues**
   - Look for authentication errors in logs
   - Verify IAM/Vault integration is working
   - Check if credentials need to be updated

`# Check if extension can access vault`

`kubectl exec -it -n extensions $(kubectl get pods -n extensions -l app=producer-extension-name -o name | head -1) -- curl -s http://vault.vault-namespace:8200/v1/health`

2. **Connection Issues**
   - Look for timeout or connection errors in logs
   - Check network connectivity to external system
   - Verify external system is operational
3. **Rate Limiting**
   - Check for rate limiting errors in logs
   - Verify rate limit configuration in extension
4. **Data Format Changes**
   - Look for parsing errors in logs
   - Check if external API has changed format

**Step 3: Producer to environment Integration**

1. **Check if producer is successfully sending to your environment**
   - Look for successful message publishing logs
2. **Check environment topic status**

`kubectl exec -it -n namespace $(kubectl get pods -n synapps -l app=synapps-admin -o name | head -1) -- bin/kafka-topics.sh --describe --topic topic-name --bootstrap-server synapps:9092`

3. **Check message count in environment topic**

`kubectl exec -it -n namespace $(kubectl get pods -n synapps -l app=synapps-admin -o name | head -1) -- bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list synapps:9092 --topic topic-name`

:::hint{type="info"}
**Note:** The example commands uses the variable *namespace* as a placeholder for the environment namespace. Insert your environment namespace in place of the placeholder.
:::

## Consumer Extensions Issues

**Step 1: Check Consumer Extension Status**

1. **Check if the consumer pods are running**

`kubectl get pods -n extensions -l app=consumer-extension-name`

2. **Check the consumer extension logs**

`kubectl logs -n extensions $(kubectl get pods -n extensions -l app=consumer-extension-name -o name | head -1) --tail=100`

3. **Check pod resource usage**

`kubectl top pod -n extensions $(kubectl get pods -n extensions -l app=consumer-extension-name -o name | cut -d/ -f2)`

**Step 2: Common Consumer Extension Issues**

If logs have not been recently populated, restart the consumer and revalidate the logs. If necessary, repeat the restart up to three times with a delay of 15 minutes between restarts. When logs have been populated, perform the following procedure.

1. **Message Processing Errors**
   - Look for deserialization or processing errors in logs
   - Check if schema validation is failing
2. **Elasticsearch Connection Issues**
   - Look for Elasticsearch connection errors
   - Check if Elasticsearch is accessible from consumer pods

`kubectl exec -it -n extensions $(kubectl get pods -n extensions -l app=consumer-extension-name -o name | head -1) -- curl -s elasticsearch.elasticsearch-namespace:9200`

3. **Index Mapping Conflicts**
   - Look for field mapping errors in logs
   - Check if new fields are causing mapping issues
4. **Consumer Group Issues**
   - Check consumer group status

`kubectl exec -it -n synapps $(kubectl get pods -n synapps -l app=synapps-admin -o name | head -1) -- bin/kafka-consumer-groups.sh --describe --group consumer-group-name --bootstrap-server synapps:9092`

**Step 3: Synapps to Consumer Integration**

1. **Check if consumer is successfully reading from Synapps**
   - Look for message consumption logs
2. **Check if messages are being processed but failing to index**
   - Look for Elasticsearch indexing errors
3. **Check consumer lag**

`kubectl exec -it -n synapps $(kubectl get pods -n synapps -l app=synapps-admin -o name | head -1) -- bin/kafka-consumer-groups.sh --bootstrap-server synapps:9092 --describe --group consumer-group-name`

## When to Escalate to L3 Developers

Escalate to L3 Developers **ONLY** when:

1. **Confirmed Code Bugs**: 
   - Reproducible extension failures with clear error logs
   - Incorrect data transformation logic
   - Parsing errors due to incorrect code logic
   - Schema validation failures due to code issues
2. **Dashboard Development Issues**: 
   - Dashboard visualization errors (when data exists in Elasticsearch)
   - Incorrect aggregations or calculations in dashboards
   - Missing fields in visualizations when fields exist in Elasticsearch
3. **Extension Performance Issues**: 
   - Slow data processing not related to infrastructure
   - Memory leaks in extension code
   - High CPU usage due to inefficient algorithms

:::hint{type="info"}
**Important:** Before escalation, gather the following information:
:::

- Detailed error logs
- Reproduction steps
- Time window of the issue
- Specific data samples showing the issue
- Screenshots of dashboard issues
- All troubleshooting steps already performed

## When to Escalate to Platform Teams

Escalate to Platform Teams when:

1. **Synapps Issues**: 
   - Synapps cluster unavailability
   - Topic configuration issues
   - Message retention issues
   - Performance problems with Synapps
2. **Elasticsearch Issues**: 
   - Elasticsearch cluster unavailability
   - Index capacity issues
   - Elasticsearch performance problems
   - Shard allocation issues
3. **AKS/Kubernetes Issues**: 
   - Node failures
   - Pod scheduling issues
   - Network connectivity between services
   - Resource constraints at the cluster level
4. **IAM/Vault Issues**: 
   - Authentication service unavailability
   - Token expiration issues
   - Secret access problems
   - Role permission issues

**Appendix: Useful kubectl Commands**

**Pod Status Commands**

`# Get all pods in extensions namespace`

`kubectl get pods -n extensions`

`# Get detailed information about a specific pod`

`kubectl describe pod [pod-name] -n extensions`

`# Get logs from a specific pod`

`kubectl logs [pod-name] -n extensions`

`# Get logs from previous instance of a pod (if it restarted)`

`kubectl logs [pod-name] -n extensions --previous`

`# Watch pod status in real-time`

`kubectl get pods -n extensions -w`

**Resource Usage Commands**

`# Get CPU and memory usage for all pods`

`kubectl top pods -n extensions`

`# Get CPU and memory usage for nodes`

`kubectl top nodes`

**Configuration Commands**

`# Check ConfigMaps used by extensions`

`kubectl get configmaps -n extensions`

`# View a specific ConfigMap`

`kubectl describe configmap [configmap-name] -n extensions`

`# Check Secrets used by extensions`

`kubectl get secrets -n extensions`

`# Check service accounts`

`kubectl get serviceaccounts -n extensions`

**Network Commands**

`# Check services in the extensions namespace`

`kubectl get services -n extensions`

`# Describe a service to see endpoints`

`kubectl describe service [service-name] -n extensions`

`# Test network connectivity from a pod`

`kubectl exec -it [pod-name] -n extensions -- curl -v [service-url]`

`# Check DNS resolution from a pod`

`kubectl exec -it [pod-name] -n extensions -- nslookup [service-name]`

**Troubleshooting Extensions**

`# Check events related to a pod`

`kubectl get events -n extensions --field-selector involvedObject.name=[pod-name]`

`# Get a shell in a pod for advanced troubleshooting`

`kubectl exec -it [pod-name] -n extensions -- /bin/`

`# Check environment variables in a pod`

`kubectl exec [pod-name] -n extensions -- env`

`#  files from a pod for inspection`

`kubectl cp [pod-name]:/path/to/file /local/path -n extensions`

**Elasticsearch Specific Commands**

`# Check Elasticsearch cluster health`

`kubectl exec -it $(kubectl get pods -n elasticsearch -l app=elasticsearch -o name | head -1) -n elasticsearch -- curl -s 'localhost:9200/_cluster/health?pretty'`

`# List all indices`

`kubectl exec -it $(kubectl get pods -n elasticsearch -l app=elasticsearch -o name | head -1) -n elasticsearch -- curl -s 'localhost:9200/_cat/indices?v'`

`# Check index settings`

`kubectl exec -it $(kubectl get pods -n el curl -s 'localhost:9200/[index-name]/_settings?pretty'`

**Synapps (Kafka) Specific Commands**

`asticsearch -l app=elasticsearch -o name | head -1) -n elasticsearch --`

`# List all topics`

`kubectl exec -it -n synapps $(kubectl get pods -n synapps -l app=synapps-admin -o name | head -1) -- bin/kafka-topics.sh --list --bootstrap-server synapps:9092`

`# Describe a topic`

`kubectl exec -it -n synapps $(kubectl get pods -n synapps -l app=synapps-admin -o name | head -1) -- bin/kafka-topics.sh --describe --topic [topic-name] --bootstrap-server synapps:9092`

`# Check consumer groups`

`kubectl exec -it -n synapps $(kubectl get pods -n synapps -l app=synapps-admin -o name | head -1) -- bin/kafka-consumer-groups.sh --list --bootstrap-server synapps:9092`