AIOps Extensions Framework Troubleshooting Guide

introduction this guide is designed to help l1 and l2 support personnel troubleshoot issues with the aiops extension and dashboard problems before creating a snow ticket for l3 developers the goal is to identify and resolve issues by way of either l1 or l2 support and, failing that, enlist either development or platform teams as appropriate aiops relies on data collection extensions to optimize the health and uptime of network resources and are therefore an integral part of workflows and planning kyndryl has therefore implemented incident management and ticketing systems to notify support personnel of when data collection extensions fail using this guide, support personnel can perform procedures specific to individual extensions and understand when to escalate the issue at hand key components of the manual architecture overview visual diagrams explaining how producer extensions, synapps messaging, consumer extensions and kibana dashboards interact within the aks environment dashboard troubleshooting detailed procedures for addressing missing data issues stale data problems data inconsistencies complete troubleshooting flow chart for efficient issue identification producer extensions troubleshooting checking pod status and logs addressing authentication issues resolving external connectivity problems verifying successful data transmission to synapps consumer extensions troubleshooting diagnosing message processing errors checking elasticsearch connectivity resolving index mapping conflicts monitoring consumer group lag clear escalation guidelines specific criteria for when to escalate to l3 developers when to route issues to platform teams instead azure kubernetes service commands aks specific kubectl commands for pod status monitoring resource usage tracking configuration management network diagnostics specific elasticsearch and synapps (kafka) troubleshooting architecture overview the aiops platform uses an extensions framework to integrate with external systems there are two types of extensions producer extensions pull data from external systems and push it to synapps topics consumer extensions pull data from synapps topics and push it to elasticsearch indices the data is then visualized through kibana dashboards common issue categories issues can generally be categorized as dashboard issues problems with data visualization in kibana producer extension issues problems with collecting data from external systems consumer extension issues problems with processing data and storing in elasticsearch platform issues problems with underlying platform services (synapps, elasticsearch, kubernetes, iam, vault, etc ) dashboard related issues dashboard issues are often the first problems reported by users these typically manifest as missing data, stale data, or data that does not add up correctly missing data troubleshooting flow note in the case of a consumer extension failure, restart the consumer extension if the problem persists, escalate to l3 step 1 verify dashboard configuration check time range ensure the dashboard is set to the correct time range try expanding the time range to see if data appears check filters verify that no filters are inadvertently excluding data reset all filters and see if data appears check index pattern verify the dashboard is using the correct index pattern command to check available indices kubectl exec it $(kubectl get pods n elasticsearch l app=elasticsearch o name | head 1) n elasticsearch curl s 'localhost 9200/ cat/indices?v' step 2 check elasticsearch data check if data exists in the elasticsearch index kubectl exec it $(kubectl get pods n elasticsearch l app=elasticsearch o name | head 1) n elasticsearch curl s 'localhost 9200/index name/ search?size=1\&pretty' check index mapping kubectl exec it $(kubectl get pods n elasticsearch l app=elasticsearch o name | head 1) n elasticsearch curl s 'localhost 9200/index name/ mapping?pretty' check document count kubectl exec it $(kubectl get pods n elasticsearch l app=elasticsearch o name | head 1) n elasticsearch curl s 'localhost 9200/index name/ count?pretty' step 3 stale data issues check the last updated timestamp in the index kubectl exec it $(kubectl get pods n elasticsearch l app=elasticsearch o name | head 1) n elasticsearch curl s 'localhost 9200/index name/ search?sort=@timestamp\ desc\&size=1\&pretty' verify consumer extension schedule check the consumer extension logs for scheduled runs kubectl logs n extensions $(kubectl get pods n extensions l app=consumer extension name o name | head 1) tail=100 step 4 data consistency issues if data does not add up correctly check for duplicate data look for duplicate entries in elasticsearch check if the consumer extension is properly handling deduplication check aggregation settings verify that dashboard aggregations are configured correctly ensure the proper time field is being used for time based aggregations producer extensions issues step 1 check producer extension status check if the producer pods are running kubectl get pods n extensions l app=producer extension name check the producer extension logs kubectl logs n extensions $(kubectl get pods n extensions l app=producer extension name o name | head 1) tail=100 check pod resource usage kubectl top pod n extensions $(kubectl get pods n extensions l app=producer extension name o name | cut d/ f2) step 2 common producer extension issues authentication issues look for authentication errors in logs verify iam/vault integration is working check if credentials need to be updated \# check if extension can access vault kubectl exec it n extensions $(kubectl get pods n extensions l app=producer extension name o name | head 1) curl s http //vault vault namespace 8200/v1/health connection issues look for timeout or connection errors in logs check network connectivity to external system verify external system is operational rate limiting check for rate limiting errors in logs verify rate limit configuration in extension data format changes look for parsing errors in logs check if external api has changed format step 3 producer to environment integration check if producer is successfully sending to your environment look for successful message publishing logs check environment topic status kubectl exec it n namespace $(kubectl get pods n synapps l app=synapps admin o name | head 1) bin/kafka topics sh describe topic topic name bootstrap server synapps 9092 check message count in environment topic kubectl exec it n namespace $(kubectl get pods n synapps l app=synapps admin o name | head 1) bin/kafka run class sh kafka tools getoffsetshell broker list synapps 9092 topic topic name note the example commands uses the variable namespace as a placeholder for the environment namespace insert your environment namespace in place of the placeholder consumer extensions issues step 1 check consumer extension status check if the consumer pods are running kubectl get pods n extensions l app=consumer extension name check the consumer extension logs kubectl logs n extensions $(kubectl get pods n extensions l app=consumer extension name o name | head 1) tail=100 check pod resource usage kubectl top pod n extensions $(kubectl get pods n extensions l app=consumer extension name o name | cut d/ f2) step 2 common consumer extension issues if logs have not been recently populated, restart the consumer and revalidate the logs if necessary, repeat the restart up to three times with a delay of 15 minutes between restarts when logs have been populated, perform the following procedure message processing errors look for deserialization or processing errors in logs check if schema validation is failing elasticsearch connection issues look for elasticsearch connection errors check if elasticsearch is accessible from consumer pods kubectl exec it n extensions $(kubectl get pods n extensions l app=consumer extension name o name | head 1) curl s elasticsearch elasticsearch namespace 9200 index mapping conflicts look for field mapping errors in logs check if new fields are causing mapping issues consumer group issues check consumer group status kubectl exec it n synapps $(kubectl get pods n synapps l app=synapps admin o name | head 1) bin/kafka consumer groups sh describe group consumer group name bootstrap server synapps 9092 step 3 synapps to consumer integration check if consumer is successfully reading from synapps look for message consumption logs check if messages are being processed but failing to index look for elasticsearch indexing errors check consumer lag kubectl exec it n synapps $(kubectl get pods n synapps l app=synapps admin o name | head 1) bin/kafka consumer groups sh bootstrap server synapps 9092 describe group consumer group name when to escalate to l3 developers escalate to l3 developers only when confirmed code bugs reproducible extension failures with clear error logs incorrect data transformation logic parsing errors due to incorrect code logic schema validation failures due to code issues dashboard development issues dashboard visualization errors (when data exists in elasticsearch) incorrect aggregations or calculations in dashboards missing fields in visualizations when fields exist in elasticsearch extension performance issues slow data processing not related to infrastructure memory leaks in extension code high cpu usage due to inefficient algorithms important before escalation, gather the following information detailed error logs reproduction steps time window of the issue specific data samples showing the issue screenshots of dashboard issues all troubleshooting steps already performed when to escalate to platform teams escalate to platform teams when synapps issues synapps cluster unavailability topic configuration issues message retention issues performance problems with synapps elasticsearch issues elasticsearch cluster unavailability index capacity issues elasticsearch performance problems shard allocation issues aks/kubernetes issues node failures pod scheduling issues network connectivity between services resource constraints at the cluster level iam/vault issues authentication service unavailability token expiration issues secret access problems role permission issues appendix useful kubectl commands pod status commands \# get all pods in extensions namespace kubectl get pods n extensions \# get detailed information about a specific pod kubectl describe pod \[pod name] n extensions \# get logs from a specific pod kubectl logs \[pod name] n extensions \# get logs from previous instance of a pod (if it restarted) kubectl logs \[pod name] n extensions previous \# watch pod status in real time kubectl get pods n extensions w resource usage commands \# get cpu and memory usage for all pods kubectl top pods n extensions \# get cpu and memory usage for nodes kubectl top nodes configuration commands \# check configmaps used by extensions kubectl get configmaps n extensions \# view a specific configmap kubectl describe configmap \[configmap name] n extensions \# check secrets used by extensions kubectl get secrets n extensions \# check service accounts kubectl get serviceaccounts n extensions network commands \# check services in the extensions namespace kubectl get services n extensions \# describe a service to see endpoints kubectl describe service \[service name] n extensions \# test network connectivity from a pod kubectl exec it \[pod name] n extensions curl v \[service url] \# check dns resolution from a pod kubectl exec it \[pod name] n extensions nslookup \[service name] troubleshooting extensions \# check events related to a pod kubectl get events n extensions field selector involvedobject name=\[pod name] \# get a shell in a pod for advanced troubleshooting kubectl exec it \[pod name] n extensions /bin/ \# check environment variables in a pod kubectl exec \[pod name] n extensions env \# files from a pod for inspection kubectl cp \[pod name] /path/to/file /local/path n extensions elasticsearch specific commands \# check elasticsearch cluster health kubectl exec it $(kubectl get pods n elasticsearch l app=elasticsearch o name | head 1) n elasticsearch curl s 'localhost 9200/ cluster/health?pretty' \# list all indices kubectl exec it $(kubectl get pods n elasticsearch l app=elasticsearch o name | head 1) n elasticsearch curl s 'localhost 9200/ cat/indices?v' \# check index settings kubectl exec it $(kubectl get pods n el curl s 'localhost 9200/\[index name]/ settings?pretty' synapps (kafka) specific commands asticsearch l app=elasticsearch o name | head 1) n elasticsearch \# list all topics kubectl exec it n synapps $(kubectl get pods n synapps l app=synapps admin o name | head 1) bin/kafka topics sh list bootstrap server synapps 9092 \# describe a topic kubectl exec it n synapps $(kubectl get pods n synapps l app=synapps admin o name | head 1) bin/kafka topics sh describe topic \[topic name] bootstrap server synapps 9092 \# check consumer groups kubectl exec it n synapps $(kubectl get pods n synapps l app=synapps admin o name | head 1) bin/kafka consumer groups sh list bootstrap server synapps 9092