Zabbix Multi-Instance Onboarding
The Network Service Gateway supports multiple Zabbix environments. Instead of using a single Zabbix server, devices can be onboarded to different regional or functional Zabbix instances, such as:
- ZBXEUAP1
- ZBXGBL1
- ZBXUSN1
- ZBXDNS1
A single onboarding request may contain devices destined for different Zabbix instances.
The system determines the correct destination by using the Zabbix Instance Alias specified for each device record.
Runtime design: the lazy-init connection pool
Connections to Zabbix instances are managed by ZabbixInstancePool (app/api/modules/zabbix.py), wired up as the ZABBIX_POOL singleton and the get_zabbix_instance(alias) helper in app/api/modules/environment.py.
Key properties
- Lazy initialization — a connection for a given alias is created only on the first ZABBIX_POOL.get(alias) call, not at process startup. This avoids needing to know every possible alias in advance and avoids connecting to instances that are never used.
- Caching — once created, the Zabbix instance (and the credentials used to build it) are cached in memory for that alias. Subsequent requests reuse the same connection with no extra Vault reads, up to a TTL (30 minutes by default).
- Self-healing — if a call fails with a session-expiry error (Zabbix.is_session_error), the caller evicts the cached entry (ZABBIX_POOL.evict(alias)) and re-resolves the alias, which forces a fresh Vault read and a new login. This means a rotated Vault token or an expired Zabbix session recovers automatically, without restarting the pod.
- Thread-safety — a lock in the pool prevents two concurrent requests from initializing the same alias twice.
This design was chosen over "connect to everything at startup" because startup-init cannot recover from token rotation or session expiry without a restart, and over "read Vault + reconnect on every request" because that would multiply Vault reads and TCP/login overhead by every row in every request. See docs_review/zabbix-multi-instance-review-2026-06-24.md for the full design tradeoff analysis.
Request-level flow (/onboarding/zabbix and /onboarding/zabbix/validate)
- The endpoint reads the JSON payload (a list of device rows) and normalizes SNMPv3 protocol casing.
- Every row's Zabbix Instance Alias is validated for format (string, ≤255 chars, matches ^[a-zA-Z0-9_.\-]+$) and for presence (non-blank — there is no default-instance fallback; every row must declare its alias). Either failure short-circuits the whole request with a 422 before any Vault/Zabbix call is made.
- Rows are grouped by alias (_group_rows_by_alias), preserving each row's original index so error reporting can still point at the exact `[row, col]` cell in a request that interleaves rows for different aliases.
- For each alias group:
- get_zabbix_instance(alias) resolves (or reuses) the pooled connection.
- If resolution fails (unknown alias, unreachable host, DNS failure), the rows in that group get a per-row error and other groups continue processing independently — one bad alias does not fail devices destined for a different, healthy instance.
- If resolution succeeds, ZabbixOnboarding.create_inventory() / validate_inventory() runs against that instance for just that group's rows.
- If a session-expiry error is detected mid-call, the pool entry is evicted and the group is retried once against a freshly authenticated instance.
- Errors from all groups are merged back into a single response keyed by the original row/column coordinates, so the response format is identical to the single-instance case from the caller's point of view.
Vault secret structure for Zabbix
All Zabbix instance credentials live under one Vault secret — there is no per-alias sub-path. The secret is a single JSON object keyed by alias, and follows the structure of a plain Python dictionary:
Path: services/network-service-gateway/secrets/zabbix
{ "<ALIAS01>": { "token": "<token for ALIAS01>", "url": "<url for ALIAS01>", "user": "<user for ALIAS01>" }, "<ALIAS02>": { "token": "<token for ALIAS02>", "url": "<url for ALIAS02>", "user": "<user for ALIAS02>" } }
Real, production example (values sanitized):
{ "ZBXUSN1": { "token": "<API Token here>", "url": "https://zabbix-pg-tsdb.ona.kyndryl.net/api_jsonrpc.php", "user": "<API user here>" } }
Rules
- The top-level key is the alias. This is the exact string that must appear in the Zabbix Instance Alias column of the templates CSV. If a row's alias does not match a key in this secret, ZabbixInstancePool raises KeyError and the row is reported as an invalid/unresolvable alias.
- Each alias entry carries its own `token`, `url`, and `user` (token auth is the standard for production instances; user/passwordis also supported by the underlying Zabbix client shape if a given instance uses password auth instead of a token).
- `url` points at the Zabbix JSON-RPC API endpoint, i.e. it ends in /api_jsonrpc.php (e.g. https://zabbix-pg-tsdb.ona.kyndryl.net/api_jsonrpc.php), not just the host root.
- Aliases are case-sensitive. Vault paths and the credential lookup are case-sensitive — real-world provisioned aliases are commonly upper case (e.g. ZBXUSN1), and the gateway deliberately does not normalize case, so the CSV value must match the Vault key exactly.
- There is no default/fallback instance. Every alias used in a CSV must be provisioned as a key in this secret; there is no "use ZABBIX singleton if alias is missing" behavior for onboarding — the alias column is always required and always resolved through the pool.
- Adding a new instance means adding a new key to this one secret (no Helm values.yaml change is needed per instance, since it is all one path) and making sure whoever prepares templates CSVs knows the new alias.
Updating the secret via script
Production updates to this secret are performed with the DxVaultApiClient CLI script (temp/vault.py), not the Vault UI directly. The script authenticates (token file or AppRole), then prompts for an action.
When prompted for the service path (create/update/read/delete actions), enter just zabbix — the script automatically prepends SECRET_BASE_PATH (services/network-service-gateway/secrets), so the resulting path resolves to services/network-service-gateway/secrets/zabbix, matching what environment.py and ZabbixInstancePool read from at runtime. Do not enter the full path, and do not enter a per-alias path (e.g. zabbix/ZBXUSN1) — this deployment uses the single-object layout (§3), where every alias is a key inside the one zabbix secret.
For the generic create/update action, the script prompts for the secret contents as a Python dict literal (parsed with ast.literal_eval, not JSON) — this is where the "must be minified" requirement comes from: the value must be typed as a single-line dict, e.g.:
Enter the secrets dict (e.g. {'token': 'abc', 'url': 'https://example.com'}): {'ZBXUSN1': {'token': '<API Token here>', 'url': '<https://<zabbix-url-here>>', 'user': '<API user here>'}}
Equivalently, JSON is accepted as long as it is valid Python-dict syntax on one line (double quotes work fine with ast.literal_eval), e.g.:
{"ZBXUSN1":{"token":"<API Token here>","url":"https://zabbix-pg-tsdb.ona.kyndryl.net/api_jsonrpc.php","user":"<API user here>"}}
Under the create/update action, the script already reads the existing secret first and merges your input on top of it (merged_secret_data = {**existing_secret_data, **new_secret_data}) before writing back — so submitting just the one alias you are adding/changing is safe and will not delete other already-provisioned aliases. (This merge safety only applies to the generic create/update action; the dedicated put-zabbix action in the script instead overwrites the whole secret with its local ZABBIX_INSTANCES dict, so that action should only be used with ZABBIX_INSTANCES fully populated with every alias that must remain.)
Templates CSV: the Zabbix Instance Alias column
The onboarding templates CSV/JSON schema (ZabbixOnboarding.schema in app/api/modules/onboarding.py, mirrored by ZabbixTemplate.schema in app/api/modules/zabbix_templates.py) includes a Zabbix Instance Alias column alongside the existing columns (Hostname, Host IP, Host Group, Proxy, SNMP version, etc.). The column rules are as follows:
Metric | Description |
|---|---|
Type | String |
Maximum Length | 255 characters |
Allowed Characters | Letters, numbers, periods (.), underscores (_),
hyphens (-), ^[a-zA-Z0-9_.\-]+$ |
Required | Every row must have a non-blank value. There is no default instance. |
Case | Not normalized — must match the Vault secret key exactly (case-sensitive). |
Why this matters on every row
- Routing, not documentation. The alias isn't metadata — it is what the gateway uses to decide which live Zabbix connection handles that row. Two rows in the same file can (and often will) target completely different Zabbix servers.
- A blank or wrong alias fails only that row. Because grouping happens per alias, a mistyped alias on one row produces a per-row [row, col] error pointing at that exact cell — it does not block or misroute the other rows in the file, but it also means that row's device will not be onboarded until the alias is corrected.
- No silent fallback exists. If a CSV omits the alias for a row (e.g. because it was optional in an older template version), the request fails fast with a clear "non-blank value is required" error for that cell rather than guessing an instance — guessing wrong would silently create a host on the wrong Zabbix server.
- Typos look like "invalid alias", not "instance down". An alias that doesn't exist as a key in the Vault secret is indistinguishable, from the CSV author's point of view, from a genuine typo — both surface as "Zabbix instance alias value is not valid. Check with your administrator for possible values." Always double-check the exact alias spelling/case against the list of provisioned aliases before assuming an instance is down.
- Mixed-instance files are expected and supported. A single onboarding submission may legitimately contain, e.g., 40 devices for ZBXEUAP1 and 10 for ZBXGBL1 in the same file — this is by design (Class B: one alias per logical group of rows), not an edge case to avoid.
What NOT to do
- Do not leave the alias blank "to use the default instance" — there is no default; the request will be rejected.
- Do not assume alias matching is case-insensitive — zbxeuap1 and ZBXEUAP1 are different keys unless Vault happens to have both.
- Do not invent new aliases in a CSV without first confirming they exist as a key under services/network-service-gateway/secrets/zabbix — an unprovisioned alias will always fail resolution.