--- title: Zabbix Multi-Instance Onboarding slug: zabbix-multi-instance-onboarding docTags: createdAt: 2026-07-22T12:19:45.796Z --- The Network Service Gateway supports multiple Zabbix environments. Instead of using a single Zabbix server, devices can be onboarded to different regional or functional Zabbix instances, such as: - ZBXEUAP1 - ZBXGBL1 - ZBXUSN1 - ZBXDNS1 A single onboarding request may contain devices destined for different Zabbix instances. The system determines the correct destination by using the **`Zabbix Instance Alias`** specified for each device record. # Runtime design: the lazy-init connection pool Connections to Zabbix instances are managed by` ZabbixInstancePool `(app/api/modules/zabbix.py), wired up as the `ZABBIX_POOL` singleton and the `get_zabbix_instance(alias) `helper in app/api/modules/environment.py. ### Key properties - **Lazy initialization** — a connection for a given alias is created only on the first `ZABBIX_POOL.get(alias)` call, not at process startup. This avoids needing to know every possible alias in advance and avoids connecting to instances that are never used. - **Caching** — once created, the `Zabbix `instance (and the credentials used to build it) are cached in memory for that alias. Subsequent requests reuse the same connection with no extra Vault reads, up to a TTL (30 minutes by default). - **Self-healing** — if a call fails with a session-expiry error (`Zabbix.is_session_error)`, the caller evicts the cached entry `(ZABBIX_POOL.evict(alias))` and re-resolves the alias, which forces a fresh Vault read and a new login. This means a rotated Vault token or an expired Zabbix session recovers automatically, without restarting the pod. - **Thread-safety** — a lock in the pool prevents two concurrent requests from initializing the same alias twice. This design was chosen over "connect to everything at startup" because startup-init cannot recover from token rotation or session expiry without a restart, and over "read Vault + reconnect on every request" because that would multiply Vault reads and TCP/login overhead by every row in every request. See docs\_review/zabbix-multi-instance-review-2026-06-24.md for the full design tradeoff analysis. ## Request-level flow (*/onboarding/zabbix *and */onboarding/zabbix/validate*) 1. The endpoint reads the JSON payload (a list of device rows) and normalizes SNMPv3 protocol casing. 2. Every row's `Zabbix Instance Alias `is validated for **format** (string, ≤255 chars, matches `^[a-zA-Z0-9_.\-]+$)` and for **presence** (non-blank — there is **no default-instance fallback**; every row must declare its alias). Either failure short-circuits the whole request with a 422 before any Vault/Zabbix call is made. 3. Rows are grouped by alias (`_group_rows_by_alias`), preserving each row's original index so error reporting can still point at the exact \`\[row, col]\` cell in a request that interleaves rows for different aliases. 4. For each alias group: - `get_zabbix_instance(alias)` resolves (or reuses) the pooled connection. - If resolution fails (unknown alias, unreachable host, DNS failure), the rows in that group get a per-row error and other groups continue processing independently — one bad alias does not fail devices destined for a different, healthy instance. - If resolution succeeds, `ZabbixOnboarding.create_inventory() / validate_inventory()` runs against that instance for just that group's rows. - If a session-expiry error is detected mid-call, the pool entry is evicted and the group is retried once against a freshly authenticated instance. 5. Errors from all groups are merged back into a single response keyed by the original row/column coordinates, so the response format is identical to the single-instance case from the caller's point of view. # Vault secret structure for Zabbix All Zabbix instance credentials live under **one** Vault secret — there is no per-alias sub-path. The secret is a single JSON object keyed by alias, and follows the structure of a plain Python dictionary: **Path:*** services/network-service-gateway/secrets/zabbix* \{ "\": \{ "token": "\", "url": "\", "user": "\" }, "\": \{ "token": "\", "url": "\", "user": "\" } } Real, production example (values sanitized): \{ "ZBXUSN1": \{ "token": "\", "url": "https\://zabbix-pg-tsdb.ona.kyndryl.net/api\_jsonrpc.php", "user": "\" } } ### Rules - **The top-level key is the alias.** This is the exact string that must appear in the `Zabbix Instance Alias` column of the templates CSV. If a row's alias does not match a key in this secret, `ZabbixInstancePool` raises KeyError and the row is reported as an invalid/unresolvable alias. - **Each alias entry carries its own \`token\`, \`url\`, and \`user\`** (token auth is the standard for production instances; *`user/password`*is also supported by the underlying `Zabbix `client shape if a given instance uses password auth instead of a token). - **\`url\` points at the Zabbix JSON-RPC API endpoint**, i.e. it ends in `/api_jsonrpc.php (e.g. https://zabbix-pg-tsdb.ona.kyndryl.net/api_jsonrpc.php)`, not just the host root. - **Aliases are case-sensitive.** Vault paths and the credential lookup are case-sensitive — real-world provisioned aliases are commonly upper case (e.g. `ZBXUSN1`), and the gateway deliberately does **not** normalize case, so the CSV value must match the Vault key exactly. - **There is no default/fallback instance.** Every alias used in a CSV must be provisioned as a key in this secret; there is no "use ZABBIX singleton if alias is missing" behavior for onboarding — the alias column is always required and always resolved through the pool. - Adding a new instance means adding a new key to this one secret (no Helm `values.yaml `change is needed per instance, since it is all one path) and making sure whoever prepares templates CSVs knows the new alias. ## Updating the secret via script Production updates to this secret are performed with the *DxVaultApiClient* CLI script (temp/vault.py), not the Vault UI directly. The script authenticates (token file or AppRole), then prompts for an **action**. When prompted for the **service path** (*`create/update/read/delete`* actions), enter just *zabbix *— the script automatically prepends `SECRET_BASE_PATH (services/network-service-gateway/secrets)`, so the resulting path resolves to `services/network-service-gateway/secrets/zabbix,` matching what `environment.py` and `ZabbixInstancePool` read from at runtime. Do not enter the full path, and do not enter a per-alias path (e.g. `zabbix/ZBXUSN1`) — this deployment uses the single-object layout (§3), where every alias is a key inside the one `zabbix` secret. For the generic `create/update` action, the script prompts for the secret contents as a **Python dict literal** (parsed with `ast.literal_eval`, not JSON) — this is where the "must be minified" requirement comes from: the value must be typed as a single-line dict, e.g.: `Enter the secrets dict (e.g. {'token': 'abc', 'url': 'https://example.com'}):` `{'ZBXUSN1': {'token': '', 'url': '>', 'user': ''}}` Equivalently, JSON is accepted as long as it is valid Python-dict syntax on one line (double quotes work fine with `ast.literal_eval),` e.g.: `{"ZBXUSN1":{"token":"","url":"https://zabbix-pg-tsdb.ona.kyndryl.net/api_jsonrpc.php","user":""}}` Under the `create/update `action, the script **already reads the existing secret first and merges** your input on top of it `(merged_secret_data = {**existing_secret_data, **new_secret_data})` before writing back — so submitting just the one alias you are adding/changing is safe and will not delete other already-provisioned aliases. (This merge safety only applies to the generic `create/update` action; the dedicated `put-zabbix `action in the script instead overwrites the whole secret with its local `ZABBIX_INSTANCES` dict, so that action should only be used with `ZABBIX_INSTANCES` fully populated with every alias that must remain.) # Templates CSV: the Zabbix Instance Alias column The onboarding templates CSV/JSON schema (`ZabbixOnboarding.schema` in app/api/modules/onboarding.py, mirrored by` ZabbixTemplate.schema` in app/api/modules/zabbix\_templates.py) includes a `Zabbix Instance Alias` column alongside the existing columns (`Hostname, Host IP, Host Group, Proxy, SNMP version, `etc.). The column rules are as follows: | **Metric** | **Description** | | --------------------------------------- | ---------------------------------------------------------------------------------------- | | Type | String | | Maximum Length | 255 characters | | Allowed Characters | Letters, numbers, periods (.), underscores (\_),
hyphens (-), ^\[a-zA-Z0-9\_.\\-]+$ | | Required | Every row must have a non-blank value.
There is no default instance. | | Case | Not normalized — must match the Vault secret key
exactly (case-sensitive). | ## Why this matters on every row - **Routing, not documentation.** The alias isn't metadata — it is what the gateway uses to decide *which live Zabbix connection* handles that row. Two rows in the same file can (and often will) target completely different Zabbix servers. - **A blank or wrong alias fails only that row.** Because grouping happens per alias, a mistyped alias on one row produces a per-row `[row, col]` error pointing at that exact cell — it does not block or misroute the other rows in the file, but it also means that row's device will not be onboarded until the alias is corrected. - **No silent fallback exists.** If a CSV omits the alias for a row (e.g. because it was optional in an older template version), the request fails fast with a clear "non-blank value is required" error for that cell rather than guessing an instance — guessing wrong would silently create a host on the wrong Zabbix server. - **Typos look like "invalid alias", not "instance down".** An alias that doesn't exist as a key in the Vault secret is indistinguishable, from the CSV author's point of view, from a genuine typo — both surface as "Zabbix instance alias value is not valid. Check with your administrator for possible values." Always double-check the exact alias spelling/case against the list of provisioned aliases before assuming an instance is down. - **Mixed-instance files are expected and supported.** A single onboarding submission may legitimately contain, e.g., 40 devices for `ZBXEUAP1` and 10 for `ZBXGBL1 `in the same file — this is by design (Class B: one alias per logical group of rows), not an edge case to avoid. ## What NOT to do - Do not leave the alias blank "to use the default instance" — there is no default; the request will be rejected. - Do not assume alias matching is case-insensitive — `zbxeuap1 `and `ZBXEUAP1 `are different keys unless Vault happens to have both. - Do not invent new aliases in a CSV without first confirming they exist as a key under `services/network-service-gateway/secrets/zabbix` — an unprovisioned alias will always fail resolution.