Zabbix Multi-Instance Onboarding

the network service gateway supports multiple zabbix environments instead of using a single zabbix server, devices can be onboarded to different regional or functional zabbix instances, such as zbxeuap1 zbxgbl1 zbxusn1 zbxdns1 a single onboarding request may contain devices destined for different zabbix instances the system determines the correct destination by using the zabbix instance alias specified for each device record runtime design the lazy init connection pool connections to zabbix instances are managed by zabbixinstancepool ( app/api/modules/zabbix py ), wired up as the zabbix pool singleton and the get zabbix instance(alias) helper in app/api/modules/environment py key properties lazy initialization — a connection for a given alias is created only on the first zabbix pool get(alias) call, not at process startup this avoids needing to know every possible alias in advance and avoids connecting to instances that are never used caching — once created, the zabbix instance (and the credentials used to build it) are cached in memory for that alias subsequent requests reuse the same connection with no extra vault reads, up to a ttl (30 minutes by default) self healing — if a call fails with a session expiry error ( zabbix is session error) , the caller evicts the cached entry (zabbix pool evict(alias)) and re resolves the alias, which forces a fresh vault read and a new login this means a rotated vault token or an expired zabbix session recovers automatically, without restarting the pod thread safety — a lock in the pool prevents two concurrent requests from initializing the same alias twice this design was chosen over "connect to everything at startup" because startup init cannot recover from token rotation or session expiry without a restart, and over "read vault + reconnect on every request" because that would multiply vault reads and tcp/login overhead by every row in every request see docs review/zabbix multi instance review 2026 06 24 md for the full design tradeoff analysis request level flow ( /onboarding/zabbix and /onboarding/zabbix/validate ) the endpoint reads the json payload (a list of device rows) and normalizes snmpv3 protocol casing every row's zabbix instance alias is validated for format (string, ≤255 chars, matches ^\[a za z0 9 \\ ]+$) and for presence (non blank — there is no default instance fallback ; every row must declare its alias) either failure short circuits the whole request with a 422 before any vault/zabbix call is made rows are grouped by alias ( group rows by alias ), preserving each row's original index so error reporting can still point at the exact `\[row, col]` cell in a request that interleaves rows for different aliases for each alias group get zabbix instance(alias) resolves (or reuses) the pooled connection if resolution fails (unknown alias, unreachable host, dns failure), the rows in that group get a per row error and other groups continue processing independently — one bad alias does not fail devices destined for a different, healthy instance if resolution succeeds, zabbixonboarding create inventory() / validate inventory() runs against that instance for just that group's rows if a session expiry error is detected mid call, the pool entry is evicted and the group is retried once against a freshly authenticated instance errors from all groups are merged back into a single response keyed by the original row/column coordinates, so the response format is identical to the single instance case from the caller's point of view vault secret structure for zabbix all zabbix instance credentials live under one vault secret — there is no per alias sub path the secret is a single json object keyed by alias, and follows the structure of a plain python dictionary path services/network service gateway/secrets/zabbix { "\<alias01>" { "token" "\<token for alias01>", "url" "\<url for alias01>", "user" "\<user for alias01>" }, "\<alias02>" { "token" "\<token for alias02>", "url" "\<url for alias02>", "user" "\<user for alias02>" } } real, production example (values sanitized) { "zbxusn1" { "token" "\<api token here>", "url" "https //zabbix pg tsdb ona kyndryl net/api jsonrpc php", "user" "\<api user here>" } } rules the top level key is the alias this is the exact string that must appear in the zabbix instance alias column of the templates csv if a row's alias does not match a key in this secret, zabbixinstancepool raises keyerror and the row is reported as an invalid/unresolvable alias each alias entry carries its own `token`, `url`, and `user` (token auth is the standard for production instances; user/password is also supported by the underlying zabbix client shape if a given instance uses password auth instead of a token) `url` points at the zabbix json rpc api endpoint , i e it ends in /api jsonrpc php (e g https //zabbix pg tsdb ona kyndryl net/api jsonrpc php) , not just the host root aliases are case sensitive vault paths and the credential lookup are case sensitive — real world provisioned aliases are commonly upper case (e g zbxusn1 ), and the gateway deliberately does not normalize case, so the csv value must match the vault key exactly there is no default/fallback instance every alias used in a csv must be provisioned as a key in this secret; there is no "use zabbix singleton if alias is missing" behavior for onboarding — the alias column is always required and always resolved through the pool adding a new instance means adding a new key to this one secret (no helm values yaml change is needed per instance, since it is all one path) and making sure whoever prepares templates csvs knows the new alias updating the secret via script production updates to this secret are performed with the dxvaultapiclient cli script ( temp/vault py ), not the vault ui directly the script authenticates (token file or approle), then prompts for an action when prompted for the service path ( create/update/read/delete actions), enter just zabbix — the script automatically prepends secret base path (services/network service gateway/secrets) , so the resulting path resolves to services/network service gateway/secrets/zabbix, matching what environment py and zabbixinstancepool read from at runtime do not enter the full path, and do not enter a per alias path (e g zabbix/zbxusn1 ) — this deployment uses the single object layout (§3), where every alias is a key inside the one zabbix secret for the generic create/update action, the script prompts for the secret contents as a python dict literal (parsed with ast literal eval , not json) — this is where the "must be minified" requirement comes from the value must be typed as a single line dict, e g enter the secrets dict (e g {'token' 'abc', 'url' 'https //example com'}) {'zbxusn1' {'token' '\<api token here>', 'url' '\<https //\<zabbix url here>>', 'user' '\<api user here>'}} equivalently, json is accepted as long as it is valid python dict syntax on one line (double quotes work fine with ast literal eval), e g {"zbxusn1" {"token" "\<api token here>","url" "https //zabbix pg tsdb ona kyndryl net/api jsonrpc php","user" "\<api user here>"}} under the create/update action, the script already reads the existing secret first and merges your input on top of it (merged secret data = { existing secret data, new secret data}) before writing back — so submitting just the one alias you are adding/changing is safe and will not delete other already provisioned aliases (this merge safety only applies to the generic create/update action; the dedicated put zabbix action in the script instead overwrites the whole secret with its local zabbix instances dict, so that action should only be used with zabbix instances fully populated with every alias that must remain ) templates csv the zabbix instance alias column the onboarding templates csv/json schema ( zabbixonboarding schema in app/api/modules/onboarding py , mirrored by zabbixtemplate schema in app/api/modules/zabbix templates py ) includes a zabbix instance alias column alongside the existing columns ( hostname, host ip, host group, proxy, snmp version, etc ) the column rules are as follows \<font color="#f3f4f6"> metric \</font> \<font color="#f3f4f6"> description \</font> type string maximum length 255 characters allowed characters letters, numbers, periods ( ), underscores ( ), hyphens ( ), ^\[a za z0 9 \\ ]+$ required every row must have a non blank value there is no default instance case not normalized — must match the vault secret key exactly (case sensitive) why this matters on every row routing, not documentation the alias isn't metadata — it is what the gateway uses to decide which live zabbix connection handles that row two rows in the same file can (and often will) target completely different zabbix servers a blank or wrong alias fails only that row because grouping happens per alias, a mistyped alias on one row produces a per row \[row, col] error pointing at that exact cell — it does not block or misroute the other rows in the file, but it also means that row's device will not be onboarded until the alias is corrected no silent fallback exists if a csv omits the alias for a row (e g because it was optional in an older template version), the request fails fast with a clear "non blank value is required" error for that cell rather than guessing an instance — guessing wrong would silently create a host on the wrong zabbix server typos look like "invalid alias", not "instance down" an alias that doesn't exist as a key in the vault secret is indistinguishable, from the csv author's point of view, from a genuine typo — both surface as "zabbix instance alias value is not valid check with your administrator for possible values " always double check the exact alias spelling/case against the list of provisioned aliases before assuming an instance is down mixed instance files are expected and supported a single onboarding submission may legitimately contain, e g , 40 devices for zbxeuap1 and 10 for zbxgbl1 in the same file — this is by design (class b one alias per logical group of rows), not an edge case to avoid what not to do do not leave the alias blank "to use the default instance" — there is no default; the request will be rejected do not assume alias matching is case insensitive — zbxeuap1 and zbxeuap1 are different keys unless vault happens to have both do not invent new aliases in a csv without first confirming they exist as a key under services/network service gateway/secrets/zabbix — an unprovisioned alias will always fail resolution