Live OSCAR telemetry in the Ops Console — and the platform auth work it uncovered

The integration became observable, priced, and demoable

This supplement records the cross-repository work after the original 11 August status cut. It is deliberately split by evidence: a merged PR is not described as a fresh Azure observation, while the push path is described as proven because it carried a real dev turn end to end. The Linear records are in the canopy-airiam team, across the PMS Backend and Ops Console projects.

A single agentic turn, end to end

What to show in a short walkthrough

1. Start at Services. The live category wall now leads; the export-freshness banner expressly describes the snapshot panels below it. In the endpoint table, select a service, search for a route, or isolate endpoints with recorded errors—these are local controls over the same labelled snapshot rows, not a new live query.
2. Show measured cost in context. On Instrumentation, the daily table exposes each rendered measured row's day, service, priced calls, and cents. Its optional date bounds filter only that table and leave the separately labelled historical estimate untouched.
3. Open Instrumentation before the turn. The live tail is intentionally empty until it sees a new event; that is evidence it is not replaying a fixture or historical transcript.
4. Drive one real agentic turn. The stream should gain a humanised step and coloured timeline lane in the same browser session. The measured cost enters OSCAR telemetry; its appearance in export-backed totals depends on the next capture, rather than being falsely presented as instantaneous.
5. Close on provenance. Measured cost, estimated historical cost, snapshot data, and live push each say what they are. The console does not flatten them into a single convenient but misleading number.

Replace the export with a live read — through a governed seam

The easy version of this is a public endpoint and a bearer token. That was considered and rejected. OSCAR's dev environment is network-isolated, and the console runs in a different resource group with its own managed identity, so the naive shape would have meant making an OSCAR service internet-reachable to a caller whose identity could not be pinned.

The shape we built instead: the console never talks to a data service directly. It authenticates as itself to the Action Gateway, which is the only component permitted to reach monitoring-read, which is the only component that queries Azure Monitor. Every hop is a separate Entra audience with its own app role, and the whole graph is declared in one file that generates both the directory grants and the runtime allow-lists.

Four hops, three identities, one direction

Solid lines are built and running. The dashed line is the one hop that is configured in code but has never been applied to the running console — see section 09.

graph LR
  subgraph RGC["rg-opsconsole-dev"]
    UI["Ops Console
ca-opsconsole-dev"]:::gate
    MI["Console managed identity"]:::gate
  end

  subgraph RGO["rg-oscar-dev · network isolated"]
    AFD["Front Door edge
+ WAF containment"]:::svc
    GW["Action Gateway
ca-action-gateway-dev"]:::svc
    MR["monitoring-read
ca-monitoring-read-dev"]:::svc
  end

  subgraph AZM["Azure Monitor"]
    AI["App Insights
appi-oscar-dev"]:::data
    LAW["Log Analytics
law-oscar-dev"]:::data
  end

  UI ==>|"1 acquire token"| MI
  MI -.->|"2 Bearer · aud=gateway
NOT YET APPLIED"| AFD
  AFD ==>|"3 /api/v1/read-proxy"| GW
  GW ==>|"4 Bearer · aud=monitoring-read"| MR
  MR ==>|"5 KQL"| AI
  MR ==>|"5 KQL"| LAW
  MR -.->|"6 envelope + as_of"| UI

  classDef svc fill:#1b3a4d,stroke:#5ad1b4,color:#dcf3ec,font-size:16px;
  classDef gate fill:#3a2f17,stroke:#d8a13a,color:#f4e6c8,font-size:16px;
  classDef data fill:#173049,stroke:#4aa8d8,color:#d6ecf7,font-size:16px;
        

Both repositories, merged

Shipped since this document was first written (2026-08-04 → 08-06)

The decision at that point was to take as much of the remaining work in-house as possible rather than queue it behind another team. This is what that produced.

27 merged pull requests, across two repositories — the 11 August baseline

All authored under danielchavezs. Grouped rather than listed chronologically, because the shape matters more than the sequence: an observability foundation, then the seam that consumed it, then a week of authentication work that the seam made unavoidable.

The auth & security lane — seven PRs, 2026-08-04 → 08-11

The platform accepted more than anyone thought it did

Each row was found by looking, not by review comment, and each is stated with what the code actually did rather than a category label. The right-hand column is the honest part: several of these are merged and not yet deployed, and a status page that blurs that distinction is worse than one that omits it.

Separately, a pre-existing production configuration issue was found while verifying the production side. It is unrelated to any of the above, recorded in the security dossier, and tracked on its own.

One piece of this was originally someone else's, and it is worth saying whose

The staff-identity model this work builds on is Abhijit Ramesh's. He designed the practice-membership concept, proposed it into the information model, wrote the migrations that created the identity schema and its resolver role, and implemented the original resolver in the application-api service on 2026-07-15.

With that workstream committed elsewhere for a fortnight, the team agreed on 2026-08-06 to absorb the remaining piece here rather than hold the push channel behind it. What changed is who carried the extraction, not whose design it is. Git history shows the shape plainly: four commits by him, then a 78%-similarity rename into lib/auth, then extensions on top.

Read from Azure, not from the plan — verified 6 August

Every row below was re-queried against the live subscription on 6 August 2026, not carried over from the previous revision of this page. Two rows had inverted since it was written — the console's environment and its auth state both read "not applied" here until the deployment landed, and leaving them would have understated the position rather than overstated it.

The containment, verified against the deployed rules

Six probes, run from outside Azure with no credentials. This is the check the runbook specifies, and it is the difference between “we wrote a WAF rule” and “the edge is closed.”

“Live” is per-surface, and the UI says so

This is the part most worth showing, because it is the part most systems get wrong. Flipping DATA_SOURCE=live does not make every panel live. Twelve of the console's features read App Insights export tables that monitoring-read does not serve and was never intended to serve. Under live, those keep reading the export.

The seam therefore carries two fields, not one: kind is the mode the console is running in, and readsKind is where this particular table's numbers actually came from. They are allowed to differ, and the honest thing to render is the difference.

The Bicep had never been applied. Now it has.

The console's infrastructure — Easy Auth, the Key Vault wiring, and the OSCAR variables — was fully declared in Bicep and had never been applied. The cause was structural rather than an oversight: the deploy pipeline promotes the container image and sets two variables, and no step ever ran the template. For this file, merged did not mean deployed.

PR #11 deliberately declined to paper over that by having CI apply it silently — the apply needs an app registration and a client secret. Instead it made the gap detectable, and that verification script is what now confirms the fix: it went from 3 of 7 checks failing to 11 of 11 passing.

Connecting the two systems found a real defect in one of them

Wiring the seam meant auditing what the gateway's public edge actually serves. That audit found that nine gateway routes carry no authentication dependency at all and read the tenant from a caller-supplied query parameter. Confirmed reachable from outside Azure with no credentials of any kind: audit records and event payloads, plus a self-documenting schema endpoint.

Two aggravating factors made it worse than a missing check. One route mints the gateway's own managed identity token to fetch data from a second service, so an anonymous caller borrows a service identity. And a prefix-based edge control turns out to be defeatable by a dot-segment prefix, which the audit also verified against the live endpoint rather than assuming.

All five steps are done

1. Gateway auth dependency. ✅ Every route now carries one and derives tenancy from the verified caller rather than from caller input. The route-level assertions cover complete inventories, not a sample.
2. A real issuer for the console's token. ✅ Reframed, then resolved. This step was written believing the gateway had to move off its local auth backend. It did not: the service-to-service plane is verified first and never consults that setting. The original symptom was a different misconfiguration whose real cause was being swallowed without a log line.
3. Apply the console's Bicep once. ✅ Done — section 09.
4. Reopen the read path at the edge. ✅ Already in the committed allow-list default.
5. Flip DATA_SOURCE=live. ✅ Committed and applied; the live read returns outcome: ok.

Nothing here is waiting on another team

Resolved: the export-backed panels now render

The MONITORING_EXPORT_NOT_FOUND message the twelve export-backed panels showed was not an authentication failure and had nothing to do with the live read. Those panels read a committed telemetry export from disk; .dockerignore is deny-by-default and excludes obs_data/ from the image on purpose, and nothing mounted it at runtime. They had no source and — correctly — rendered as unavailable rather than as zeroes. That refusal to substitute a placeholder is the console working as designed; the missing input was the defect.

Fixed by mounting the export from an Azure Files share, which is the remedy both .dockerignore and the reading code already prescribed. Confirmed rendering in a signed-in session. The one-line alternative — un-excluding obs_data/ so it ships inside the image — was declined: it trades a documented data-handling boundary for build convenience, and that boundary is what stops the next directory of production-shaped data from riding along unnoticed.

Ours: the first read of a cold process overruns its budget

The console allows a request 12s; the gateway allows 10s. The first live read after a cold start took 15.0s and was aborted as MONITORING_MODULE_UNREACHABLE; the next took 4.9s and succeeded. monitoring-read holds minReplicas: 1, so this is not a container cold start — it is first-call warmup, most plausibly the JWKS key fetch plus a first uncached Analytics query. It self-heals on reload, which is precisely why it needs recording: an operator's first impression is a failure banner on an otherwise working path.

Resolved: the resolver is no longer upstream, and no longer pending

The earlier version of this page listed a shared identity resolver as an upstream item belonging to another workstream. Both halves of that are now out of date: it was absorbed here on 2026-08-06 (section 06) and shipped across PRs #173, #174 and #180. It was never required for live data on this page — the live read succeeds without it, because the console authenticates as a service while the resolver serves user identity.

What it unlocks is a different capability: real-time push. Since that phrase gets read to mean several things, here is exactly what it does and does not mean.

Pull versus push, in plain terms

Today the console pulls. You load a page, the server asks OSCAR, you see a number. To see a change you reload — and the live panel caches for five minutes, so a reload may show you the same number anyway.

With push, the server sends. The browser holds a connection open; when something happens in OSCAR, a message arrives and the panel updates itself. No reload, no clicking.

The behaviour that matters most is not auto-refreshing dashboards, though. The design is named real-time escalation and safety interrupt, and that is the point: the console stops being something you check and becomes something that interrupts you. An agent run going wrong, a safety interrupt firing — you would know as it happens, not on your next visit.

What is gating it now — and none of it is code

The resolver was one of three gates and it is closed. What replaced it on the list is not more engineering: it WAS a signature, a decision, and two switches. All three have since landed — see the strikethroughs below.

1. ✅ The shared resolver. It answers "which practice's data is this person allowed to see?" Done, and it went further than the original ask — it also fixed a single-issuer assumption that would have permanently locked out every staff member signing in from the consumer tenant.
2. ✓ A human security review of the push boundary. Signed 2026-08-12 on the OSCAR side, scoped to dev. The console-side boundary is covered by DR-26: the stream is BFF-terminated (the negotiate URL is a 60-minute all-tenants receive credential and never reaches a browser), and the console independently re-imposes the closed 44-name frame projection rather than trusting the upstream one — that exact boundary regressed upstream once, when PHI rode a KEY past every value gate.
3. ✓ Ratifying the transport (DR-11, Web PubSub). Ratified 2026-08-13, with acceptance gate 3 (topic.priority.p0) explicitly carved out rather than pretended: there is no p0 producer in OSCAR at all, so that gate is not closeable by this repo. See the DR-11 detail section in the decision register.
4. ✓ Turning the user plane on. Live since 2026-08-12: ca-orchestrator-dev now runs OSCAR_AUTH_BACKEND=oidc with a real issuer set and OSCAR_AUTH_ALLOW_DEV_TOKENS=false. That is what let the push transport's own code predicate (_live_hub_auth_is_production_grade()) pass, which is why the channel could be switched on at all — the gate was never an environment name, it was the forgery capability. The console still calls as a SERVICE principal (DR-26): its Easy Auth token store is disabled and an Airiam operator holds no practice membership, so the user path is foreclosed for us regardless.