OSCAR's observability plane is a swappable, in-process fabric over OpenTelemetry that captures metrics, traces and cost from every service — and redacts at a single explicit boundary so telemetry carries no PHI by construction. This page shows the target architecture: where each part lives, how signal moves across the platform, and what is built vs. proposed.
A span or metric is born inside a service, passes through the shared provider, and hits the single redacting exporter — the one gate where PHI and health-check noise are dropped. Only clean, low-cardinality signal reaches the Azure backend, gets aggregated once, and is served to the console over a read-only contract. Watch a packet make the trip.
The gate is the load-bearing invariant: a single explicit exporter with an
allow-list redactor and a health-span filter. There is no OTEL_EXPORTER_OTLP_* env-bridge —
that would spawn a second, unfiltered egress and leak /health spans. One filtered door out,
by construction.
Durable governance truth and high-volume sampled telemetry are kept deliberately separate — but
every record in both planes carries correlation_id and practice_id, so a cost
spike can always be joined back to the governed action that caused it.
The durable, ordered, replayable source of truth.
EventEnvelope with a validate_no_phi() gateMetrics · traces · cost, behind a swappable OTel provider.
Most of Plane 2 is not a new server. It is one shared library imported in-process, a managed
Azure backend, and in-place maturation of existing services — plus two small new deployables
(ca-monitoring-read + the prober, merged in Increment 2).
The narrow interface (start_span, record_event, …), the redactor, the single
exporter, propagation + structured logging. Feature code never imports the backend.
Core (Inc 1): action-gateway · agent-orchestrator · delivery-service · outbox-relay · sentinel · thread-intelligence. Extended (Inc 2): rcm · scheduling · clinical · webapp · system-console. Each imports the lib + one middleware; chokepoints get spans.
The one filtered door out: allow-list redaction (only IDs / enums / durations / codes / sizes /
counts) + /health span drop. No OTLP env-bridge.
Workspace-based Application Insights wired to law-oscar-{env}, in-boundary under the
existing Azure BAA. Diagnostic settings fan SQL/Redis/SB/KV into the workspace. Provisioned + live in dev.
The one genuinely-new service. Fans in both planes + health + prober, computes the three query classes, and serves the OSCAR-owned read contract to the console. Merged to dev, inert pending Promotion-2 activation.
Replaces the ephemeral ca-netprobe: a standing worker + native App Insights availability
tests — the synthetic / secondary tier for the black-box DICOM services and external-dependency
reachability. Merged to dev, inert pending Promotion-2 activation.
Scales the orchestrator's live agentic-trace stream off the single in-process /ws hub onto
the provisioned wps-oscar-{env} for cross-replica console escalation (poll→push). Every
pushed frame goes through a closed, versioned no-PHI projection (DR-12) — the load-bearing
deliverable. p0 safety-interrupt designed; gated. Inert by default (LIVE_HUB_BACKEND=local)
— activation is Promotion 3.
W3C traceparent is injected at every hop; the existing correlation_id stays the
durable business join key and rides as a span attribute / baggage.
Injected in the SDK transport and the read-proxy service-to-service headers,
alongside X-Correlation-Id.
Carried in message application_properties (producer-side landed);
consumers bind the context back via a shared extraction helper.
OSCAR owns aggregation; the console is a thin renderer. Three GET query classes, every datum stamped with a provenance / freshness envelope. Route shapes are illustrative pending DPIM.
Redaction happens at source, twice:
Plane 1's validate_no_phi() refuses to construct a PHI-bearing envelope, and Plane 2's
exporter enforces a default-deny allow-list before anything leaves the process. The console inherits a
contract that carries none.
Delivered one PR per workstream onto feat/observability, then promoted to dev
per increment — never one big-bang merge. Everything defaults to MONITORING_MODE=off,
so partial landings are inert.
✅ Increment 1 activated in dev — 2026-07-14. App Insights
(appi-oscar-dev) provisioned, the connection-string seeded, and all six first-party
services flipped to MONITORING_MODE=otel with a resolving Key Vault
secretRef. Live telemetry — request spans, W3C traceparent continuity,
/health filtered, source-redaction (no PHI), and RED metrics — confirmed via KQL.
Evidence: docs/DC/OBS-PROMOTION-1-dev-activation-evidence.md (Linear CAN-108).
▸ Increment 2 merged to dev — 2026-07-15 (PR #114), review-hardened, still inert.
Adds the read-contract service (ca-monitoring-read), the standing synthetic prober
(ca-monitoring-prober), module/tool-app instrumentation (rcm · scheduling · clinical · webapp ·
system-console), and causation through the governed-write path. Activation is Promotion 2 — a
separate gated deploy that turns it on and live-validates it (Linear CAN-154). Module-app telemetry
lights up on the dev deploy; the two new services need the activation step.
▸ Increment 3 (OBS-7) — PR into feat/observability, the last workstream. Lands inert.
Scales the orchestrator live agentic-trace stream onto Web PubSub for cross-replica console escalation
(the LiveHubBackend seam), proves no PHI at the push boundary via a closed, versioned
egress projection (DR-12, the core deliverable — also closes the pre-existing /ws tee
exposure), adds SLO/error-budget math + the retuned ingestion-volume & SLO-burn alerts, and formalizes
the console push-handoff consumer contract. The p0 safety-interrupt is designed, gated
(DR-08 v2.1 / DPIM amendment). Activation is Promotion 3 — a separate gated deploy;
the no-PHI push boundary + client-subscription auth need human security review before switch-on.
Feature code touches only the narrow interface; the backend is selected by config and swappable. The default is a no-op, so local and test runs emit nothing.
# lib/observability — feature code never imports OTel or the Azure exporter from lib.observability import get_obs_provider, ATTR obs = get_obs_provider() with obs.start_span("governed_write") as span: span.set(ATTR.PRACTICE_ID, practice_id) # keys are lib-owned constants, span.set(ATTR.ACTION_CLASS, action_class) # never inline strings (master §5.4) # selected by env — off by default MONITORING_MODE=off | otel APPLICATIONINSIGHTS_CONNECTION_STRING=… # delivered by OBS-4 via Key Vault
Attribute key names live in one attributes module — so adding
or renaming an attribute fleet-wide is a 1–2 file change in the lib, not an edit across every service.
Instrument the chokepoints first, then let per-service RED middleware cover the rest. Each of the nine console health categories maps to a concrete OSCAR source:
| Category | OSCAR source | Emits |
|---|---|---|
| Availability | /health/ready + prober | uptime · up/down · incidents |
| Reliability | Plane-1 events + DLQ | events/24h · MTTR · dead-letters |
| Performance | OTel traces/metrics | p50/p95/p99 latency (RED) |
| Load | SB depth + outbox | queue depth · oldest-item age |
| Resource | Azure metrics + gauges | CPU/mem · pool · LLM concurrency |
| Token / Cost | orchestrator llm_call | gen_ai.* tokens · cents · model |
| Agentic | governed_write metrics + PRS | verdicts · approval queue · blocks |
| Security | scanner feed | vuln counts by severity |
| Administration | governance + WORM | config-change rate · audit stream |
Chokepoints: the Action Gateway governed-write path (folds in the existing
Prometheus metrics), the orchestrator turn → tool_chain → llm_call span tree (adds
gen_ai.* cost/latency, never content), and per-service request middleware for RED.