Case study overview

Enterprise Application Observability: A Pragmatic Approach with OpenTelemetry

On-call rotations in enterprise systems tend to reveal the same failure mode eventually: the team finds out about a problem from someone outside the team. A support ticket, a customer email, sometimes a compliance inquiry, all pointing at something that had already been broken for a while. In systems that touch regulated data, that lag matters more than it does elsewhere, because the question isn't just "how fast did we fix it" but "how long were we blind to it."

That's the pattern behind this piece. Not a single outage, but a slow accumulation of near-misses, each one traceable back to the same root complaint: nobody could see where a request actually went once it left the front door. The application had grown past the point where a single log file or a developer tailing output on a VM told you anything useful. Requests crossed several services, some owned by different teams, and by the time an issue surfaced downstream, the actual point of failure was three hops back and long since rotated out of any log retention window.

The business risk wasn't abstract. In a regulated environment, "we didn't know" is a worse answer than "we knew and it took time to fix." Observability here wasn't a platform maturity item to get to eventually. It was the difference between catching a data-handling problem internally and having a regulator or a customer catch it first.

Technical Problem

The technical version of that problem is narrower than the business one, and it's worth separating the two. The business problem was "we find out too late." The technical problem was "we have no way to correlate a single request across service boundaries once it's in flight."

Each service logged locally, well enough for its own operator to debug in isolation, but there was no shared identifier tying a request in the intake layer to the work it triggered three services downstream. Every incident investigation started the same way: someone grepping timestamps across four or five log sources, hoping the clocks were close enough to line things up. That's not really debugging. That's archaeology.

If there's a North Star metric for this kind of problem, it isn't latency or throughput, it's trace continuity: what fraction of a request's actual path through the system you can reconstruct after the fact, without guessing. I don't have a number to put on that for this system, and I'd be suspicious of anyone who threw one out without having measured it first. What I can say is that before instrumentation, the honest answer was "however far the logs happen to line up," which isn't something anyone should be comfortable operating on.

Requirements

The requirements split cleanly into what the system had to do and what it was never allowed to do.

Functional:

• Correlate a single request across every service it touches, including async hops through queues, not just synchronous HTTP calls.
• Emit structured, queryable telemetry rather than free-text logs.
• Support alerting on the handful of signals that actually predict a bad outcome, not everything the system could technically report.

Non-functional:

• No PHI or PII in any span, log line, or trace attribute, under any circumstance, including debug-level detail.
• Low enough overhead that instrumentation itself never becomes the performance story.
• A rollout that doesn't require every team to re-architect service boundaries or adopt one vendor's agent on the same timeline.

That last one turned out to matter more than it looked like on paper. A requirement that assumes synchronized adoption across teams with different priorities is a requirement that will quietly fail.

Constraints

Two constraints shaped almost every decision after this point.

The first was regulatory. Telemetry pipelines are an easy place to accidentally leak sensitive data, because nobody thinks of a trace attribute or a log line as "data" the way they think of a database column. A request ID is safe. A request ID with a patient name attached as a debug convenience is not. Any instrumentation strategy had to assume engineers would, with good intentions, occasionally include something they shouldn't, and build for that instead of writing a policy about it and hoping.

The second constraint was organizational, not technical: this wasn't a clean-sheet system. Services were owned by different teams, at different levels of maturity, with different appetites for adding new dependencies. Anything that required a coordinated, simultaneous rollout across all of them wasn't a real option, whatever it looked like on a slide.

Architecture Overview

The shape that held up under those constraints was standard OpenTelemetry SDKs in each service, exporting to a self-hosted OpenTelemetry Collector, which forwarded to Azure Monitor and Application Insights as the backend.

The system in scope looked roughly like this: a public-facing intake gateway (call it PatientIntakeGateway) accepting requests and issuing them into the platform, two core services handling eligibility checks and claims processing (EligibilityService, ClaimsProcessor), and an async NotificationWorker consuming events off a queue rather than being called directly. Not a monolith, not a full microservices mesh either. More a core system with satellite services bolted on at different points in its life.

PatientIntakeGateway ──┐
                        ├─> EligibilityService ───┐
                        │                          ├─> Collector ──> Azure Monitor / App Insights
                        └─> ClaimsProcessor ───────┤
                                                    │
                           (queue) ──> NotificationWorker ──┘

Each service ran the OTel SDK, propagating trace context through HTTP headers on synchronous calls and through message attributes on the queue hop, since that boundary is the easiest one to lose context across and the easiest to forget about until an incident makes it obvious. Every service exported to a local Collector agent rather than talking to Azure Monitor directly. That Collector layer does more than forward data: it's a batching point, a scrubbing point, and, not incidentally, the one place in the whole pipeline where a policy about what telemetry is allowed to leave the building can actually be enforced in code instead of in a wiki page.

The backend choice, Azure Monitor over a dedicated vendor APM suite, mattered less architecturally than it might seem. The SDK and Collector layer meant the backend was closer to a pluggable detail than a foundational decision, which is exactly the property worth designing for.

Key Decisions (chosen vs rejected)

The decision that mattered most here wasn't which backend to send data to. It was where to put the boundary between "instrumented application code" and "everything downstream of that."

Two options were realistically on the table. One: instrument each service directly against the Application Insights SDK, since that's what the backend already was, and skip the extra hop. Two: instrument against the vendor-neutral OpenTelemetry SDK and route everything through a self-hosted Collector before it ever reaches Application Insights.

The first option is less work up front, and it's easy to see why it's the default choice on a lot of teams. It's fewer moving parts, one less thing to operate, and it gets a service reporting data on day one. In a system without the compliance constraint, I'd lean that way more often than people expect, because an extra hop is an extra thing that can break, and simplicity has real value.

What ruled it out here was the redaction requirement. If a service talks directly to the vendor SDK, there's no interception point between "an engineer added a field to a log statement" and "that field is sitting in a third-party product." You're relying entirely on code review and developer discipline to keep sensitive data out, forever, across every team, every pull request. That's not a strategy. That's a hope.

Routing through a self-hosted Collector puts a real boundary in the pipeline. Processors on the Collector can strip or hash known-sensitive attribute keys before anything leaves the environment, which means the safety net doesn't depend on every engineer remembering the rule every time. It's defense in depth rather than a single point of trust, and in a regulated environment that trade is usually worth making even though it costs more to run.

The secondary decision, closely related, was where redaction logic should live: application code, the Collector, or both. Doing it only in application code means every team has to reimplement and maintain the same logic. Doing it only at the Collector means one missed rule in one place is a system-wide gap. The pattern that tends to hold up is both: lightweight guardrails in application code as a first pass, with the Collector as the enforcement point that doesn't rely on anyone getting the first pass right.

Vendor APM suites, Datadog and New Relic among them, were also considered and set aside, mostly on data-residency grounds and because there was already meaningful investment in the Azure ecosystem. That wasn't a technical rejection so much as a "why introduce a new vendor relationship to solve a problem the existing platform can solve" one.

Trade-offs

None of that comes free, and it's worth being specific about what it costs instead of presenting the Collector approach as strictly better.

Running a self-hosted Collector means there's now a piece of infrastructure whose entire job is watching the rest of the system, and that piece needs its own care: patching, scaling, monitoring, someone paged when it falls behind. Observability tooling has a way of becoming invisible right up until it's the thing that's down, at which point it's suddenly the most important service in the building and nobody remembers who owns it.

There's a real latency cost too, small but not zero. Every span makes an extra hop before it's durable, and under load, that hop can become a queue. I don't have a precise number for the overhead here, and stating one without having measured it under this system's actual traffic pattern would be worse than saying nothing.

The bigger cost is organizational, not technical. Standardizing on the OpenTelemetry SDK instead of the vendor SDK means every team learns a slightly different instrumentation API than the one the vendor's own documentation walks them through. That's a real adoption tax. It shows up as friction in code review and slower onboarding, not as a line item anyone budgets for up front.

And redaction logic is never finished. Data models change, new fields get added, and a scrubbing rule written against last year's schema quietly stops covering this year's. Treating the Collector as a one-time setup rather than an ongoing responsibility is the easiest way to end up back where this project started, just with more infrastructure between you and the problem.

Failure & Resilience

The failure mode that has to be designed for deliberately is what happens when the Collector itself is down or backed up.

The instinct is to make sure nothing gets lost. In practice, the right call is closer to the opposite. Telemetry export should be non-blocking and fail open, meaning if the Collector can't accept spans fast enough, the application drops them rather than waiting. A monitoring system that can take down the thing it's supposed to be observing has its priorities backwards. I've seen teams learn this the hard way, usually by configuring a synchronous exporter with no timeout, and then watching an observability outage turn into an application outage.

That means accepting real trace loss during a Collector outage as a designed trade-off, not an accident. It's an uncomfortable thing to write down explicitly, because it sounds like planning to lose data. It is, on purpose, in a narrow set of conditions, because the alternative is worse.

Redaction failure is the other case worth naming. If the Collector's scrubbing logic has a bug or a gap, the failure is silent unless something else is checking for it. That's the argument for the belt-and-suspenders approach from the decisions section: a second, lighter check at the application layer isn't redundant engineering. It's the difference between a redaction gap caught in testing and one found out about from a compliance audit.

Operational Considerations

The clearest operational shift is in how an incident starts. Before, it started with an escalation from outside the team. After, it starts with an alert built on an actual signal: a trace showing where a request stalled or failed, correlated across the services it touched. That's a different job. Triage becomes "here's what broke and where" instead of "something's wrong somewhere, go find it."

Cost is the part I can only speak to qualitatively here, since there aren't real ingest numbers for this system to point to. What I can say in general is that observability cost is driven far more by cardinality and sampling decisions than by traffic volume itself. A service that adds a high-cardinality attribute, a raw request ID as a tag instead of an indexed field, for instance, can multiply ingest cost without a single additional request happening. That's usually where an observability budget quietly blows past what anyone expected, and it has nothing to do with how much traffic the system is actually handling.

There's also a new ownership question that didn't exist before: who owns the Collector configuration as the system evolves. It's not purely an application team's job, and it's not purely a platform team's job either, and systems shaped like this tend to let it fall into a gap unless someone is explicitly on the hook for it. Collector version upgrades deserve the same caution as any other piece of shared infrastructure: staged rollout, not a Friday afternoon deploy.

Lessons Learned

If I were starting this over, I'd resist the urge to instrument everything at once. The natural instinct once the tooling is in place is to wire up every service immediately, because more visibility feels like it can only help. What actually happens is a flood of data with no way yet to tell signal from noise, and redaction rules that haven't been tested against real traffic patterns yet either. Starting at the boundaries, the intake gateway and the points where trust actually changes hands, and working inward gives you something usable much faster than trying to light up the whole system on day one.

I'd also push harder, earlier, on treating the Collector configuration as a real, owned piece of the platform rather than something set up once during rollout and then left alone. The redaction rules in particular need a review cadence tied to schema changes, not just a one-time write and trust. The assumption going in is usually that the hard part is the initial setup. The hard part is the maintenance, and it's less visible, which is exactly why it gets skipped.

The last thing worth naming is a mindset shift more than a technical one. Observability isn't a feature you ship and move on from. It's closer to a second system running alongside the one you're actually building, with its own failure modes, its own cost profile, and its own need for an owner. Teams that treat it as a one-time platform initiative tend to end up back at the same blind spots a year or two later, just with more dashboards nobody trusts.

ADR Summary

Context: A partially decomposed enterprise application, handling regulated data, had no way to correlate a request across service boundaries. Incidents were routinely discovered externally rather than internally, and existing logs couldn't be reliably correlated after the fact.

Decision: Instrument all services with the OpenTelemetry SDK rather than a vendor-specific agent. Route all telemetry through a self-hosted OpenTelemetry Collector, which handles batching, redaction, and export to Azure Monitor / Application Insights. Telemetry export is configured to fail open under Collector backpressure rather than block the application.

Status: Adopted as the standard pattern for new and migrated services in this environment.

Consequences:

• Positive: request-level trace correlation across service and queue boundaries; a real enforcement point for data-handling policy instead of a documentation-only rule; the backend became a swappable detail rather than a foundational dependency.
• Negative: the Collector is now critical shared infrastructure requiring its own operational ownership; an added latency hop, unmeasured but nonzero, under load; adoption friction from a non-vendor-native SDK; redaction rules require ongoing maintenance as data models change, not a one-time setup.
• Accepted risk: telemetry loss during Collector outages, by design, in exchange for the application never blocking on the monitoring path.