Spans and traces
The data model here is small.
A span is one timed operation, with a name, a start time, a duration, a service, and attributes like a status code or a query. A trace is the tree of every span belonging to one request, linked parent to child.
Picture the tree. The root is the request arriving at your edge, its children are the calls the edge made, and their children are the calls those services made.
Read it as a waterfall and the shape tells you the problem at a glance. Two hundred stacked database spans is a loop that should have been one query. Three identical spans in a row is a retry. Sibling spans running one after another are a chance to parallelise.
Context propagation
Pass the context along, because that is the glue. When one service calls another it forwards two identifiers: the one naming the whole request, and its own span, so the callee can record its parent.
Use the standard header format, and expect to meet a couple of older ones from specific vendors. Every span reports independently to a collector that reassembles the tree by identifier, so no service ever needs the whole picture.
Treat propagation across HTTP as the easy half. The hard half is inside your own process and across asynchronous hops.
Your context has to survive handoffs between thread pools, await boundaries, and job queues, and any hop that drops it splits your trace into orphaned fragments. Publishing onto a queue means stuffing the context into the message headers so the consumer can continue the trace.
Lean on automatic instrumentation, which handles most of this for mainstream frameworks, HTTP clients and database drivers. What is left is the unglamorous part: auditing the custom queue, the legacy service and the hand-rolled thread pool where context silently falls on the floor.
Worked example
Sanjay's team at a ticketing company rolls out OpenTelemetry auto-instrumentation across nine Java and Node services in a two-week push. Eight services trace cleanly on day one. The ninth, an order fulfillment worker consuming from RabbitMQ, produces orphaned traces: the consumer's spans start new trace IDs because nobody propagates context through the queue. One engineer spends an afternoon injecting traceparent into message headers on publish and extracting it on consume. The first complete trace through the async boundary pays for the whole project: it shows orders spending a median 41 seconds sitting in the queue before the worker even starts, dwarfing the 800 ms of actual processing. The team had spent the previous sprint optimizing the 800 ms. They add two consumers and order confirmation latency drops from 45 seconds to 4.