One instrumentation standard
The instrumentation question has a settled answer now: OpenTelemetry.
It came from merging two earlier projects in 2019. You get vendor-neutral libraries, automatic instrumentation for common frameworks, and a collector that ships your data to whichever backend you choose.
Instrument once and switching backends later becomes a configuration change rather than a rewrite. That neutrality is why writing directly against one vendor's library today is a mistake.
Choose a backend from two camps: the self-hosted open-source ones, and the paid platforms that bundle storage, search and analytics.
Look past the waterfall view when you compare them, because the querying is the real differentiator. Finding every trace where checkout exceeded two seconds and touched the promotions service, grouped by cart size, is where the paid tools earn their invoice.
Reading a trace
Read traces by pattern recognition, and a short list covers most of the wins. A staircase of sequential sibling spans sharing no data dependency is parallelism waiting to happen.
Dozens of identical short database spans under one parent is a loop that should be one query. Repeated identical spans with gaps between them are retries. A span that ends in a timeout while its child keeps running means work continued after the caller gave up.
Watch for wide gaps between a parent and its first child, which are queue time or waits on a connection pool, and invisible to every other tool you own.
Adopt the practice that multiplies all three signals. Stamp the trace identifier onto every log line, and attach example traces to your latency metrics.
See what that gives you. A latency spike on a dashboard links to example traces, and each trace links to its logs. Metric to trace to log in three clicks is what people mean when they say observability rather than monitoring.
Worked example
Camila joins a fintech as a senior engineer and inherits a loan application API whose slowest one request in twenty, its p95, takes 6 seconds that previous efforts had failed to crack. The service already ships traces to Grafana Tempo, so she skips profiling and queries for traces over 5 seconds. The waterfall shows a tidy staircase: credit bureau call, 1.8 seconds, then income verification, 1.5 seconds, then fraud scoring, 1.9 seconds, each waiting for the previous despite none consuming another's output. A two-day change fires all three concurrently and joins the results. p95 drops from 6 seconds to 2.3, bounded by the slowest bureau. Her writeup notes that the fix required no optimization of any service, just seeing the call graph, and the follow-up ticket adds a CI check flagging sequential spans with no data dependency.