From Logs to Traces: Understanding the Three Pillars of Observability

Logs, metrics and traces are the three main kinds of telemetry used to understand a running system. Metrics tell you that something is wrong and how widespread it is. Traces show where a request spent its time and which component failed. Logs record exactly what happened inside that component. Each answers a different question, and troubleshooting is fastest when you can move between them using shared identifiers rather than searching each one separately.

This article explains what each signal is, what it costs, and where it falls short, then follows a single slow API from alert to cause. It builds on how full stack observability transforms troubleshooting, which covers why connected telemetry matters across the layers of a system.

What are the three pillars of observability?

The phrase "three pillars" describes the three signal types most observability practices are built on:

The order above is deliberate. Investigations usually start broad, with a metric that shows a change, narrow to a specific request with a trace, and finish with the detailed record in a log. Many teams learn the signals in the opposite order, because logs come first in most software, and that is part of why log searching becomes the default troubleshooting habit.

What do logs tell you, and where do they fall short?

A log is a timestamped record of an event that the code was written to report: a request received, a payment declined, a retry attempted, an exception thrown.

Logs are irreplaceable for detail. An error message, a stack trace, an input value that broke validation, or the exact configuration a service loaded at start-up will rarely appear anywhere else.

Their quality depends heavily on structure. Compare:

2026-10-14 09:12:03 ERROR Failed to get connection

{
  "timestamp": "2026-10-14T09:12:03.418Z",
  "level": "WARN",
  "service": "inventory",
  "message": "connection pool exhausted",
  "pool": "inventory-db",
  "waited_ms": 2000,
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

The first is readable by a person. The second can be filtered, aggregated and, crucially, joined to a trace by its trace_id. Structured logs with consistent field names do more for troubleshooting than any search tool.

Where logs fall short:

• They only record what someone anticipated. An unexpected failure mode may produce no useful log line.
• They show events, not trends. Ten errors could be normal or a crisis; the log does not say which without counting.
• Volume and cost scale with traffic. Teams often reduce log levels or retention to control cost, then find the detail they need was never kept.
• They are scattered. One user action can produce lines in many services, each with its own format and clock.

What do metrics tell you, and where do they fall short?

A metric is a numeric measurement recorded over time, such as request count, error rate, latency or memory use. Metrics are aggregated before storage, so a million requests can become a handful of numbers per minute.

Most systems use three basic types:

• Counters only go up, such as total requests or total errors. Rates are derived from them.
• Gauges go up and down, such as queue depth, memory in use or active connections.
• Histograms group measurements into buckets so percentiles such as p95 latency can be calculated. They show the shape of a distribution, not just the average.

Metrics are the right tool for alerting, dashboards and capacity planning because they are cheap to store and fast to query over long periods. A useful starting set for any service is rate, errors and duration (often called the RED method) for request-driven work, and utilisation, saturation and errors (the USE method) for resources such as CPU, disks and connection pools.

Where metrics fall short:

• They show that something changed, not why. A latency spike says nothing about which request or code path caused it.
• Aggregation hides individuals. An average can look healthy while a small group of users has a terrible experience. Percentiles help, but they still summarise.
• Cardinality is the hidden cost. Every unique combination of labels creates a separate time series. Labelling a metric with a user ID or request ID can multiply storage and query cost dramatically. Those values belong in traces and logs.

What do distributed traces tell you, and where do they fall short?

A distributed trace records the path of a single request as it moves through services. It is made of spans: each span represents one unit of work, such as an HTTP call, a database query or a message being processed, with a start time, a duration, a status and attributes describing it. Spans share a trace ID and point to their parent, which lets tools draw the request as a waterfall.

Traces work across services because of context propagation: each service passes the trace ID and parent span ID to the next, usually in an HTTP header. The W3C Trace Context specification standardises this as the traceparent header, so services built with different libraries can still join the same trace.

Traces are the best tool for answering "where did the time go?" in a system with many services. They also expose dependencies that nobody documented, such as a call that fans out to twenty downstream requests.

Where traces fall short:

• They need instrumentation. Automatic instrumentation covers common frameworks and libraries, but business-specific operations often need manual spans.
• Propagation breaks easily. A proxy, queue or legacy service that drops the header splits one trace into disconnected fragments.
• They are usually sampled. Recording every request is expensive, so most systems keep a percentage. Head sampling decides at the start of a request, which is cheap but may discard the rare slow request you need. Tail sampling decides after the request completes, so it can keep errors and slow requests, but it requires buffering whole traces before deciding.
• They show timing, not reasoning. A trace shows that a span waited two seconds; the log usually explains why.

How do the three signals work together? A troubleshooting walkthrough

The scenario below is illustrative, not a customer case. It describes a common failure pattern.

A retailer's product search starts to feel slow for some customers mid-morning. No one has reported an outage.

1. The metric raises the alarm. An alert fires on the p95 latency of the /search endpoint, which has moved from its usual range of around 300 ms to over 2 seconds. The error rate has also risen slightly as some requests time out. Request volume is normal, so this is not a traffic surge. Other endpoints are unaffected. The metric has narrowed the problem to one endpoint and one time window, and told us it is affecting real users.

2. The trace shows where the time goes. Opening slow traces from that window (many tools link directly from a point on the latency chart to example traces, a feature often called exemplars) shows a consistent pattern. The search API calls an inventory service to check stock. Inside the inventory span, a child span called db.pool.acquire takes most of the two seconds. The database query itself, once it runs, takes a few milliseconds. The database is not slow. Requests are waiting for a connection to it.

3. The log explains why. Filtering the inventory service's logs by the trace ID from that slow trace returns a warning: connection pool exhausted, waited_ms=2000. Looking at the surrounding lines shows the pool size the service loaded at start-up was 10, down from the 50 it previously used.

4. The change record closes the loop. A configuration change deployed that morning had replaced an environment-specific setting with a default. Restoring the pool size resolves the latency within minutes, which the p95 metric confirms.

Each step answered a question the previous signal could not:

Without the shared trace ID, step 3 would have been a search across every inventory log line in a time window, hoping to spot the relevant warning among normal traffic. Without the trace, the team might have started by investigating the database, because "search is slow" and "the database is slow" are easy to confuse.

What makes the signals correlate?

The walkthrough depended on a few practices that are easy to overlook:

• Trace IDs in logs. Logging libraries can usually add the active trace ID and span ID to every log record automatically once tracing is enabled.
• Consistent resource attributes. The same service name, environment and version on metrics, traces and logs. If one tool calls it inventory and another inventory-svc-prod, correlation becomes manual.
• Exemplars on metrics. Linking a point on a latency chart to example traces turns "something is slow" into "here is a slow request" in one click.
• Change events. Deployments and configuration changes recorded on the same timeline as the telemetry.
• Accurate clocks. Correlation by time is only as good as the time sources across hosts.

These are mostly conventions, not products. OpenTelemetry helps by defining standard attribute names and handling propagation, but teams still have to agree how services are named.

Are the three pillars still the right model?

The three pillars are a useful starting vocabulary, but the model has two weaknesses worth knowing.

First, it can encourage treating the signals as three separate products with three separate tools. The value is in the connections between them, as the walkthrough shows. Three well-run silos are still silos.

Second, it is incomplete. Other signals are increasingly common:

• Continuous profiling shows which functions consume CPU and memory in production, which is useful when a trace says a span is slow but not which code inside it.
• Events, such as deployments, feature-flag changes and incidents, provide the context that explains why behaviour changed.
• Real user monitoring captures performance as experienced in the browser or app, including time the backend never sees.

A better framing is that observability is the ability to ask new questions of a system, and the signals are raw material. Choose the signal by the question you need answered.

Which signal should you use for which question?

Where should you start?

1. Get the RED metrics right for your most important service: request rate, error rate and latency percentiles. They are the basis for useful alerts.
2. Move to structured logs with consistent field names, and add the trace ID to every record.
3. Add tracing to one important user journey end to end, using automatic instrumentation first and manual spans where business logic matters.
4. Make sure the names match across all three signals.
5. After the next incident, note which step of the investigation took longest and fix that gap first.

If your alerts are already noisy, read how to move from alert fatigue to actionable signals before adding more.

Frequently asked questions

What is the difference between logs, metrics and traces?

Metrics are aggregated numbers that show whether a system is healthy and how behaviour is trending. Traces follow individual requests across services to show where time is spent or where errors occur. Logs are detailed event records that explain what happened inside a component. They are complementary rather than interchangeable.

Do I need distributed tracing if I have good logs?

For a single service, good structured logs may be enough. Once a request crosses several services, tracing becomes valuable because it shows the whole path and timing in one view, which is very difficult to reconstruct from separate logs.

Why not just log everything?

Log volume grows with traffic and becomes expensive to store and slow to search. Metrics are far cheaper for trends and alerting, and traces are better for request timing. Logging everything also increases the risk of capturing personal data or secrets.

What is a span?

A span is one timed unit of work within a trace, such as an API call or a database query. It records a start time, duration, status, attributes and the ID of its parent span, which is how a trace is assembled into a tree.

Conclusion

Logs, metrics and traces are not competing options. They answer different questions in a predictable order: metrics detect and scope a problem, traces locate it, and logs explain it. The practical skill is moving between them quickly, which depends less on the tools than on shared trace IDs, consistent naming and structured data. Get those foundations right for one important journey, and every later investigation benefits. For a vendor-neutral way to build that foundation, see OpenTelemetry explained.