What I’ve Learned from Building My Own Stack

When you build your own stack, interoperability becomes the quiet, persistent problem you didn’t budget time for. Different services, vendors, and endpoints each speak their own telemetry dialect. That makes it hard to answer the single most important question when something goes wrong: what do I need to look at to understand the problem?

This post is about a simple discipline that saved me countless hours: start with the question you want answered, then pick the signals that answer it, then wire up monitoring, and finally tune alerts. I’ll walk through how to map questions to metrics/logs/traces, common interoperability pitfalls, and practical checklists you can use today.

Start with the question

If you don’t know the question, every dashboard looks relevant and every alert looks noisy. Be explicit.

Example questions:

• Why is the web app slow for users?
• Is the database the bottleneck for checkout latency?
• Are third-party APIs causing tail latency spikes?
• Did the latest deployment increase error rates?

Each question implies a different set of signals. Write the question down and keep it visible while you instrument.

Define the facets you might need to understand

Different problems live in different facets of the system. Pick the facet(s) that match your question.  Be conscious of elements in your application stack that you don’t “own” and figure out ways to instrument data from them.

• User experience: p50/p95/p99 latency, page load time, error rate, Apdex.
• Infrastructure: CPU, memory, disk I/O, network throughput, socket counts.
• Application internals: DB query latency, cache hit/miss rates, queue lengths.
• Network: RTT, packet loss, DNS resolution times, CDN edge metrics.
• External dependencies: third-party API latency, error rates, rate-limit responses.
• Deployment/CI: rollout timestamps, feature flags, config changes.

Map questions to signals

Think of metrics as trend indicators, logs as context, and traces as causal paths.

Your mental mapping should go from the question → metrics → logs → traces.  You might find the answer at any level.  Just remember the question you are trying to answer.

• Why is the web app slow?
  • Metrics: p95/p99 latency by region; CDN edge latency; backend request latency; error rate.
  • Logs: client IPs, geo tags, CDN cache status, backend request IDs.
  • Traces: end-to-end trace showing where time is spent (edge → app → DB → external API).
• Is the DB the bottleneck?
  • Metrics: Database Management System CPU, active connections, slow query count, average query latency.
  • Logs: slow query logs, deadlock traces, connection errors.
  • Traces: spans showing DB query durations per request.
• Are third-party APIs causing tail latency?
  • Metrics: external API latency histogram, error rate, retry counts.
  • Logs: request/response headers, status codes, payload sizes.
  • Traces: spans for external calls with timing and error annotations.

Use metrics to detect and quantify, logs to investigate and add context, and traces to follow the causal chain.

Interoperability pitfalls I’ve run into

The OpenTelemetry standard was introduced to minimize or reduce translation problems.  When it’s been fully embraced, it’s great, but sometimes when telemetry crosses platform boundaries, expect friction.

• Naming and unit mismatches: One system reports data size as bytes, while another uses KB – or was it Kb?  The 1,000 vs. 1,024 multipliers can sneak in all kinds of places.  Normalization makes sure dashboards and metrics are factually accurate.
• Sampling and aggregation differences: Traces may be sampled at different rates across services, hiding tail behavior.  When troubleshooting, the most important parts of a trace are the start and the end.  We need to make sure they all come together over a span.
• Clock skew: If hosts aren’t time-synced, correlating logs and traces becomes guesswork.  Pick a time zone (UTC is a favorite) and let the dashboards adjust according to your local time.
• Missing correlation IDs: If requests don’t carry a propagated ID, stitching logs and traces across services is painful where it’s even possible.  Think about unique identifiers and plan them in advance.  Don’t be afraid to revisit this as your needs expand.
• Vendor formats and rate limits: Different vendors expose different fields and may throttle telemetry ingestion.  Be sure to understand what data you are getting, how recent it is, and how to determine if it’s stale.
• Cost constraints: High-cardinality metrics and full-trace retention are expensive; you need a strategy for ingestion, storage, archival, and recovery.

Practical steps to make telemetry interoperable

My experience in this space started with SNMP and WMI and has expanded over the years.  There is no one way to figure out how endpoint telemetry will work for you, but there are a few things I try to do whenever I’m working with any observability solution.

1. Inventory endpoints and telemetry. List every service, storage device, network interface, database system, and third-party API and note what telemetry they expose.  This is a great time to start that documentation we’ve discussed in the past.
2. Normalize names and units. Create a small mapping layer or use a telemetry pipeline to rename and convert units.  Some systems will do this automagically, but not all.  When in doubt collect and test.  Be especially aware that 1TB != 1Tb and that can inject a growing percentage delta. (2.4% for K, 4.8% for M, 7.3% for G, 9.9% for T)
3. Propagate correlation IDs. Generate a request ID at the edge and pass it through headers to every downstream service.  This has gotten easier over time thanks to the proliferation of OpenTelemetry, but it does add a small margin of overhead.  Hopefully it’s already been put into place by the developers and it’s no worry.  Regardless, on highly transactional systems, be sure to test to avoid a potential bottleneck.
4. Ensure time sync. NTP everywhere; logs and metrics must share a common clock.  Don’t exclusively rely on NIST, Microsoft, Apple, or another for accurate time.  Whenever possible point to a local endpoint running NTP services.  Active Directory Domain Controllers are an excellent option if you run services locally.
5. Decide sampling strategy. Be wise when it comes to sampling.  I could spend hours talking about how NetFlow is good, but sFlow is better when talking local communications.  Suffice it to say, I’ve learned my lessons. Ramp up sampling for traces when actively working incidents and revert to collect fewer during normal times.
6. Choose retention and aggregation windows. Keep high-resolution data for short windows and aggregated summaries for longer windows.  Your needs will define these thresholds.  If you don’t need detailed data after 14 days, summarize it, so you don’t have millions of unnecessary data records.
7. Build dashboards per question. Your dashboards should answer your original questions, not show off the edge cases your solution supports.

Alerts: from noise to action

Alerts should be actionable and tied to the question.  If it doesn’t require a human, it’s a log entry.  If it doesn’t require a human and we won’t care about it for a while, it’s in a report.  Alerts are different and should be treated so.

• Start with the question. If the question is “Is checkout broken?”, an alert should reflect that (e.g., checkout success rate < 99% for 5 minutes).
• Use multi-signal alerts. Combine signals to reduce false positives.  A high p95 latency + increased error rate + Database CPU spike is probably a problem, but any single one of these alone is a red herring.
• Avoid single-metric noise. A CPU spike alone rarely requires alerting.  If a CPU in the forest goes hot, but no one is affected, is it an alert?
• Set escalation and runbooks. You probably already have runbooks.  They may not be written down yet, but there’s always tribal knowledge.  Do not rely on that.  Instead, take the time to write them up in some environment and link to them from alerts.  At minimum, the runbooks should answer “what to check first” and “who to call.”
• Tune thresholds and windows. It’s ok to start with your best guess and adapt as you go.  In fact, that’s the best strategy.  Alter them as you learn more about how your systems operate together.

Example: Users see slow page loads

1. Question: Why are web app users experiencing slow page loads since 10:00 UTC?
2. Signals to watch: p95 page load time by region; CDN cache hit ratio; backend p95 latency; database slow queries; external API latency.
3. Monitoring setup: dashboard with region-filtered p95, CDN edge metrics, backend traces sampled at 100% for affected endpoints, and slow-query logs enabled.
4. Investigation flow: check region p95 → check CDN cache status → open a trace for a slow request → inspect DB spans and external calls → correlate with deployment logs.
5. Alerting: page-load p95 for web app > 2s for 5 minutes AND CDN cache hit ratio < 80% → page on-call.

Final thought

Interoperability isn’t a one-time project; it’s a practice. If you start every incident by asking the right question, the rest becomes a sequence: pick the signals that answer it, make sure those signals are comparable across platforms, monitor them, and alert only when the question demands action. That discipline turns noisy telemetry into a reliable source of truth.