If I Had to Build an Observability Platform from Scratch Today
If you work in technology long enough, you eventually reach the same conclusion: every system becomes more complicated than you intended, faster than you expected, and in ways you didn’t predict. That’s why observability matters. It’s not about collecting data. It’s about keeping the system understandable as it grows. Does your database cluster depend on an NFS share on a server completely unrelated? Are you investigating the Software Bill of Materials (SBOM) by doing dual build pipelines and comparing hashes?
I’ve built observability stacks for home labs, enterprise environments, SaaS-connected platforms, and hybrid infrastructures. I’ve built them well, and I’ve built them poorly. I’ve built them under pressure, under constraints, and under the weight of legacy decisions I didn’t make. I’ve used Commercial-Off-The-Shelf (COTS) software, and I’ve used Open-Source tools. Hell, I’ve even used a mix when appropriate.
So, if I had to build an observability platform from scratch today by starting fresh, with no baggage, and no sunk costs here’s exactly how I’d do it.
Note: I’ll be using software names for clarity, but what I’m really talking about are the features they provide.
And yes, this includes something I didn’t fully appreciate until recently: observability is meaningless if the underlying system isn’t resilient. After all, your observability data is (or should be) a record of what’s happening iwithn your infrastructure. In my opinion, it’s just as important as a Security Information and Event Management (SIEM) tool, if leveraged differently. But more on that later.
Questions first, tooling later
Most observability platforms fail before they’re even deployed because they start with the wrong question:
“What tools should we use?”
That’s backwards. The right question is:
“What do we need to understand?”
If you don’t know what questions you’re trying to answer, you’ll end up with a pile of disconnected tools, a mountain of metrics, and dashboards that look impressive but tell you nothing. Ask me how I know…
So I start with three foundational questions:
- What does “normal” look like and how do I confirm that?
- How do we know when something is wrong and who do we tell?
- How do we know what to do next?
If your observability platform can answer those three questions clearly, you’re already ahead of most organizations.
The Core Pillars (And Why They Matter)
If I were building from scratch, I’d structure the platform around four pillars. Not the usual “metrics, logs, traces” list—those are data types, not outcomes. They are the building blocks that make the platform, but not the platform.
The pillars I care about are:
1. Understanding
Can a human look at the system and understand what’s happening without needing tribal knowledge?
2. Correlation
Can we connect events across systems, layers, and time without guesswork?
3. Narrative
Does the platform tell a coherent story about the system’s behavior?
4. Resilience
Can we trust the data when things go wrong? Can we recover affected systems when things go really wrong?
That last one is where backup and cyber-resilience come in. Observability without resilience is just a very fancy way of watching your system fail. Fun (if you like to watch the world burn), but ultimately useless.
The Architecture I’d Choose Today
If I were starting fresh, here’s the architecture I’d build—not because it’s trendy, but because it’s understandable, maintainable, and resilient.
Collection Layer
- Zero agents or Lightweight agents
- OpenTelemetry everywhere it’s supported or could be implemented
- Minimal custom instrumentation (if you need instrumentation or enrichment, add it as a separate layer “above” the raw data)
- No proprietary lock-in
The goal here is consistency. If every service emits data differently, you’ve already lost.
Storage Layer
- Logs: Loki, Elasticsearch, or similar
- Metrics: Prometheus, Mimir, or similar
- Traces: Tempo, Jaeger, or similar
- Events: A simple, durable queue (Kafka, NATS, or cloud equivalent)
The specific tools matter less than the principle: store data in systems designed for that data type.
Correlation Layer
This is the glue. This is where observability becomes understanding.
I’d use:
- OpenTelemetry semantic conventions
- Consistent service naming
- Shared trace IDs across logs, metrics, and events
- A unified metadata model
If your correlation layer is weak, your observability platform becomes a scavenger hunt. This may be the most difficult layer – even in a conceptual system – because everyone has their preferences. Naming scheme discussions can get… … … passionate. Just trust me here. Stick to your guns and make sure that what you’re proposing is logical and flexible.
Visualization Layer
Here I’m going to dispense with the foreplay and just say, “Grafana,” without hesitation.
Not because it’s perfect, but because:
- it’s flexible,
- it’s fast,
- it’s widely supported,
- and it forces you to think about dashboards intentionally.
And as we covered in the last post: dashboards are narratives, not decoration. Thankfully, Grafana has an excellent community, is well documented (remember when we mentioned documentation), and gradually becomes clearer the more you work with it. Unfortunately, it will involve a learning curve, but it’s not onerous and is still far and away the best choice for visualization. That’s why it’s still what I’d choose.
Resilience Layer
This is the part most observability discussions ignore – I certainly know I did when I got into the business.
If I were building today, I’d integrate:
- immutable backups of observability data,
- rapid restore paths,
- isolated storage domains,
- and cyber-resilience controls that ensure the observability platform itself can’t be tampered with.
Because if your observability data can be encrypted, corrupted, or erased during an incident, you lose the ability to understand what happened. And if you can’t understand what happened, you can’t recover intelligently.
The beauty of this is most (if not all) telemetry data is perfect for WORM (Write-Once/Read-Many). All the observability data should be timestamped and immutable. Yes, you can have summarization layers on top (think SQL Views as opposed to SQL Tables), but the underlying data doesn’t ever need to change. Maybe at some point you’ll need to archive old data, but you can make sure that’s archived cleanly (and safely) as well. Also, it’s been forever since I’ve spoken about WORM storage and it’s nice to brush off that term again.
Protecting observability data shouldn’t be an afterthought – especially in this world of potential cyber-attacks. If a system can infiltrate your observability data and remove events correlating to the breach, then do you even have a record?
What this Theoretical Platform Gives You
If you build an observability platform this way, you get:
- A clear definition of normal for your environment
- A fast path to identifying anomalies
- A narrative that explains why
- A resilient foundation that ensures the story is trustworthy
- A system that grows with you instead of fighting you
- A self-documenting ecosystem (if expanded systematically)
You also get something more important: confidence.
Confidence that you can understand your system. Confidence that you can respond to incidents. Confidence that you can recover. Confidence that your data is telling the truth.
Observability isn’t about charts. It’s about clarity, telling the story, understanding the whole system while simultaneously being able to interrogate the individual data points.
Afterall, clarity is what keeps systems alive.