Learning-first

The usual advice for building event streaming into a system is short: don’t. Use Kafka. Use Redpanda. Use NATS. Use a managed service. The problem is solved; the incumbents are good; the operational literature is deep. Building your own broker is, by every reasonable metric, a bad idea.

We built Holocron anyway. Not because we thought we could beat Kafka — we can’t, and we’re not trying to — but because “use Kafka” answers a different question than the one we were asking.

The question we were actually asking

The industry question is what should I run in production? The answer is Kafka. That answer is correct and boring and we agree with it.

The question we were asking is what would I have to understand to build the thing that ends up being Kafka? That’s a different question, and “use Kafka” is not an answer to it. You can operate Kafka for years and never once be forced to reason about the segmented log-structured storage layer, the leader-election protocol, or the interplay between retention policy and consumer offsets. The abstraction is doing its job — it’s supposed to spare you those details.

We wanted to not be spared.

Learning-first as a constraint

“Learning-first” isn’t a marketing word. Inside the code it’s a hard constraint that keeps rejecting design choices:

• The moving parts have to be legible. A partition’s entire on-disk representation — the segmented append-only log, the sparse index, the CRC-framed records, the compaction path — is one Go package. The active-segment abstraction lives in a single 378-line file with the kind of comments that explain why rather than what. When something misbehaves on disk, the surface area to reason about is small enough to hold in one head.

• The core interfaces have to be narrow. The Store interface the broker uses to talk to storage is four methods: Append, Read, HighWater, DeleteTopic. That’s it. Two implementations exist — an in-memory one for tests, a disk-backed one for real use — and both fit in one page each. Retention, compaction, and consumer-offset tracking sit outside the storage layer on purpose, so the storage layer never has to grow to accommodate them.

• Single binary, single language for the core. holocrond is one Go binary. No sidecars, no companion services, no polyglot orchestra, no external coordinator — Raft runs embedded, using boltdb for its own log. When you want to understand what the broker is doing, there is exactly one process to attach to.

• No performance heroics. The index that lets a fetch skip to a target offset is sparse, not dense: it stores one entry per 4 KiB of log, then scans forward in the log file from there. That’s Kafka’s granularity, but we deliberately kept the lookup as an obvious binary-search-then-scan rather than something clever. Cluster-wide Raft (not per-partition) limits write throughput to a single leader — and we picked it anyway, because one Raft group is easier to reason about than N. Consumer offsets live in a JSON file you can cat. Every one of these choices leaves performance on the table in exchange for something a reader can hold in their head.

• The broker stays dumb. No user-supplied code runs inside the broker process. No content-based routing, no server-side transforms, no filter-in-the-broker cleverness. Every transformation lives in the SDK or in the connector tier. This is a documented architectural principle, not an accident — Kafka Streams is a superb piece of engineering that we consciously did not want to imitate, because the moment the broker starts running expressions the “small and legible” property is gone.

Every one of those constraints would be wrong if we were building for production traffic at a startup. They are correct for what we’re actually building, which is a working artifact of a design you can hold in your head end-to-end.

What you give up

You give up scale. You give up the community, the operator ecosystem, the years of production hardening, the exhaustive client libraries in every language, the certifications enterprise buyers want. You give up any reasonable path to becoming someone’s Kafka. Holocron is explicitly pre-alpha: the on-disk format and the wire protocol still change without notice, and that is a feature rather than a defect at this stage — if we froze either one now, the artifact would ossify before it had finished teaching us anything.

What you get in return is a broker where the interesting mechanisms are legible one at a time — a codebase that teaches while it runs. Which turns out to be worth quite a lot, if that’s what you came for.

The pattern beyond Holocron

We’ve noticed the same shape shows up in other projects here. The identity platform is a from-scratch OAuth2 + OIDC implementation, structured as microservices, when Auth0 exists. Kyber rescores what golangci-lint already catches, from a different angle. Neospec reimplements a test-runner shape that had been fine for years.

None of these are “better than” the incumbent they resemble. That was never the goal. Each one exists because the act of building it is the artifact — and because the resulting code, once it exists, happens to be small enough and pointed enough that a working engineer can read it in an afternoon and come out understanding the problem better than they went in.

That’s the collective’s actual product. Everything else is a side effect.

Named for the discipline, not the swords.