# Building my own DB engine for fun

Creating my own db engine for fun

Canonical: https://valtterisavonen.fi/blog/building-db-engine
Published: 2026-10-15

Author: Valtteri Savonen

Over the past few months, I’ve somehow found myself in the middle of the database wars on X (form. Twitter), surrounded by people building database companies, arguing about database internals, tools and especially uptimes of managed databases.

Somewhere along the trenches, I discovered how interesting databases really are and found out about *databasemaxxing.*

Naturally, I decided to look further and scratch this itch.

For the past month, I’ve been going down the database rabbit hole. Reading articles, watching YouTube videos, going through blog posts, and even reading research papers from the 1980’s. Eventually, I came across @andy_pavlo’s lectures on the CMU Database Group’s YouTube channel, which have been an absolute goldmine.

It’s fascinating how databases seem so easy and hard at the same time. The high-level architecture makes sense relatively quickly, but actually implementing everything correctly and understanding little things is a completely different story.

This journey initially started with contributing to an open-source database project alongside some German and Dutch folks. As I kind of saw it coming it died within a few weeks and I wasn’t collaborator so if I wanted to keep doing I needed to create my own engine.

## Creating my own database engine

I initially went with a key-value store, and that’s what I have today, sort of. It uses MVCC for concurrent reads, allows a single writer at a time, and uses a write-ahead log (WAL) for durability and recovery after failures. And bunch of other small stuff.

One of the current problems is that the keys are kept in memory, and the way data is persisted to disk is pretty inefficient. Andy’s lectures have already helped me rethink parts of the storage implementation.

I’m now implementing a page-based storage strategy built around 4 KB pages, with a buffer manager that enforces a hard memory limit. Instead of loading all the data into memory, the database will load pages when needed, modify them in memory, mark them as dirty, and flush them back to disk when appropriate. This should let the database handle datasets larger than available RAM while making better use of caching and reducing unnecessary disk I/O.

However, this doesn't solve the indexing problem yet. The database still relies on an in-memory collection of keys to locate data, which means the index itself will continue to grow with the dataset. The next step is to introduce an on-disk data structure, such as a B+tree, that organizes keys into pages and allows the database to perform efficient lookups without keeping every key in memory. More on that later.

The current implementation provides ACID transaction guarantees, but they're still limited. Consistency is particularly weak because we're working with arbitrary keys and values, without constraints to enforce application-level data integrity or anything else really. Isolation is snapshot-based: transactions operate on a frozen view of the database state, can read their own writes, and don't expose uncommitted changes. Atomicity is ensured by committing only successful transactions and cancelling them on errors. Durability is provided by the WAL, which allows us to recover committed changes after crashes or power loss, with log records flushed to stable storage through group commit.

These guarantees are based on the current implementation, but they're not yet proven by a comprehensive test suite. They're the properties the database is designed to provide, rather than guarantees I’ve verified across all relevant failure scenarios.

## Conclusion

I still have a lot of Andy’s lectures to go through. There’s a 100% chance I’m missing important details that could change some of my current design decisions, so I’ll probably spend more time on the theory before going much further with implementation. This may allow me to stop me from going too far at wrong direction. Or maybe even give me some real direction to go :D

The long-term goal is to build an **actual database management system**, with a relational model and SQL support. But I’ll move forward part by part and let’s see what this will become. Also current implementation is in Go which is not a regular language you would write a database with. More common would be C/C++ but the goal here was not about learning C pointers but learning internals of databases. Maybe C++/Rust rewrite later??

I’m very open to feedback and if you think I’m doing something wrong, please contact me!

DM on X: [https://x.com/vvaltterisa](https://x.com/vvaltterisa)

Email me: savonen.emppu@gmail.com