Durability is the process of persisting the writes that clients make to the database to a non-volatile storage disk. Caveat is that the writes should also be committed. This means once the commit has succeeded and right after that our system loses power or the system crashes and we need to restart, then our changes and writes should still be reflected in the database. It is the fourth (and last) of the ACID properties of a database transaction.
While it all sounds quite simple. After-all , that’s one of the most basic things that computers are used for, right? Saving Data! But when it comes to databases it’s not really that easy. Writing data to disk is an expensive task and involves a lot of work – (writing/ updating) indexes, data files, columns, rows and whatnot. Then We have the hardware on top of which sits some kind of Operating System (OS) and on top of that sits our DBMS. And the DBMS talks to the OS for reading-writing from-to the underlying hardware disk and the OS itself might play tricks for optimisation and other purposes. Like the problem of OS cache as discussed…
The problem of OS Cache
- OS usually cache data in any write requests to its cache instead of flushing directly to disk. This is to save the number of writes to disk which is a slow operation.
- So when the write happens just to OS cache and not to disk actually, and we have a system crash/ machine restart.. Then we’d have loss of data
- There’s something called
force synccommand provided by most OSes.. which can be used to force flushing the writes to disks – need some sort of smartness on DB/ application part to optimize writes to disk as this again is expensive and slows down the system/ application
Durability techniques
Let’s talk about the various durability techniques out there employed by different DBMS systems.
- WAL – Write ahead logs (pronounced ‘wall’)
- Asynchronous snapshot
- AOF – append only file
WAL
DBMS often persist compressed version of the changes known as WAL (write-ahead-log segments)
Asynchronous snapshot
In this approach everything we write is kept in-memory and later on, asynchronously, it’s snapshot is synced/ written to the drive (persisted disk/ SSD/ Hard-drive) all at once.
AOF (Append Only File)
Very similar to the WAL strategy, It keeps track of only the changes happening. A very lightweight way of storing the changes and also gives the possibility of reconstructing the state of the system in case of failures.