RDB File Format

Redis RDB File Format

Redis’ RDB file is a binary representation of the in-memory store. This binary file is sufficient to completely restore Redis’ state.

The RDB file format is optimized for fast read and writes. Where possible LZF compression is used to reduce the file size. In general, objects are prefixed with their lengths, so before reading the object you know exactly how much memory to allocate.

Optimizing for fast read/writes means the on-disk format should be as close as possible to the in-memory representation. This is the approach taken by the RDB file. As a consequence, you cannot parse the RDB file without some understanding of Redis’ in-memory representation of data structures.

High Level Algorithm to parse RDB

At a high level, the RDB file has the following structure

----------------------------#
52 45 44 49 53              # Magic String "REDIS"
30 30 30 33                 # RDB Version Number as ASCII string. "0003" = 3
----------------------------
FA                          # Auxiliary field
$string-encoded-key         # May contain arbitrary metadata
$string-encoded-value       # such as Redis version, creation time, used memory, ...
----------------------------
FE 00                       # Indicates database selector. db number = 00
FB                          # Indicates a resizedb field
$length-encoded-int         # Size of the corresponding hash table
$length-encoded-int         # Size of the corresponding expire hash table
----------------------------# Key-Value pair starts
FD $unsigned-int            # "expiry time in seconds", followed by 4 byte unsigned int
$value-type                 # 1 byte flag indicating the type of value
$string-encoded-key         # The key, encoded as a redis string
$encoded-value              # The value, encoding depends on $value-type
----------------------------
FC $unsigned long           # "expiry time in ms", followed by 8 byte unsigned long
$value-type                 # 1 byte flag indicating the type of value
$string-encoded-key         # The key, encoded as a redis string
$encoded-value              # The value, encoding depends on $value-type
----------------------------
$value-type                 # key-value pair without expiry
$string-encoded-key
$encoded-value
----------------------------
FE $length-encoding         # Previous db ends, next db starts.
----------------------------
...                         # Additional key-value pairs, databases, ...

FF                          ## End of RDB file indicator
8-byte-checksum             ## CRC64 checksum of the entire file.

Magic Number

The file starts off with the magic string “REDIS”. This is a quick sanity check to know we are dealing with a redis rdb file.

52 45 44 49 53  # "REDIS"

RDB Version Number

The next 4 bytes store the version number of the rdb format. The 4 bytes are interpreted as ASCII characters and then converted to an integer using string to integer conversion.

30 30 30 33 # "0003" => Version 3

Op Codes

Each part after the initial header is introduced by a special op code. The available op codes are:

Byte Name Description
0xFF EOF End of the RDB file
0xFE SELECTDB Database Selector
0xFD EXPIRETIME Expire time in seconds, see Key Expiry Timestamp
0xFC EXPIRETIMEMS Expire time in milliseconds, see Key Expiry Timestamp
0xFB RESIZEDB Hash table sizes for the main keyspace and expires, see Resizedb information
0xFA AUX Auxiliary fields. Arbitrary key-value settings, see Auxiliary fields

Database Selector

A Redis instance can have multiple databases.

A single byte 0xFE flags the start of the database selector. After this byte, a variable length field indicates the database number. See the section Length Encoding to understand how to read this database number.

Resizedb information

This op code was introduced in RDB version 7.

It encodes two values to speed up RDB loading by avoiding additional resizes and rehashing. The op code is followed by two length-encoded integers indicating:

Auxiliary fields

This op code was introduced in RDB version 7.

The op code is followed by two Redis Strings, representing the key and value of a setting. Unknown fields should be ignored by a parser.

Currently the following settings are implemented:

Key Value Pairs

After the database selector, the file contains a sequence of key value pairs.

Each key value pair has 4 parts:

Key Expiry Timestamp

This section starts with a one byte flag. This flag is either:

During the import process, keys that have expired must be discarded.

Value Type

A one byte flag indicates encoding used to save the Value.

Key

The key is simply encoded as a Redis string. See the section String Encoding to learn how the key is encoded.

Value

The value is parsed according to the previously read Value Type

Encodings

Length Encoding

Length encoding is used to store the length of the next object in the stream. Length encoding is a variable byte encoding designed to use as few bytes as possible.

This is how length encoding works : Read one byte from the stream, compare the two most significant bits:

Bits How to parse
00 The next 6 bits represent the length
01 Read one additional byte. The combined 14 bits represent the length
10 Discard the remaining 6 bits. The next 4 bytes from the stream represent the length
11 The next object is encoded in a special format. The remaining 6 bits indicate the format. May be used to store numbers or Strings, see String Encoding

As a result of this encoding:

String Encoding

Redis Strings are binary safe - which means you can store anything in them. They do not have any special end-of-string token. It is best to think of Redis Strings as a byte array.

There are three types of Strings in Redis:

Length Prefixed String

Length prefixed strings are quite simple. The length of the string in bytes is first encoded using Length Encoding. After this, the raw bytes of the string are stored.

Integers as String

First read the section Length Encoding, specifically the part when the first two bits are 11. In this case, the remaining 6 bits are read.

If the value of those 6 bits is:

Compressed Strings

First read the section Length Encoding, specifically the part when the first two bits are 11. In this case, the remaining 6 bits are read. If the value of those 6 bits is 3, it indicates that a compressed string follows.

The compressed string is read as follows:

List Encoding

A Redis list is represented as a sequence of strings.

Set Encoding

Sets are encoded exactly like lists.

Sorted Set Encoding

Note: Depending on the value you may loose precision. Redis saves the score as a double.

Note: It’s not guaranteed that the set is sorted already.

Example:

04
01 63 12 34 2E 30 31 39 39 39 39 39 39 39 39 39 39 39 39 39 36
01 64 FE
01 61 12 33 2E 31 38 39 39 39 39 39 39 39 39 39 39 39 39 39 39
01 65 FF

The resulting sorted set will be:

{ "e" => "-inf", "a" => "3.189999999999999", "c" => "4.0199999999999996", "d" => "+inf" }

Hash Encoding

Example:

2 us washington india delhi

represents the map

{"us" => "washington", "india" => "delhi"}

Zipmap Encoding

A Zipmap is a hashmap that has been serialized to a string. In essence, the key value pairs are stored sequentially. Looking up a key in this structure is O(N). This structure is used instead of a dictionary when the number of key value pairs are small.

To parse a zipmap, first a string is read from the stream using String Encoding. The contents of this string represent the zipmap.

The structure of a zipmap within this string is as follows:

<zmlen><len>"foo"<len><free>"bar"<len>"hello"<len><free>"world"<zmend>

Example:

18 02 06 4D 4B 44 31 47 36 01 00 32 05 59 4E 4E 58 4b 04 00 46 37 54 49 FF ..

Ziplist Encoding

A Ziplist is a list that has been serialized to a string. In essence, the elements of the list are stored sequentially along with flags and offsets to allow efficient traversal of the list in both directions.

To parse a ziplist, first a string is read from the stream using String Encoding. The contents of this string represent the ziplist.

The structure of a ziplist within this string is as follows:

<zlbytes><zltail><zllen><entry><entry><zlend>

Each entry in the ziplist has the following format :

<length-prev-entry><special-flag><raw-bytes-of-entry>

length-prev-entry: stores the length of the previous entry, or 0 if this is the first entry. This allows easy traversal of the list in the reverse direction. This length is stored in either 1 byte or in 5 bytes. If the first byte is less than or equal to 253, it is considered as the length. If the first byte is 254, then the next 4 bytes are used to store the length. The 4 bytes are read as an unsigned integer.

special-flag: This flag indicates whether the entry is a string or an integer. It also indicates the length of the string, or the size of the integer. The various encodings of this flag are shown below:

Bytes Length Meaning
00pppppp 1 byte String value with length less than or equal to 63 bytes (6 bits)
01pppppp|qqqqqqqq 2 bytes String value with length less than or equal to 16383 bytes (14 bits)
10______|<4 byte> 5 bytes Next 4 byte contain an unsigned int. String value with length greater than or equal to 16384 bytes
1100____ 3 bytes Integer encoded as 16 bit signed (2 bytes)
1101____ 5 bytes Integer encoded as 32 bit signed (4 bytes)
1110____ 9 bytes Integer encoded as 64 bit signed (8 bytes)
1111____ 4 bytes Integer encoded as 24 bit signed (3 bytes)

raw-byte: After the special flag, the raw bytes of entry follow. The number of bytes was previously determined as part of the special flag.

Example:

23 23 00 00 00 1E 00 00 00 04 00 00 E0 FF FF FF FF FF
FF FF 7F 0A D0 FF FF 00 00 06 C0 FC 3F 04 C0 3F 00 FF ...

Intset Encoding

An Intset is a binary search tree of integers. The binary tree is implemented in an array of integers. An intset is used when all the elements of the set are integers. An Intset has support for up to 64 bit integers. As an optimization, if the integers can be represented in fewer bytes, the array of integers will be constructed from 16 bit or 32 bit integers. When a new element is inserted, the implementation takes care to upgrade if necessary.

Since an Intset is a binary search tree, the numbers in this set will always be sorted.

An Intset has an external interface of a Set.

To parse an Intset, first a string is read from thee stream using String Encoding. The contents of this string represent the Intset.

Within this string, the Intset has a very simple layout :

<encoding><length-of-contents><contents>

Example

14 04 00 00 00 03 00 00 00 FC FF 00 00 FD FF 00 00 FE FF 00 00 ...

Sorted Set in Ziplist Encoding

A sorted set in ziplist encoding is stored just like the Ziplist described above. Each element in the sorted set is followed by its score in the ziplist.

Example

['Manchester City', 1, 'Manchester United', 2, 'Tottenham', 3]

As you see, the scores follow each element.

Hashmap in Ziplist Encoding

In this, key=value pairs of a hashmap are stored as successive entries in a ziplist.

Note: This was introduced in RDB version 4. This deprecates the zipmap encoding that was used in earlier versions.

Example

{"us" => "washington", "india" => "delhi"}

is stored in a ziplist as :

["us", "washington", "india", "delhi"]

Quicklist Encoding

RDB Version 7 introduced a new variant of list encoding: Quicklist.

Quicklist is a linked list of ziplists. Quicklist combines the memory efficiency of small ziplists with the extensibility of a linked list allowing us to create space-efficient lists of any length.

To parse a quicklist, first a string is read from the stream using String Encoding. The contents of this string represent the ziplist.

The structure of a quicklist within this string is as follows:

<len><ziplist><ziplist>...

A complete list needs to be constructed from all elements of all ziplists.

Example:

01 1F 1F 00 00 00 17 00 00 00 02 00 00 0B 6F 6E 65 2D 65 6C 65 6D 65 6E 74 0D 05 65 6C 65 6D 32 FF

CRC64 Checksum

Starting with RDB Version 5, an 8 byte CRC64 checksum is added to the end of the file. It is possible to disable this checksum via a parameter in redis.conf. When the checksum is disabled, this field will have zeroes.

Copyright © 2015 Jan-Erik Rediger, released under the MIT license | to the TOC