Efficient Go serialization of struct to disk

If you zip a file named a.txt containing the text "hello" (which is 5 characters), the result zip will be around 115 bytes. Does this mean the zip format is not efficient to compress text files? Certainly not. There is an overhead. If the file contains "hello" a hundred times (500 bytes), zipping it will result in a file being 120 bytes! 1x"hello" => 115 bytes, 100x"hello" => 120 bytes! We added 495 byes, and yet the compressed size only increased by 5 bytes.

Something similar is happening with the encoding/gob package:

The implementation compiles a custom codec for each data type in the stream and is most efficient when a single Encoder is used to transmit a stream of values, amortizing the cost of compilation.

When you “first” serialize a value of a type, the definition of the type also has to be included / transmitted, so the decoder can properly interpret and decode the stream:

A stream of gobs is self-describing. Each data item in the stream is preceded by a specification of its type, expressed in terms of a small set of predefined types.

Let’s return to your example:

var buf bytes.Buffer
enc := gob.NewEncoder(&buf)
e := Entry{"k1", "v1"}
enc.Encode(e)
fmt.Println(buf.Len())

It prints:

48

Now let’s encode a few more of the same type:

enc.Encode(e)
fmt.Println(buf.Len())
enc.Encode(e)
fmt.Println(buf.Len())

Now the output is:

60
72

Try it on the Go Playground.

Analyzing the results:

Additional values of the same Entry type only cost 12 bytes, while the first is 48 bytes because the type definition is also included (which is ~26 bytes), but that is a one-time overhead.

So basically you transmit 2 strings: "k1" and "v1" which are 4 bytes, and the length of strings also has to be included, using 4 bytes (size of int on 32-bit architectures) gives you the 12 bytes, which is the “minimum”. (Yes, you could use a smaller type for length, but that would have its limitations. A variable-length encoding would be a better choice for small numbers, see encoding/binary package.)

All in all, encoding/gob does a pretty good job for your needs. Don’t get fooled by initial impressions.

If this 12 bytes for one Entry is too “much” for you, you can always wrap the stream into a compress/flate or compress/gzip writer to further reduce the size (in exchange for slower encoding/decoding and slightly higher memory requirement for the process).

Demonstration:

Let’s test the following 5 solutions:

• Using a “naked” output (no compression)
• Using compress/flate to compress the output of encoding/gob
• Using compress/zlib to compress the output of encoding/gob
• Using compress/gzip to compress the output of encoding/gob
• Using github.com/dsnet/compress/bzip2 to compress the output of encoding/gob

We will write a thousand entries, changing keys and values of each, being "k000", "v000", "k001", "v001" etc. This means the uncompressed size of an Entry is 4 byte + 4 byte + 4 byte + 4 byte = 16 bytes (2×4 bytes text, 2×4 byte lengths).

The code looks like this:

for _, name := range []string{"Naked", "flate", "zlib", "gzip", "bzip2"} {
    buf := &bytes.Buffer{}

    var out io.Writer
    switch name {
    case "Naked":
        out = buf
    case "flate":
        out, _ = flate.NewWriter(buf, flate.DefaultCompression)
    case "zlib":
        out, _ = zlib.NewWriterLevel(buf, zlib.DefaultCompression)
    case "gzip":
        out = gzip.NewWriter(buf)
    case "bzip2":
        out, _ = bzip2.NewWriter(buf, nil)
    }

    enc := gob.NewEncoder(out)
    e := Entry{}
    for i := 0; i < 1000; i++ {
        e.Key = fmt.Sprintf("k%3d", i)
        e.Val = fmt.Sprintf("v%3d", i)
        enc.Encode(e)
    }

    if c, ok := out.(io.Closer); ok {
        c.Close()
    }
    fmt.Printf("[%5s] Length: %5d, average: %5.2f / Entry\n",
        name, buf.Len(), float64(buf.Len())/1000)
}

Output:

[Naked] Length: 16036, average: 16.04 / Entry
[flate] Length:  4120, average:  4.12 / Entry
[ zlib] Length:  4126, average:  4.13 / Entry
[ gzip] Length:  4138, average:  4.14 / Entry
[bzip2] Length:  2042, average:  2.04 / Entry

Try it on the Go Playground.

As you can see: the “naked” output is 16.04 bytes/Entry, just little over the calculated size (due to the one-time tiny overhead discussed above).

When you use flate, zlib or gzip to compress the output, you can reduce the output size to about 4.13 bytes/Entry, which is about ~26% of the theoretical size, I’m sure that satisfies you. If not, you can reach out to libs providing compression with higher efficiency like bzip2, which in the above example resulted in 2.04 bytes/Entry, being 12.7% of the theoretical size!

(Note that with “real-life” data the compression ratio would probably be a lot higher as the keys and values I used in the test are very similar and thus really well compressible; still ratio should be around 50% with real-life data).