How to do serialization of Class having members of custom data types in C++? [closed]

by

I commented:

Perhaps the examples here give you some inspiration. It’s possible to write them without any boost, obviously Boost Serialization Binary Archive giving incorrect output

Because I hate when people say “obviously” on a Q&A site, let me show you. I’d suggest the interface to look like this:

std::vector<uint8_t> serialize(Mango const& Man);
Mango                deserialize(std::span<uint8_t const> data);

Alternatively, for file IO you could support e.g.:

void serialize_to_stream(std::ostream& os, Mango const& Man);
void deserialize(std::istream& is, Mango& Man);

Using the approach from the linked example, the suggested implementations would look like:

std::vector<uint8_t> serialize(Mango const& Man) {
    std::vector<uint8_t> bytes;
    do_generate(back_inserter(bytes), Man);
    return bytes;
}

Mango deserialize(std::span<uint8_t const> data) {
    Mango result;
    auto  f = begin(data), l = end(data);
    if (!do_parse(f, l, result))
        throw std::runtime_error("deserialize");
    return result;
}

void serialize_to_stream(std::ostream& os, Mango const& Man)  {
    do_generate(std::ostreambuf_iterator<char>(os), Man);
}

void deserialize(std::istream& is, Mango& Man) {
    Man = {}; // clear it!
    std::istreambuf_iterator<char> f(is), l{};
    if (!do_parse(f, l, Man))
        throw std::runtime_error("deserialize");
}

Of course, that assumes do_generate and do_parse customizations for all the relevant types (ValType, FunctionMango, MangoType, Mango):

Live On Coliru

#include <algorithm>
#include <iomanip> // debug output
#include <iostream>
#include <string>
#include <vector>
#include <span>

namespace MangoLib {
    // your requested signatures:
    class Mango;

    void serialize_to_stream(std::ostream& os, Mango const& Man);
    void deserialize(std::istream& is, Mango& Man);
    std::vector<uint8_t> serialize(Mango const& Man);
    Mango                deserialize(std::span<uint8_t const> data);

    // your specified types (with some demo fill)
    enum class ValType : uint8_t {
#define UseValType
#define Line(NAME, VALUE, STRING) NAME = VALUE
        Line(void_,   0, "void"),
        Line(int_,    1, "int"),
        Line(bool_,   2, "bool"),
        Line(string_, 3, "string"),
#undef Line
#undef UseValType
    };

    using ValTypes = std::vector<ValType>;
    class FuntionMango {
      public:
        const ValTypes& getParamTypes() const noexcept { return ParamTypes; }
        ValTypes& getParamTypes() noexcept { return ParamTypes; }

        const ValTypes& getReturnTypes() const noexcept { return ReturnTypes; }
        ValTypes& getReturnTypes() noexcept { return ReturnTypes; }

      private:
        ValTypes ParamTypes, ReturnTypes;
    };

    using FuntionMangos = std::vector<FuntionMango>;

    class MangoType {
      public:
        FuntionMangos&       getContent() noexcept { return Content; }
        const FuntionMangos& getContent() const noexcept { return Content; }

      private:
        FuntionMangos Content;
    };

    class Mango {
      public:
        const MangoType& getMangoType() const { return typeMan; }
        MangoType&       getMangoType() { return typeMan; }

      private:
        MangoType typeMan;
        // many other members
    };
} // namespace MangoLib

namespace my_serialization_helpers {

    ////////////////////////////////////////////////////////////////////////////
    // This namespace serves as an extension point for your serialization; in
    // particular we choose endianness and representation of strings
    //
    // TODO add overloads as needed (signed integer types, binary floats,
    // containers of... etc)
    ////////////////////////////////////////////////////////////////////////////
    
    // decide on the max supported container capacity:
    using container_size_type = std::uint32_t;
    
    ////////////////////////////////////////////////////////////////////////////
    // generators
    template <typename Out>
    Out do_generate(Out out, std::string const& data) {
        container_size_type len = data.length();
        out = std::copy_n(reinterpret_cast<char const*>(&len), sizeof(len), out);
        return std::copy(data.begin(), data.end(), out);
    }

    template <typename Out, typename T>
    Out do_generate(Out out, std::vector<T> const& data) {
        container_size_type len = data.size();
        out = std::copy_n(reinterpret_cast<char const*>(&len), sizeof(len), out);
        for (auto& el : data)
            out = do_generate(out, el);
        return out;
    }

    template <typename Out> Out do_generate(Out out, uint8_t const& data) {
        return std::copy_n(&data, sizeof(data), out);
    }

    template <typename Out>
    Out do_generate(Out out, uint16_t const& data) {
        return std::copy_n(reinterpret_cast<char const*>(&data), sizeof(data), out);
    }

    template <typename Out>
    Out do_generate(Out out, uint32_t const& data) {
        return std::copy_n(reinterpret_cast<char const*>(&data), sizeof(data), out);
    }

    ////////////////////////////////////////////////////////////////////////////
    // parsers
    template <typename It>
    bool parse_raw(It& in, It last, char* raw_into, size_t n) { // length guarded copy_n
        while (in != last && n) {
            *raw_into++ = *in++;
            --n;
        }
        return n == 0;
    }

    template <typename It, typename T>
    bool parse_raw(It& in, It last, T& into) {
        static_assert(std::is_trivially_copyable_v<T>);
        return parse_raw(in, last, reinterpret_cast<char*>(&into), sizeof(into));
    }

    template <typename It>
    bool do_parse(It& in, It last, std::string& data) {
        container_size_type len;
        if (!parse_raw(in, last, len))
            return false;
        data.resize(len);
        return parse_raw(in, last, data.data(), len);
    }

    template <typename It, typename T>
    bool do_parse(It& in, It last, std::vector<T>& data) {
        container_size_type len;
        if (!parse_raw(in, last, len))
            return false;
        data.clear();
        data.reserve(len);
        while (len--) {
            data.emplace_back();
            if (!do_parse(in, last, data.back()))
                return false;
        };
        return true;
    }

    template <typename It>
    bool do_parse(It& in, It last, uint8_t& data) {
        return parse_raw(in, last, data);
    }

    template <typename It>
    bool do_parse(It& in, It last, uint16_t& data) {
        return parse_raw(in, last, data);
    }

    template <typename It>
    bool do_parse(It& in, It last, uint32_t& data) {
        return parse_raw(in, last, data);
    }
}

namespace MangoLib {

    template <typename Out> Out do_generate(Out out, ValType const& x) {
        using my_serialization_helpers::do_generate;
        return do_generate(out,
                           static_cast<std::underlying_type_t<ValType>>(x));
    }
    template <typename It> bool do_parse(It& in, It last, ValType& x) {
        using my_serialization_helpers::do_parse;
        std::underlying_type_t<ValType> tmp;
        bool ok = do_parse(in, last, tmp);
        if (ok)
            x = static_cast<ValType>(tmp);
        return ok;
    }

    template <typename Out> Out do_generate(Out out, FuntionMango const& x) {
        using my_serialization_helpers::do_generate;
        out = do_generate(out, x.getParamTypes());
        out = do_generate(out, x.getReturnTypes());
        return out;
    }
    template <typename It> bool do_parse(It& in, It last, FuntionMango& x) {
        using my_serialization_helpers::do_parse;
        return do_parse(in, last, x.getParamTypes()) &&
            do_parse(in, last, x.getReturnTypes());
    }

    template <typename Out> Out do_generate(Out out, MangoType const& x) {
        using my_serialization_helpers::do_generate;
        out = do_generate(out, x.getContent());
        return out;
    }
    template <typename It> bool do_parse(It& in, It last, MangoType& x) {
        using my_serialization_helpers::do_parse;
        return do_parse(in, last, x.getContent());
    }

    template <typename Out> Out do_generate(Out out, Mango const& x) {
        out = do_generate(out, x.getMangoType());
        return out;
    }
    template <typename It> bool do_parse(It& in, It last, Mango& x) {
        return do_parse(in, last, x.getMangoType());
    }
}

#include <cassert>

MangoLib::Mango makeMango() {
    MangoLib::Mango mango;

    using MangoLib::ValType;
    MangoLib::FuntionMango f1;
    f1.getParamTypes()  = {ValType::bool_, ValType::string_};
    f1.getReturnTypes() = {ValType::void_};

    MangoLib::FuntionMango f2;
    f2.getParamTypes()  = {ValType::string_};
    f2.getReturnTypes() = {ValType::int_};

    mango.getMangoType().getContent() = {f1, f2};
    return mango;
}

#include <fstream>

int main() {
    auto const mango = makeMango();

    auto const bytes = serialize(mango);
    auto const roundtrip = serialize(MangoLib::deserialize(bytes));
    assert(roundtrip == bytes);

    // alternatively with file IO:
    {
        std::ofstream ofs("output.bin", std::ios::binary);
        serialize_to_stream(ofs, mango);
    }
    // read back:
    {
        std::ifstream ifs("output.bin", std::ios::binary);
        MangoLib::Mango from_file;
        deserialize(ifs, from_file);

        assert(serialize(from_file) == bytes);
    }

    std::cout << "\nDebug dump " << std::dec << bytes.size() << " bytes:\n";
    for (auto ch : bytes)
        std::cout << "0x" << std::hex << std::setw(2) << std::setfill('0')
                  << static_cast<int>((uint8_t)ch) << " " << std::dec;
    std::cout << "\nDone\n";
}

// suggested implementations:
namespace MangoLib {
    std::vector<uint8_t> serialize(Mango const& Man) {
        std::vector<uint8_t> bytes;
        do_generate(back_inserter(bytes), Man);
        return bytes;
    }

    Mango deserialize(std::span<uint8_t const> data) {
        Mango result;
        auto  f = begin(data), l = end(data);
        if (!do_parse(f, l, result))
            throw std::runtime_error("deserialize");
        return result;
    }

    void serialize_to_stream(std::ostream& os, Mango const& Man)  {
        do_generate(std::ostreambuf_iterator<char>(os), Man);
    }

    void deserialize(std::istream& is, Mango& Man) {
        Man = {}; // clear it!
        std::istreambuf_iterator<char> f(is), l{};
        if (!do_parse(f, l, Man))
            throw std::runtime_error("deserialize");
    }
}

Which roundtrips correctly and prints the debug output:

Debug dump 25 bytes:
0x02 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x02 0x03 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x03 0x01 0x00 0x00 0x00 0x01 
Done

Portability

This assumes endianness is not an issue. Of course you might want to normalize endianness. You can do it manually (using ntoh/hton family e.g.), or you could use Boost Endian – which does not require linking to any boost library (Boost Endian is header-only).

E.g.: http://coliru.stacked-crooked.com/a/288829ec964a3ca9

More Related Contents:

  1. what is the concept of evaluation of expression z = (y = 30) + (y = 10) + (y = 20); [closed]
  2. C++ 11: conversion const int* to int* using unordered_set::push
  3. Is cout synchronized/thread-safe?
  4. Is local static variable initialization thread-safe in C++11? [duplicate]
  5. Do rvalue references to const have any use?
  6. How to succinctly, portably, and thoroughly seed the mt19937 PRNG?
  7. Does constexpr imply inline?
  8. C++11: The range-based for statement: “range-init” lifetime?
  9. Do rvalue references allow dangling references?
  10. Initializer list not working with vector in Visual Studio 2012? [duplicate]
  11. Why is my object not being copied when move constructor is deleted?
  12. On how to recognize Rvalue or Lvalue reference and if-it-has-a-name rule
  13. `auto` specifier type deduction for references
  14. Difference between std::system_clock and std::steady_clock?
  15. C++ Modules – why were they removed from C++0x? Will they be back later on?
  16. Which headers in the C++ standard library are guaranteed to include another header?
  17. Default template argument when using std::enable_if as templ. param.: why OK with two template functions that differ only in the enable_if parameter?
  18. std::auto_ptr to std::unique_ptr
  19. How can an incomplete type be used as a template parameter to vector here?
  20. How can I use C++ 11 features in Clang?
  21. The new keyword “auto”; When should it be used to declare a variable type? [duplicate]
  22. When Does Move Constructor get called?
  23. Questions about Hinnant’s stack allocator
  24. Equivalent of “using namespace X” for scoped enumerations?
  25. The implementation of random_device in VS2010?
  26. Memory alignment : how to use alignof / alignas?
  27. Does C++11 change the behavior of explicitly calling std::swap to ensure ADL-located swap’s are found, like boost::swap?
  28. C++ Serialization Performance
  29. Does ‘auto’ type assignments of a pointer in c++11 require ‘*’?
  30. Is it safe to reinterpret_cast an enum class variable to a reference of the underlying type?

Leave a Comment