Serialization 101 — Objects to Bytes and Back

The Core Problem

Scenario: Process A has a User object with pointers, hash maps, and language-specific layout. Process B — maybe another language, another machine, another CPU architecture — needs "the same" user.

You cannot ship the memory. Pointers are meaningless across address spaces; struct padding and endianness differ; the other side may not even have a User class.

Serialization (a.k.a. marshalling) flattens the in-memory object into a portable byte sequence. Deserialization (unmarshalling) rebuilds an equivalent object on the far side. The byte sequence is the wire format; the class is the runtime representation.

The Wire ≠ The Runtime

The single most useful mental model: the bytes on the wire are a different thing from the object in your code, and a format is a contract about the bytes, not about your classes.

Runtime: rich, typed, full of pointers — optimized for the CPU and the language.
Wire: flat, self-contained, portable — optimized for size, forward/backward compatibility, or parse speed (you rarely get all three).

Why this matters: two programs interoperate if they agree on the wire bytes — never because they share code. That decoupling is what lets a Go service talk to a Python service and lets a new version read old data. Keep the wire in your head, not the struct.

Axis 1 — Text vs Binary

Text (JSON, XML, YAML, CSV)

{ "name": "Bob", "id": 150, "admin": true }

Human-readable, debuggable with curl and your eyes, trivially logged. But every number is spelled out in ASCII, every key is repeated in full on every record, and the parser must scan for quotes, braces, and escapes.

Binary (Protobuf, Avro, MessagePack)

0A 03 42 6F 62 10 96 01 18 01

Compact and fast to parse — a number is a few bytes, not a string of digits — but opaque without the schema or a decoder. You trade eyeballs for bytes and CPU.

	Text	Binary
Readable by humans	✅	❌
Size on the wire	Large	Small
Parse cost	Higher	Lower
Debug with plain tools	✅	Needs decoder

Axis 2 — Schema-ful vs Schemaless

The second axis is orthogonal: does the reader need an out-of-band description of the data?

Schemaless / self-describing (JSON, MessagePack): the field names and type tags travel inside every message. You can parse it knowing nothing in advance. The cost: those names ride along on every single record.

Schema-ful (Protobuf, Avro, Thrift): a separate schema (.proto, .avsc) defines the fields. The wire bytes carry only compact numeric tags — or, with Avro, no tags at all. Smaller and faster, but the reader must have the schema.

The trade in one line: self-describing formats pay bytes to stay flexible; schema-ful formats pay coordination to stay small. Protobuf sits in the middle — it keeps tiny numeric field tags on the wire, so a decoder can skip unknown fields without the schema, but needs the schema to name them. See The Wire Format.

What a "Good" Format Optimizes

Different formats pick different corners of the same triangle. Know which corner you need:

Priority	Reach for	Because
Debuggability / public API	JSON	Everyone can read and generate it
Small, fast internal RPC	Protobuf	Numeric tags, varints, codegen
Schema-evolving data lake	Avro	Schema stored with the data, no tags on the wire
Read without parsing	FlatBuffers	Zero-copy access to fields in place

Real-world: a typical shop uses both — JSON at the public edge where humans and browsers live, Protobuf between internal services where the same messages fly millions of times a second and every byte and microsecond compounds. Format choice is per-boundary, not per-company.