Repository navigation
Description
Add protobuf serialization and deserialization support for the unified CastExpr (logical) and CastColumnExpr (physical) to enable distributed execution, Flight SQL, CLI round‑trips, and cross-process plan exchange.
Scope
Implement complete serialization/deserialization round‑trip support in datafusion/proto/ for both logical and physical cast expressions.
Tasks
Logical Cast Serialization:
- Update
datafusion/proto/proto/logical_plan.prototo support enhancedCastExpr - Add proto serde for enhanced
CastExprindatafusion/proto/src/logical_plan.rs - Implement round-trip tests for logical cast expressions
Physical Cast Serialization:
- Add proto message for
CastColumnExprindatafusion/proto/proto/physical_plan.proto - Implement serialization logic (
PhysicalExpr→ protobuf) - Implement deserialization logic (protobuf →
PhysicalExpr) - Add round-trip tests for
CastColumnExprvariants - Test with distributed/Flight SQL scenarios
Integration & Testing:
- Validate round-trip equality (expr → proto → expr)
- Test with struct casting scenarios
- Ensure backward compatibility (if applicable)
- Add comprehensive proto round-trip test suite
Files/Modules
datafusion/proto/proto/logical_plan.proto(logical cast message)datafusion/proto/proto/physical_plan.proto(physical cast message)datafusion/proto/src/logical_plan.rs(logical cast serde)datafusion/proto/src/physical_expr.rs(physical cast serde)- Proto-related integration tests
Key Features
- Lossless round-trip: Serialized and deserialized expressions are equivalent
- Struct awareness: Serialization preserves struct field casting semantics
- Nullability propagation: Field-level nullability metadata survives round-trip
- Backward compatibility: Graceful handling of older proto versions (if applicable)
- Clear error messages: Helpful errors when deserialization fails
Acceptance Criteria
✅ Protobuf messages defined for both logical and physical cast expressions
✅ Serialization/deserialization code is complete and tested
✅ Round-trip tests pass for all CastExpr and CastColumnExpr variants
✅ Struct casting scenarios work end-to-end with serialization
✅ Code compiles and all proto tests pass
✅ Documented any proto breaking changes or version notes
✅ Integration tests validate distributed/Flight SQL compatibility
Notes
- Scope: This PR enables distributed execution, Flight SQL, and CLI serialization for the cast expressions introduced in Extend CastExpr and Converge on Single Cast Expression #20164.
- Breaking changes: Proto updates may require versioning or migration notes if deployed to existing systems.
Activity
kosiew commentedon Feb 6, 2026
Proto changes cannot be decoupled into its own PR.
The proto changes have to be implemented together with CastColumnExpr as it is part of the physical expression tree, which is serialized/deserialized in datafusion-proto. Once we add a new physical expression variant or change its data model (like OwnedCastOptions, schema-aware fields, etc.), we need to update the proto layer so:
Plans can be serialized/deserialized correctly
CastColumnExpr gets sent over the wire (or stored) when physical plans are serialized. Without proto updates, roundtrip tests fail or the plan can’t be restored.
Owned cast/format options now need to be preserved
The new design stores options in owned Strings. That means the protobuf needs fields for PhysicalCastOptions and FormatOptions, plus the corresponding conversion logic in from_proto.rs and to_proto.rs.
Round-trip tests assert fidelity
The roundtrip_physical_plan.rs tests exercise proto serialization, so those must be updated whenever expression structs or option handling changes.