Repository navigation
Description
Scope
| file | type |
|---|---|
datasource-csv/src/file_format.rs |
CsvSink |
2 hooks (1 encoder, 1 decoder).
CsvSource in datasource-csv/src/source.rs is being converted as part of the fix for #24609 and is not in scope here.
Notes
CsvSink holds config and writer_options; the encoder reads neither by name today. writer_options is where the CSV writer settings live, so the destructure should confirm that everything a sink needs to reproduce its output format is actually on the wire.
Why
Serde hooks that read state through getters or self.field make an added field invisible to serialization: nothing breaks at compile time, the field simply stops round-tripping, and Debug-comparing round-trip tests do not notice. HashJoinExec::fetch was lost exactly this way (#24165), and #24609 is a second live instance found by applying the convention to one file.
#24164 established the fix -- exhaustive destructuring in both directions -- and applied it to the join plans. The physical-plan plan nodes are done. physical-expr and the datasource* crates were never converted.
What to do
For each hook in scope:
- In
try_to_proto, start with an exhaustivelet Self {{ .. }}-- no..rest pattern. Fields that are genuinely not serialized bind to_with a short comment saying why (derived at construction, runtime state, recomputed on decode, carried by a parent message). - In
try_from_proto, destructure the prost-generated node struct the same way, so adding a field todatafusion.protois a compile error in every decoder. - If the destructure turns up a field that should round-trip but has no wire representation, add it to the message and cover it with a test that fails without the fix.
Definition of done
-
Every field of the plan/expression struct is either serialized or bound to
_with a reason. -
Every field of the prost node is destructured in the decoder.
-
Any newly serialized field has a round-trip test, verified to fail before the fix.
-
Part of EPIC: destructure proto serde hooks in the
datasourcecrates #24612.
Activity
datasourcecrates #24612buraksenn commentedon Aug 24, 2026
take
adriangb commentedon Sep 7, 2026
@buraksenn are you still working on this?
buraksenn commentedon Sep 7, 2026
I'll open a PR tomorrow for remaining 3 proto issues that are assigned to me
adriangb commentedon Sep 7, 2026
Thanks so much, sorry for the ping!
fix(proto): preserve CSV sink writer options (apache#25058)
fix(proto): preserve CSV sink writer options (apache#25058)