Skip to content

Destructure proto hooks for the CSV sink #24623

Closed

Description

@adriangb

Scope

file type
datasource-csv/src/file_format.rs CsvSink

2 hooks (1 encoder, 1 decoder).

CsvSource in datasource-csv/src/source.rs is being converted as part of the fix for #24609 and is not in scope here.

Notes

CsvSink holds config and writer_options; the encoder reads neither by name today. writer_options is where the CSV writer settings live, so the destructure should confirm that everything a sink needs to reproduce its output format is actually on the wire.

Why

Serde hooks that read state through getters or self.field make an added field invisible to serialization: nothing breaks at compile time, the field simply stops round-tripping, and Debug-comparing round-trip tests do not notice. HashJoinExec::fetch was lost exactly this way (#24165), and #24609 is a second live instance found by applying the convention to one file.

#24164 established the fix -- exhaustive destructuring in both directions -- and applied it to the join plans. The physical-plan plan nodes are done. physical-expr and the datasource* crates were never converted.

What to do

For each hook in scope:

  1. In try_to_proto, start with an exhaustive let Self {{ .. }} -- no .. rest pattern. Fields that are genuinely not serialized bind to _ with a short comment saying why (derived at construction, runtime state, recomputed on decode, carried by a parent message).
  2. In try_from_proto, destructure the prost-generated node struct the same way, so adding a field to datafusion.proto is a compile error in every decoder.
  3. If the destructure turns up a field that should round-trip but has no wire representation, add it to the message and cover it with a test that fails without the fix.

Definition of done

Activity

  1. buraksenn commented on Aug 24, 2026

    @buraksenn
    Contributor

    take

  2. adriangb commented on Sep 7, 2026

    @adriangb
    ContributorAuthor

    @buraksenn are you still working on this?

  3. buraksenn commented on Sep 7, 2026

    @buraksenn
    Contributor

    @buraksenn are you still working on this?

    I'll open a PR tomorrow for remaining 3 proto issues that are assigned to me

  4. adriangb commented on Sep 7, 2026

    @adriangb
    ContributorAuthor

    Thanks so much, sorry for the ping!

  5. added a commit that references this issue on Sep 16, 2026
    2cb6b54
  6. added a commit that references this issue on Oct 11, 2026
    469ee29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

protoRelated to proto crate

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions