You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If you take an ExecutionPlan and serialize it, all occurrences of HashTableLookupExpr are replaced with lit(true). This makes it difficult to send dynamic filters across the network.
Describe the solution you'd like
Ideally, there's some way to serialize and deserialize these expressions, potentially by serializing the seed
I'll implement this soon, but at high-level, the design I'm thinking is:
Serialization
For evaluation purposes, we only need the hashes in the HashMap case. The protobuf then looks like this, with the contents of ArrayMapNode to be determined:
For the ArrayMapNode case, we just deserialize to the ArrayMap variant of HashTableLookupExpr. For the HashSetMapNode case, we add a new impl of JoinHashMapType with contain_hashes/len/is_empty implemented. It will return internal errors from the build-side mutation and get_matched_indices methods. This will allow it be evaluated after deserialization but not updated. (Will document that this should be constructed during deserialization only and is an immutable snapshot of the expression).
Activity
barbarj commentedon Aug 11, 2026
take
barbarj commentedon Aug 11, 2026
I'll implement this soon, but at high-level, the design I'm thinking is:
Serialization
For evaluation purposes, we only need the hashes in the
HashMapcase. The protobuf then looks like this, with the contents ofArrayMapNodeto be determined:We add a
hashesaccess method:Deserialization
For the
ArrayMapNodecase, we just deserialize to theArrayMapvariant ofHashTableLookupExpr. For theHashSetMapNodecase, we add a new impl ofJoinHashMapTypewithcontain_hashes/len/is_emptyimplemented. It will return internal errors from the build-side mutation and get_matched_indices methods. This will allow it be evaluated after deserialization but not updated. (Will document that this should be constructed during deserialization only and is an immutable snapshot of the expression).Rich-T-kid commentedon Sep 1, 2026
@barbarj are you still working on this?
barbarj commentedon Sep 1, 2026
Yeah! I was on vacation last week, hence the inactivity. I'll be following up on #24382 shortly.
coordinator: display consumer dynamic filters after execution (#623)
barbarj commentedon Nov 10, 2026
untake