You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ListingTable silently drops paths when given many files #18242
In our system, we hand a large list of individual S3 paths (taken from an iceberg manifest) to ListingTable. With high levels of paralellism, we've observed ListingTable dropping 5-10% of those files silently, leading to incorrect results.
There are 3283 paths in the input, and the scan is unfiltered, so it should read all of them. Instead, it reads around ~2950 on my machine (the number is different every time).
My laptop is very warm, but the issue appears to bisect to 528ed60 (#17364).
I think what's happening here is that I'm seeing transient errors doing the HEAD operation, and then it's falling back to treating the path like a prefix - but it's not a prefix.
Given the results of your bisect (I feel sorry for your laptop) I'm initially inclined to agree with your assessment of the regression here. The commit/PR referenced matches any error from the HEAD request and then assumes it's a path. It seems like the code could be more selective in the errors it matches (e.g. only match against object_store::Error::NotFound).
Even when matching error pattern object_store::Error::NotFound, silent data regressions are still possible - imagine an object-store implementation with relaxed data consistency guarantees, that can return transient NotFound errors for files recently written. From a Datafusion client perspective, it feels reasonable to expect a failfast by default rather than a data-regression in such cases - that was our understanding of the listing_table implementation before #17364.
I wonder if there are ways to tweak the implementation of the quality-of-life PR #17364 so that it doesn't risk major silent data regressions - some clients have non-ambiguous call pattern where they expect a exactly single Parquet file read and not a directory.
Activity
colinmarc commentedon Oct 23, 2025
My laptop is very warm, but the issue appears to bisect to 528ed60 (#17364).
I think what's happening here is that I'm seeing transient errors doing the
HEADoperation, and then it's falling back to treating the path like a prefix - but it's not a prefix.BlakeOrth commentedon Oct 23, 2025
Given the results of your bisect (I feel sorry for your laptop) I'm initially inclined to agree with your assessment of the regression here. The commit/PR referenced matches any error from the
HEADrequest and then assumes it's a path. It seems like the code could be more selective in the errors it matches (e.g. only match againstobject_store::Error::NotFound).@colinmarc Since you've already run a bisect, would you have time to quickly modify the DataFusion code to print (or panic, something) on the error condition so we can better understand if/what error may be happening there?
https://github.com/apache/datafusion/pull/17364/files#diff-3f8fd00d313a1d20cfd060fd853f21df01318a1afbbdfdf1886d31c57291ff4cR255
It seems @alamb's comment on that line may have been quite prescient.
colinmarc commentedon Oct 24, 2025
Looks like transient DNS:
Pretty odd, and indeed maybe a macOS thing (my coworker was able to reproduce, but we don't see it in CI).
fix: only fall back to listing prefixes on 404 errors
fix: only fall back to listing prefixes on 404 errors
fix: only fall back to listing prefixes on 404 errors (#18263)
fix: only fall back to listing prefixes on 404 errors (apache#18263)
16pierre commentedon Feb 17, 2026
Even when matching error pattern object_store::Error::NotFound, silent data regressions are still possible - imagine an object-store implementation with relaxed data consistency guarantees, that can return transient
NotFounderrors for files recently written. From a Datafusion client perspective, it feels reasonable to expect a failfast by default rather than a data-regression in such cases - that was our understanding of thelisting_tableimplementation before #17364.I wonder if there are ways to tweak the implementation of the quality-of-life PR #17364 so that it doesn't risk major silent data regressions - some clients have non-ambiguous call pattern where they expect a exactly single Parquet file read and not a directory.