Skip to content

ListingTable silently drops paths when given many files #18242

Closed

Description

@colinmarc

Describe the bug

In our system, we hand a large list of individual S3 paths (taken from an iceberg manifest) to ListingTable. With high levels of paralellism, we've observed ListingTable dropping 5-10% of those files silently, leading to incorrect results.

To Reproduce

Here is a test you can run without AWS credentials: https://gist.github.com/colinmarc/2e1c62a1909b2aa63b6bdbad4b81ce64

Expected behavior

There are 3283 paths in the input, and the scan is unfiltered, so it should read all of them. Instead, it reads around ~2950 on my machine (the number is different every time).

Additional context

No response

Activity

  1. added theissue type on Oct 23, 2025
  2. colinmarc commented on Oct 23, 2025

    @colinmarc
    ContributorAuthor

    My laptop is very warm, but the issue appears to bisect to 528ed60 (#17364).

    I think what's happening here is that I'm seeing transient errors doing the HEAD operation, and then it's falling back to treating the path like a prefix - but it's not a prefix.

  3. BlakeOrth commented on Oct 23, 2025

    @BlakeOrth
    Contributor

    Given the results of your bisect (I feel sorry for your laptop) I'm initially inclined to agree with your assessment of the regression here. The commit/PR referenced matches any error from the HEAD request and then assumes it's a path. It seems like the code could be more selective in the errors it matches (e.g. only match against object_store::Error::NotFound).

    @colinmarc Since you've already run a bisect, would you have time to quickly modify the DataFusion code to print (or panic, something) on the error condition so we can better understand if/what error may be happening there?
    https://github.com/apache/datafusion/pull/17364/files#diff-3f8fd00d313a1d20cfd060fd853f21df01318a1afbbdfdf1886d31c57291ff4cR255

    It seems @alamb's comment on that line may have been quite prescient.

  4. colinmarc commented on Oct 24, 2025

    @colinmarc
    ContributorAuthor

    Looks like transient DNS:

    head err: Generic { store: "S3", source: RetryError(RetryErrorImpl { method: HEAD, uri: Some(https://s3.us-east-1.amazonaws.com/bauplan-openlake-db87a23/iceberg/default/taxi_fhvhv_partitioned_2d9d2af4-7a0e-41c2-a0d3-2096eb1a2b9a/data/pickup_datetime_day%3D2022-06-27/dropoff_datetime_day%3D2022-06-27/00000-32-25d1dcfe-769c-41fd-8210-22f445173c6c.parquet), retries: 10, max_retries: 10, elapsed: 3.204595917s, retry_timeout: 180s, inner: Http(HttpError { kind: Connect, source: reqwest::Error { kind: Request, source: hyper_util::client::legacy::Error(Connect, ConnectError("dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: nodename nor servname provided, or not known" })) } }) }) }
    

    Pretty odd, and indeed maybe a macOS thing (my coworker was able to reproduce, but we don't see it in CI).

  5. added 2 commits that reference this issue on Oct 24, 2025
    937f256
    9d0f0bc
  6. added a commit that references this issue on Oct 24, 2025
    987f333
  7. added a commit that references this issue on Nov 2, 2025
    85150dd
  8. 16pierre commented on Feb 17, 2026

    @16pierre

    Even when matching error pattern object_store::Error::NotFound, silent data regressions are still possible - imagine an object-store implementation with relaxed data consistency guarantees, that can return transient NotFound errors for files recently written. From a Datafusion client perspective, it feels reasonable to expect a failfast by default rather than a data-regression in such cases - that was our understanding of the listing_table implementation before #17364.

    I wonder if there are ways to tweak the implementation of the quality-of-life PR #17364 so that it doesn't risk major silent data regressions - some clients have non-ambiguous call pattern where they expect a exactly single Parquet file read and not a directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions