Skip to content

file_row_index() / input_file_name() wrapped in an expression error at execution time (Projection missing from leaf-extraction allowlist) #27127

Closed as not planned

Description

@cool9850311

Update — please read this comment before starting work. Investigation refined three things in the original report below:

  • Root cause is Projection missing from the leaf-expression extraction allowlist in extract_leaf_expressions.rs:259. The RepartitionExec / all_columns interaction described below is the proximate mechanism, not the root cause.
  • Simpler minimal repro, no ORDER BY or LIMIT needed: SELECT file_row_index() + 1 FROM big;. Note a bare LIMIT n is pushed into the scan and masks the bug, so a regression test using LIMIT 3, or a table smaller than batch_size, will pass against broken code.
  • Wider scope: input_file_name() is affected identically, and get_field() — the third MoveTowardsLeafNodes function — silently loses Parquet nested-column pruning from the same cause. A fix should cover all three, keyed off ExpressionPlacement rather than per-function downcasts.

The title has been updated accordingly. Everything below is the original report, left intact.


Describe the bug

file_row_index() works when called bare, but fails at execution time as soon
as the call is wrapped in any enclosing expression — on the same table, same data,
default configuration:

Execution error: file_row_index() is source dependent and cannot be evaluated directly
SELECT file_row_index(),     column1 FROM big ORDER BY column1 LIMIT 3;  -- ok
SELECT file_row_index() + 1, column1 FROM big ORDER BY column1 LIMIT 3;  -- Execution error

The marker UDF has to be rewritten into the file scan's projection. That rewrite
does not happen when a RepartitionExec sits between the DataSourceExec and the
ProjectionExec, and whether it does is decided by the round-robin heuristic in
enforce_distribution, which compares the estimated row count against
batch_size:

// datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:1167
Precision::Exact(n_rows) => n_rows > batch_size,

Three independent observations line up with that heuristic being the trigger:

change result
20 000-row table (default batch_size = 8192) fails
same query on a 3-row table ok
same 20 000-row table, SET datafusion.execution.batch_size = 30000 ok
same 20 000-row table, SET datafusion.execution.target_partitions = 1 ok

So a query that works during development against a small table starts failing once
the table grows past one batch.

On the documented behaviour

scalar_functions.md
says:

This function is intended to be rewritten at file-scan time. If the input file is
not known (for example, if this function is evaluated outside a file scan, or was
not pushed down into one), direct evaluation returns an error.

That documents the error mechanism for a call with no file-scan context. In this
report the call is written directly against a file scan, and the bare form works
on the same table
— so the deciding factor is the expression shape plus a
physical-planning decision, not where the user put the call. The docs' own example
passes only because it has 3 rows.

For precedent: #23531 was the same "... is source dependent and cannot be evaluated directly" error surfacing from a different pushdown gap
(input_file_name() in a Parquet RowFilter), and it was treated and fixed as a
bug.

To Reproduce

In datafusion-cli, stock configuration — no SET required:

COPY (SELECT value AS column1 FROM range(0, 20000))
TO '/tmp/df_fri_repro/big.parquet'
STORED AS PARQUET;

CREATE EXTERNAL TABLE big
STORED AS PARQUET
LOCATION '/tmp/df_fri_repro/big.parquet';

-- works: 0, 1, 2
SELECT file_row_index(), column1 FROM big ORDER BY column1 LIMIT 3;

-- BUG: Execution error
SELECT file_row_index() + 1, column1 FROM big ORDER BY column1 LIMIT 3;

Which expression shapes break:

shape result
file_row_index() ok
file_row_index() AS r ok
file_row_index() + 1 fails
arrow_typeof(file_row_index()) fails
SELECT r + 1 FROM (SELECT file_row_index() AS r FROM big) fails — the subquery is flattened, so this is not a workaround

It is not arithmetic specifically; any wrapping breaks it. The two escapes are
keeping the call bare or setting target_partitions = 1, which gives up all
parallelism.

EXPLAIN on the failing query succeeds, and shows the repartition between the
scan and the projection:

ProjectionExec
  RepartitionExec
    DataSourceExec

Expected behavior

Either:

  1. the rewrite survives a RepartitionExec between scan and projection — e.g. push
    the projection below it, or perform the rewrite before distribution enforcement —
    so that file_row_index() + 1 behaves like file_row_index(); or
  2. if successful pushdown is a hard precondition, reject the query at plan time
    with an actionable message, instead of accepting the plan, letting EXPLAIN
    succeed, and failing once execution has started.

Silently depending on row count vs batch_size is the part that seems worth fixing
either way.

Additional context

Willingness to contribute

I plan to contribute a fix for this bug independently.

Activity

  1. cool9850311 commented on Nov 5, 2026

    @cool9850311
    ContributorAuthor

    take

  2. cool9850311 commented on Nov 5, 2026

    @cool9850311
    ContributorAuthor

    I dug into the root cause and found the scope is wider than my original report, plus one imprecision in it that I want to correct. Summary: the proximate mechanism I described (a RepartitionExec blocking the projection) is right, but the root cause is one node type missing from the leaf-expression extraction allowlist, and it affects two more functions.

    Correction to my repro: plain LIMIT n masks the bug

    My first test matrix put LIMIT 3 on every query and almost everything passed, which nearly led me to the wrong conclusion that only the ORDER BY shape breaks. A bare LIMIT n is pushed into the scan, so the row estimate drops to n, falls below batch_size, and no RepartitionExec is inserted:

    query result
    SELECT file_row_index() + 1 FROM big LIMIT 3 ok — limit pushed into scan, bug masked
    SELECT file_row_index() + 1 FROM big error
    SELECT file_row_index() + 1, column1 FROM big ORDER BY column1 LIMIT 3 error
    SELECT file_row_index() + 1 FROM big OFFSET 5 LIMIT 3 error

    So the minimal repro is simpler than the one I filed — no ORDER BY or LIMIT needed:

    SELECT file_row_index() + 1 FROM big;   -- 20 000 rows -> error

    This also means "tables larger than one batch" in my original title is a necessary but not sufficient condition, so I have retitled the issue.

    This matters for whoever writes the regression test: a fixture with LIMIT 3, or with fewer than batch_size rows, will pass against broken code.

    Root cause: Projection is not in the extraction allowlist

    DataFusion already has the general machinery for this — ExtractLeafExpressions / PushDownLeafProjections in datafusion/optimizer/src/extract_leaf_expressions.rs — which pulls ExpressionPlacement::MoveTowardsLeafNodes sub-expressions out into a __datafusion_extracted_N projection that the source can absorb. And file_row_index() already declares that placement (functions/src/core/file_row_index.rs:85), as does input_file_name().

    The same wrapped shape works in a WHERE clause, because Filter is on the allowlist:

    SELECT column1 FROM big WHERE file_row_index() + 1 = 3;   -- ok
    Filter: __datafusion_extracted_1 + Int64(1) = Int64(3)
      Projection: file_row_index() AS __datafusion_extracted_1, big.column1
        TableScan: big projection=[column1]
    

    Note this plan contains a RepartitionExec too and still works, because by then the marker is an ordinary column and the all_columns gate no longer applies.

    In a SELECT list there is no extraction at all, because of the allowlist in extract_from_plan (extract_leaf_expressions.rs:259):

    if !matches!(
        &plan,
        LogicalPlan::Aggregate(_)
            | LogicalPlan::Filter(_)
            | LogicalPlan::Sort(_)
            | LogicalPlan::Limit(_)
            | LogicalPlan::Join(_)
    ) {
        return Ok(Transformed::no(plan));
    }

    Measured, same marker + 1 shape in each node type:

    node type on allowlist? result
    Filter (WHERE) yes ok
    Sort (ORDER BY) yes ok
    Aggregate (sum/GROUP BY/HAVING) yes ok
    Join (ON) yes error — see below
    Window (OVER) no error
    Projection (SELECT list) no error — this issue

    The module docs give the rationale for excluding Projection:

    The split has no use there: the source absorbs the leaf expressions of the original projection just as well.

    That holds only while the projection actually reaches the source. Physical planning then inserts the RepartitionExec per n_rows > batch_size, RepartitionExec::try_swapping_with_projection refuses projections that are not all_columns, and nothing re-establishes the assumption.

    This also explains why the bare call works: a bare file_row_index() projection is already in the shape the source can absorb.

    Scope is wider than the title

    input_file_name() is affected identically (all without LIMIT):

    shape result
    input_file_name() ok
    input_file_name() || 'x' error
    upper(input_file_name()) error
    input_file_name() || CAST(file_row_index() AS VARCHAR) error

    get_field() is the third member of the MoveTowardsLeafNodes family, and it silently loses Parquet nested-column pruning rather than erroring, because it is evaluable. On a 20 000-row table with a struct column:

    query scan's physical projection reads
    SELECT s['a'] FROM t projection=[get_field(s@0, a) as v] just the a leaf
    SELECT s['a'] + 1 FROM t projection=[s] the whole struct

    Same root cause, confirmed by removing the repartition — pruning comes back:

    setting scan projection for SELECT s['a'] + 1
    default (RepartitionExec present) projection=[s]
    target_partitions = 1 projection=[get_field(s@0, a) + 1 as v]
    batch_size = 30000 projection=[get_field(s@0, a) + 1 as v]

    So the general statement is: any MoveTowardsLeafNodes expression wrapped in another expression loses leaf pushdown once a RepartitionExec is inserted. For the two marker UDFs that is an execution error; for get_field it is a silent performance regression in what is arguably the extraction rule's headline feature.

    Not the same gap as #23531

    datafusion.execution.parquet.pushdown_filters makes no difference here — the projection case fails with it both on and off — so this is a distinct gap from #23531 / #23638.

    Suggested fix direction

    Extend extraction to projections whose expressions contain a MoveTowardsLeafNodes sub-expression, turning

    Projection: file_row_index() + 1 AS v, column1
    

    into

    Projection: __datafusion_extracted_1 + 1 AS v, column1
      Projection: file_row_index() AS __datafusion_extracted_1, column1
    

    The inner pure extraction projection is absorbable by the source (exactly the shape that works today), and the outer one references only columns. This reuses the already-accepted design from the Filter path and needs no loosening of the all_columns guard in RepartitionExec.

    Two constraints for whoever implements it:

    • The allowlist comment excludes node types whose output schema depends on their expressions, and Projection is one of those. The split must keep the outer projection's alias list and output field names byte-identical, changing only the inner marker into a column reference.
    • Detection should go through ExpressionPlacement::MoveTowardsLeafNodes rather than another try_downcast_func chain. fix: Handle input_file_name() pushdown into ParquetSource with filter pushdown enabled #23638 added a hardcoded chain and its author noted: "If we keep adding this sort of metadata functions, we might want a better API to detect them" — that API now exists and all three functions already declare it, so keying off placement also covers future metadata functions.

    Two separate gaps found along the way

    These are distinct from this issue and I don't think they belong in the same fix. Happy to file them separately if useful:

    1. Join is on the allowlist but still fails. Extraction routes each extracted sub-expression to an input by checking which input's schema holds all of the expression's column references. A marker call has no column references, so it cannot be routed. Pre-extracting it in a subquery works:
      SELECT count(*) FROM (SELECT file_row_index() AS r, column1 FROM big) a
      JOIN big b ON a.column1 = b.column1 AND a.r >= 0;   -- ok
    2. file_row_index() does not work on CSV or JSON at all — even the bare form errors, independently of row count or target_partitions (verified on a 3-row file and with target_partitions = 1). The only production caller of the rewrite is the Parquet source. input_file_name() bare does work on CSV. So file_row_index() is effectively Parquet-only, which the documentation does not say.

    All of the above verified on c120619de (48.0.0-rc2-4901-gc120619de).

  3. changed the title [-]`file_row_index()` wrapped in an expression errors at execution time on tables larger than one batch[/-] [+]`file_row_index()` / `input_file_name()` wrapped in an expression error at execution time (`Projection` missing from leaf-extraction allowlist)[/+] on Nov 5, 2026
  4. cool9850311 commented on Nov 5, 2026

    @cool9850311
    ContributorAuthor

    Withdrawing this report; refiling the investigation in a cleaner, consolidated form.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions