Skip to content

Optimizer rule 'push_down_projection' failed #6819

Closed

Description

@r4ntix

Describe the bug

This problem was found in ballista's test: https://github.com/apache/arrow-ballista/actions/runs/5430821519/jobs/9876790266?pr=834

❯ select l_returnflag, sum(l_extendedprice * 1) as sum_disc_price
            from lineitem
            group by l_returnflag
            order by l_returnflag;
push_down_projection
caused by
Internal error: Optimizer rule 'push_down_projection' failed, due to generate a different schema, original schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "lineitem" }), field: Field { name: "l_returnflag", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }, DFField { qualifier: None, field: Field { name: "sum_disc_price", data_type: Decimal128(38, 2), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }, new schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "lineitem" }), field: Field { name: "l_returnflag", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }, DFField { qualifier: None, field: Field { name: "sum_disc_price", data_type: Decimal128(25, 2), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

To Reproduce

datafusion = "27.0.0"

❯ CREATE EXTERNAL TABLE lineitem
STORED AS PARQUET
LOCATION '/Users/r4ntix/Workspace/github/arrow-datafusion/benchmarks/tpch-parquet/lineitem.parquet/part-0.parquet';
0 rows in set. Query took 0.006 seconds.

❯ describe lineitem;
+-----------------+-------------------+-------------+
| column_name     | data_type         | is_nullable |
+-----------------+-------------------+-------------+
| l_orderkey      | Int64             | NO          |
| l_partkey       | Int64             | NO          |
| l_suppkey       | Int64             | NO          |
| l_linenumber    | Int32             | NO          |
| l_quantity      | Decimal128(15, 2) | NO          |
| l_extendedprice | Decimal128(15, 2) | NO          |
| l_discount      | Decimal128(15, 2) | NO          |
| l_tax           | Decimal128(15, 2) | NO          |
| l_returnflag    | Utf8              | NO          |
| l_linestatus    | Utf8              | NO          |
| l_shipdate      | Date32            | NO          |
| l_commitdate    | Date32            | NO          |
| l_receiptdate   | Date32            | NO          |
| l_shipinstruct  | Utf8              | NO          |
| l_shipmode      | Utf8              | NO          |
| l_comment       | Utf8              | NO          |
+-----------------+-------------------+-------------+
16 rows in set. Query took 0.001 seconds.

❯ select l_returnflag, sum(l_extendedprice * 1) as sum_disc_price
            from lineitem
            group by l_returnflag
            order by l_returnflag;
push_down_projection
caused by
Internal error: Optimizer rule 'push_down_projection' failed, due to generate a different schema, original schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "lineitem" }), field: Field { name: "l_returnflag", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }, DFField { qualifier: None, field: Field { name: "sum_disc_price", data_type: Decimal128(38, 2), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }, new schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "lineitem" }), field: Field { name: "l_returnflag", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }, DFField { qualifier: None, field: Field { name: "sum_disc_price", data_type: Decimal128(25, 2), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

Expected behavior

No response

Additional context

No response

Activity

  1. jackwener commented on Jul 1, 2023

    @jackwener
    Member

    I'm working on it.

  2. self-assigned this
    on Jul 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions