Skip to content

Add support for size expression #1926

Closed

Description

@andygrove

What is the problem the feature request solves?

Add support for Spark SQL size expression:

https://spark.apache.org/docs/latest/api/sql/index.html#size

From the documentation:

size(expr) - Returns the size of an array or a map. This function returns -1 for null input only if spark.sql.ansi.enabled is false and spark.sql.legacy.sizeOfNull is true. Otherwise, it returns null for null input. With the default settings, the function returns null for null input.

Describe the potential solution

No response

Additional context

No response

Activity

  1. modified the milestone: 0.9.0 on Jun 23, 2025
  2. dharanad commented on Jun 26, 2025

    @dharanad
    Contributor

    @comphead Can i pick this up ?

  3. comphead commented on Jun 26, 2025

    @comphead
    Contributor

    Feel free, yes

  4. dharanad commented on Aug 11, 2025

    @dharanad
    Contributor

    @comphead there is cardinality function in the DataFusion repository that I think we can reuse. I'm considering adding a new function size to datafusion-spark that aligns with Spark's semantics whilest reusing cardinality

  5. andygrove commented on Nov 6, 2025

    @andygrove
    MemberAuthor

    Here is a test that can be added to the CometFuzzTestSuite:

      test("select size of array") {
        val df = spark.read.parquet(filename)
        df.createOrReplaceTempView("t1")
        val cols = df.schema.fields.filter(_.dataType.isInstanceOf[ArrayType])
        for (col <- cols) {
          val sql = s"SELECT size(${col.name}) FROM t1 ORDER BY ${col.name}"
          if (usingDataSourceExec) {
            checkSparkAnswerAndOperator(sql)
          } else {
            checkSparkAnswer(sql)
          }
        }
      }
  6. andygrove commented on Nov 6, 2025

    @andygrove
    MemberAuthor

    Hi @dharanad. Are you still planning on working on this, or is it ok if we let someone else pick this up? Thanks!

  7. dharanad commented on Nov 11, 2025

    @dharanad
    Contributor

    @andygrove I overlooked this issue, i will spend sometime over this weekend

  8. modified the milestones: 0.12.0, 0.13.0 on Nov 11, 2025
  9. modified the milestones: 0.13.0, 0.14.0 on Jan 30, 2026
  10. modified the milestones: 0.14.0, 0.16.0 on Apr 25, 2026
  11. andygrove commented on May 14, 2026

    @andygrove
    MemberAuthor

    This was implemented in #2862

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestexpressionSpark expression related

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions