Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/pr_build_linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -537,6 +537,7 @@ jobs:
- name: "expressions"
value: |
org.apache.comet.CometExpressionSuite
org.apache.comet.GenerateDocsSuite
org.apache.comet.CometSqlFileTestSuite
org.apache.comet.SqlFileTestParserSuite
org.apache.comet.CometExpressionCoverageSuite
Expand All @@ -558,6 +559,8 @@ jobs:
org.apache.comet.expressions.conditional.CometCoalesceSuite
org.apache.comet.expressions.conditional.CometCaseWhenSuite
org.apache.comet.CometRegExpJvmSuite
org.apache.comet.CometRegexParitySuite
org.apache.comet.expressions.CometRegexSuite
org.apache.comet.CometCodegenSuite
org.apache.comet.CometCoverageStatsSuite
org.apache.comet.CometCodegenSourceSuite
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/pr_build_macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ jobs:
- name: "expressions"
value: |
org.apache.comet.CometExpressionSuite
org.apache.comet.GenerateDocsSuite
org.apache.comet.CometSqlFileTestSuite
org.apache.comet.SqlFileTestParserSuite
org.apache.comet.CometExpressionCoverageSuite
Expand All @@ -239,6 +240,8 @@ jobs:
org.apache.comet.expressions.conditional.CometCoalesceSuite
org.apache.comet.expressions.conditional.CometCaseWhenSuite
org.apache.comet.CometRegExpJvmSuite
org.apache.comet.CometRegexParitySuite
org.apache.comet.expressions.CometRegexSuite
org.apache.comet.CometCodegenSuite
org.apache.comet.CometCoverageStatsSuite
org.apache.comet.CometCodegenSourceSuite
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,6 @@

## rlike

- See `string_funcs / regexp_replace` and the `CometRLike` notes (audited in PR [#4461](https://github.com/apache/datafusion-comet/pull/4461)). By default `CometRLike` routes through the JVM codegen dispatcher so Spark's own `Pattern` engine runs inside the Comet pipeline. The native path uses the Rust `regex` crate, which differs from Java's `Pattern` engine, so it is opt-in via `spark.comet.expression.RLike.allowIncompatible=true` and only applies when the pattern is a string literal.
- See `string_funcs / regexp_replace` and the `CometRLike` notes (audited in PR [#4461](https://github.com/apache/datafusion-comet/pull/4461)). `CometRLike` consults a plan-time whitelist (`CometRegex`) for non-null `UTF8_BINARY` literal patterns. Analyzer-admitted patterns (ASCII literals, simple character classes, greedy quantifiers, groups, alternation) run natively by default. Every other literal stays on the JVM codegen dispatcher unless `spark.comet.expression.RLike.allowIncompatible=true` forces the Rust path. Non-literal and NULL patterns, and Spark 4 non-default collations, always stay on the dispatcher. Anchors (`^`, `$`) and `.` / `\d` / `\w` / `\s` are out of subset because Java and Rust disagree on line terminators and Unicode character classes. Rust class set operators (`&&` / `~~` / `--`), unescaped class delimiters used as atoms or range endpoints, and counted, aggregate, or nested patterns over a conservative compile-size budget are also out of subset so they cannot silently change results, suppress Spark errors, or fail native plan construction.

[Spark Expression Support]: ../../user-guide/latest/expressions.md
72 changes: 49 additions & 23 deletions docs/source/user-guide/latest/compatibility/regex.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,20 @@ under the License.
Comet evaluates Spark regular-expression expressions (`rlike`, `regexp_replace`, `split`,
`regexp_extract`, `regexp_extract_all`, `regexp_instr`) two ways:

- **Codegen dispatcher (default)** — Spark's own `doGenCode` for the expression runs inside Comet's
- **Codegen dispatcher** — Spark's own `doGenCode` for the expression runs inside Comet's
Arrow-direct codegen dispatcher (the same dispatcher used by Comet's `ScalaUDF` codegen path).
This is 100% compatible with Spark, at the cost of one JNI round-trip per batch. It is enabled by
default (`spark.comet.exec.scalaUDF.codegen.enabled=true`); if the dispatcher is disabled, regex
expressions fall back to Spark.
expressions fall back to Spark. This is the default for every regex expression except an
in-subset `rlike` literal (see below).
- **Native (rust) engine** — the Rust [`regex`] crate, run natively with no JNI overhead. It is
faster but has different semantics from Java regex (see below), so it is **opt-in per expression**
via that expression's `allowIncompatible` flag. `rlike`, `regexp_replace`, `split`,
`regexp_extract`, and `regexp_extract_all` have a native implementation; `regexp_instr` does not
and always runs through the codegen dispatcher.
faster but has different semantics from Java regex (see below). For `rlike`, a plan-time
analyzer admits a conservative subset of `UTF8_BINARY` literal patterns and runs those natively
**by default**. Out-of-subset default-collation literals still require
`spark.comet.expression.RLike.allowIncompatible`. Non-literal patterns, NULL patterns, and
non-default collations stay on the codegen dispatcher even with that flag. Every other regex
expression still requires its own `allowIncompatible` flag. `regexp_instr` has no native
implementation and always runs through the codegen dispatcher.

| SQL | Native (rust) opt-in config |
| -------------------- | ----------------------------------------------------------- |
Expand All @@ -41,9 +45,14 @@ Comet evaluates Spark regular-expression expressions (`rlike`, `regexp_replace`,
| `regexp_extract_all` | `spark.comet.expression.RegExpExtractAll.allowIncompatible` |
| `split` | `spark.comet.expression.StringSplit.allowIncompatible` |

When the native path is opted in but a case has no native implementation (for example a non-scalar
`rlike` pattern, `regexp_replace` with a non-1 offset, or `regexp_extract` with a non-literal
pattern or idx), Comet routes that case through the codegen dispatcher.
`allowIncompatible` can force native execution for a non-null default-collation
literal. Non-default collations remain on the JVM dispatcher because the native
kernel does not implement Spark collation semantics. It is not needed for in-subset
literals, and it does not apply to non-literal or NULL patterns.

When the native path is selected but a case has no native implementation (for example a
non-scalar `rlike` pattern, `regexp_replace` with a non-1 offset, or `regexp_extract` with a
non-literal pattern or idx), Comet routes that case through the codegen dispatcher.

## Disabling Comet for individual regex expressions

Expand All @@ -63,19 +72,24 @@ the engine selector:

## Choosing an engine

| | Rust engine | Codegen dispatcher (default) |
| | Rust engine | Codegen dispatcher |
| -------------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **Compatibility** | Differs from Java regex (see below) | 100% compatible with Spark |
| **Compatibility** | In-subset `rlike` literals match Java regex; other patterns may differ (see below) | 100% compatible with Spark |
| **Feature coverage** | `rlike`, `regexp_replace`, `split`, `regexp_extract`, `regexp_extract_all` natively; `regexp_instr` via fallthrough | All regexp expressions (`rlike`, `regexp_extract`, `regexp_extract_all`, `regexp_instr`, `regexp_replace`, `split`) |
| **Performance** | Fully native, no JNI overhead | One JNI round-trip per batch (Arrow vectors stay columnar) |
| **Pattern support** | Linear-time subset only | All Java regex features (backreferences, lookaround, etc.) |
| **`rlike` default** | Used automatically for analyzer-admitted `UTF8_BINARY` literals | Used for every other `rlike` pattern |

The **Rust engine** is faster but cannot match Java regex semantics for every pattern. Opting in per
expression (for example `spark.comet.expression.RLike.allowIncompatible=true`) declares acceptance
of those differences.
The **Rust engine** is faster but cannot match Java regex semantics for every pattern. For `rlike`,
Comet therefore runs only the analyzer-admitted subset natively by default. Setting
`spark.comet.expression.RLike.allowIncompatible=true` can force native execution for a non-null
default-collation literal that the analyzer cannot prove equivalent, and declares acceptance of
any remaining differences. Non-default collations remain on the JVM dispatcher because the native
kernel does not implement Spark collation semantics. The other regex expressions still require
their own `allowIncompatible` flag.

The **codegen dispatcher** is the default and is enabled by `spark.comet.exec.scalaUDF.codegen.enabled`,
so it can be disabled globally to fall back to Spark for the regex family.
The **codegen dispatcher** is enabled by `spark.comet.exec.scalaUDF.codegen.enabled`, so it can be
disabled globally to fall back to Spark for out-of-subset regex expressions.

## Why the engines differ

Expand Down Expand Up @@ -131,13 +145,25 @@ Even where both engines accept a construct, the matching behavior is not always

## When the Rust engine is safe

For most ASCII-only, non-anchored patterns that use only literal characters, simple character classes, and
ordinary quantifiers, the two engines produce the same results. If you are confident your patterns fit this
shape and want to avoid the JNI overhead of the Java engine, switching to the Rust engine with
`allowIncompatible=true` is generally safe.

For anything that uses backreferences, lookaround, or relies on Java's specific Unicode or line-handling
defaults, use the Java engine.
Comet's plan-time analyzer admits a conservative whitelist of `rlike` literals and runs those on
the Rust engine by default: printable ASCII literals, simple ASCII character classes, greedy
quantifiers (`*`, `+`, `?`, `{n}`, `{n,}`, `{n,m}`), capturing and non-capturing groups, and
alternation. Anchors (`^`, `$`), `.`, `\d` / `\w` / `\s`, inline flags, lookaround, Rust-only
class set operators (`&&`, `~~`, `--`), unescaped class delimiters used as atoms or range
endpoints, and counted, aggregate, nested-group, or nested-quantifier patterns that exceed a
conservative compile-size budget stay on the Java engine. Any unrecognized construct also stays
on the Java engine.

For `regexp_replace`, `split`, `regexp_extract`, and `regexp_extract_all`, the native path is
still opt-in via `allowIncompatible`. If you are confident those patterns fit the same ASCII,
non-anchored shape, opting in is generally safe.

Native `rlike` operates on Arrow string values and assumes well-formed UTF-8.
Behavior for malformed string data supplied outside Arrow's StringArray invariants
has not been established.

For anything that uses backreferences, lookaround, or relies on Java's specific Unicode or
line-handling defaults, use the Java engine.

[`java.util.regex`]: https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
[`regex`]: https://docs.rs/regex/latest/regex/
Expand Down
6 changes: 3 additions & 3 deletions docs/source/user-guide/latest/expressions.md
Original file line number Diff line number Diff line change
Expand Up @@ -541,9 +541,9 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `like` | ✅ | Hybrid | |
| `not` | ✅ | Native | |
| `or` | ✅ | Native | |
| `regexp` | ✅ | Hybrid | Falls back by default; opt-in via allowIncompatible ([details](compatibility/regex.md)) |
| `regexp_like` | ✅ | Hybrid | Falls back by default; opt-in via allowIncompatible ([details](compatibility/regex.md)) |
| `rlike` | ✅ | Hybrid | Falls back by default; opt-in via allowIncompatible ([details](compatibility/regex.md)) |
| `regexp` | ✅ | Hybrid | In-subset literals run natively; others fall back by default ([details](compatibility/regex.md)) |
| `regexp_like` | ✅ | Hybrid | In-subset literals run natively; others fall back by default ([details](compatibility/regex.md)) |
| `rlike` | ✅ | Hybrid | In-subset literals run natively; others fall back by default ([details](compatibility/regex.md)) |

---

Expand Down
2 changes: 2 additions & 0 deletions native/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ chrono = { version = "0.4", default-features = false, features = ["clock"] }
futures = "0.3.32"
num = "0.4"
rand = "0.10"
# Before upgrading `regex`, complete the Java-parity fixture gate tracked in
# https://github.com/apache/datafusion-comet/issues/5813.
regex = "1.12.3"
thiserror = "2"
object_store = { version = "0.13.2", features = ["gcp", "azure", "aws", "http"] }
Expand Down
48 changes: 37 additions & 11 deletions spark/src/main/scala/org/apache/comet/GenerateDocs.scala
Original file line number Diff line number Diff line change
Expand Up @@ -57,17 +57,21 @@ object GenerateDocs {
* Spark-compatible path by default and the user can opt into a native path
* @param nativeOptInConfigKey
* the config key the user sets to opt into the native path
* @param conditionalNativeDefault
* whether some compatible instances run natively by default while other applicable instances
* retain the native opt-in behavior
* @param codegenDispatchFallback
* whether the serde mixes in `CodegenDispatchFallback`, meaning `unsupportedReasons` cases
* route through the JVM codegen dispatcher instead of falling back to Spark
*/
private case class ExprNotes(
private[comet] case class ExprNotes(
name: String,
compatibleNotes: Seq[String],
incompatibleReasons: Seq[String],
unsupportedReasons: Seq[String],
nativeOptIn: Boolean,
nativeOptInConfigKey: String,
conditionalNativeDefault: Boolean,
codegenDispatchFallback: Boolean)

private type CategoryNotes = Seq[ExprNotes]
Expand All @@ -87,6 +91,10 @@ object GenerateDocs {
serde.getUnsupportedReasons(),
optIn,
key,
conditionalNativeDefault = serde match {
case n: NativeOptInAvailable => n.hasConditionalNativeDefault
case _ => false
},
codegenDispatchFallback = serde.isInstanceOf[CodegenDispatchFallback])
}

Expand All @@ -100,6 +108,7 @@ object GenerateDocs {
// Aggregate serdes do not have a native opt-in path.
nativeOptIn = false,
nativeOptInConfigKey = CometConf.getExprAllowIncompatConfigKey(cls),
conditionalNativeDefault = false,
codegenDispatchFallback = false)

/**
Expand Down Expand Up @@ -389,22 +398,38 @@ object GenerateDocs {
}

private def writeExpressionCompatNotes(w: BufferedOutputStream, notes: CategoryNotes): Unit = {
w.write(renderExpressionCompatNotes(notes).getBytes)
}

/** Render expression compatibility notes as Markdown. Visible to package tests. */
private[comet] def renderExpressionCompatNotes(notes: Seq[ExprNotes]): String = {
val output = new StringBuilder
val sorted = notes.sortBy(_.name).filter { n =>
n.compatibleNotes.nonEmpty || n.incompatibleReasons.nonEmpty || n.unsupportedReasons.nonEmpty
}
for (n <- sorted) {
val name = n.name
w.write(s"\n## $name\n".getBytes)
output.append(s"\n## $name\n")
if (n.compatibleNotes.nonEmpty) {
w.write(
("\nThe following differences from Spark are always present and do not require" +
" any additional configuration:\n\n").getBytes)
val header = if (n.conditionalNativeDefault) {
"\nThe following cases use Comet's native implementation by default:\n\n"
} else {
"\nThe following differences from Spark are always present and do not require" +
" any additional configuration:\n\n"
}
output.append(header)
for (note <- n.compatibleNotes) {
w.write(s"- $note\n".getBytes)
output.append(s"- $note\n")
}
}
if (n.incompatibleReasons.nonEmpty) {
val header = if (n.nativeOptIn) {
val header = if (n.conditionalNativeDefault) {
"\nFor applicable cases that are not selected for native execution automatically," +
s" `$name` is evaluated in the JVM using Spark's own code-generated implementation" +
" (run inside the Comet pipeline) by default." +
s" Set `${n.nativeOptInConfigKey}=true` to explicitly select Comet's native" +
" implementation, which has the following differences from Spark:\n\n"
} else if (n.nativeOptIn) {
s"\nBy default, `$name` is evaluated in the JVM using Spark's own code-generated" +
" implementation (run inside the Comet pipeline), which matches Spark exactly." +
s" Set `${n.nativeOptInConfigKey}=true` to opt into Comet's native implementation" +
Expand All @@ -414,9 +439,9 @@ object GenerateDocs {
s" Set `spark.comet.expression.$name.allowIncompatible=true` to enable Comet" +
" acceleration despite these differences.\n\n"
}
w.write(header.getBytes)
output.append(header)
for (reason <- n.incompatibleReasons) {
w.write(s"- $reason\n".getBytes)
output.append(s"- $reason\n")
}
}
if (n.unsupportedReasons.nonEmpty) {
Expand All @@ -427,12 +452,13 @@ object GenerateDocs {
"\nThe following cases are not supported by Comet and always fall back to Spark," +
" regardless of any `allowIncompatible` setting:\n\n"
}
w.write(header.getBytes)
output.append(header)
for (reason <- n.unsupportedReasons) {
w.write(s"- $reason\n".getBytes)
output.append(s"- $reason\n")
}
}
}
output.result()
}

private def writeCastMatrixForMode(w: BufferedOutputStream, mode: CometEvalMode.Value): Unit = {
Expand Down
Loading
Loading