Skip to content

feat: run rlike natively by default for Java-equivalent literal patterns - #5415

Merged
sunchao merged 12 commits into
apache:mainfrom
sam-1112:rlike-native-compatible-subset-5351
Sep 16, 2026
Merged

sunchao merged 12 commits into
apache:mainfrom
sam-1112:rlike-native-compatible-subset-5351

Conversation

@sam-1112

@sam-1112 sam-1112 commented Aug 22, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #5351.

Rationale for this change

rlike already has a native kernel, but the Rust regex implementation is treated as potentially incompatible with Java regex and is therefore opt-in. Before this PR, literal patterns ran through the JVM codegen dispatcher by default unless spark.comet.expression.RLike.allowIncompatible=true.

#4310 correctly concluded that the Rust engine cannot be made fully compatible with Java regex, so this PR does not change the engine globally. Instead, it introduces a deliberately restricted, conservative whitelist for plan-time UTF8_BINARY literal patterns.

Patterns admitted by the whitelist use native execution by default and are covered by Java-versus-Rust differential tests. Out-of-subset non-null literals preserve the existing dispatcher / allowIncompatible behavior. Non-literal and NULL patterns use the dispatcher; NULL is never considered a usable native pattern.

What changes are included in this PR?

  • Add CometRegex, a recursive-descent parser implementing a conservative whitelist rather than relying on substring matching. Unrecognized constructs are Incompatible.
    • Admitted: printable ASCII literals, simple ASCII classes, greedy * + ? {n} {n,} {n,m}, capturing and non-capturing groups ((?:…)), alternation, and escaped metacharacters, provided counted and nested forms remain within conservative native compile-size and depth budgets.
    • Rejected: ^ $ . \d \w \s, lookaround, backrefs, possessive/lazy quantifiers, inline flags, \p / \u / \0, nested classes, Rust-only class set operators (&&, ~~, --), unescaped ] class atoms or range endpoints, raw [ range endpoints, counted or nested patterns exceeding the conservative compile budget, non-ASCII patterns, and non-default Spark 4 collation on either operand.
    • Compile-budget heuristics: individual counted bounds above 256, aggregate structural expansion above 4096, and group nesting deeper than 32 remain on the JVM dispatcher. These limits are conservative structural estimates, not a compiled-byte guarantee. For unbounded {n,}, a lower bound of zero still retains the inner expression's compilation cost; exact-zero {0} and {0,0} repetitions remain distinct.
  • CometRLike consults the analyzer. In-subset literals become Compatible() with no nativeOptIn hint and convert to native without opt-in.
  • Out-of-subset non-null default-collation literals stay Compatible(nativeOptIn = …) and convert to the dispatcher unless allowIncompatible=true.
  • Non-literal and NULL patterns always convert to the dispatcher (literalPattern only matches a non-null UTF8String literal).
  • allowIncompatible can force native execution for a non-null default-collation literal. Non-default collations remain on the JVM dispatcher because the native kernel does not implement Spark collation semantics.
  • Document that native rlike operates on Arrow string values and assumes well-formed UTF-8; behavior outside Arrow's StringArray invariants has not been established.
  • Update compatibility/regex.md, expressions.md (rlike / regexp / regexp_like), and the rlike expression-audit note.
  • This first PR covers rlike only. It does not add pattern rewriting such as (?-u), propagate collation into the native kernel, or change regexp_replace, split, regexp_extract, or regexp_extract_all.

A conservative whitelist identifies a deliberately restricted subset for which Java and Rust find semantics are expected to agree and are covered by differential parity tests. It is not a formal proof of equivalence.

How are these changes tested?

  • CometRegexSuite: admit / reject coverage, including:
    • lexer-boundary cases such as [(?=], \\d, and [.];
    • scanner edges such as \A, \Z, {2,1}, and an unclosed (;
    • Rust-only character-class set operators;
    • leading ] and raw [ class-range boundaries;
    • counted-expansion limits and nested-group depth;
    • aggregate compile-budget accounting;
    • rejection of the {0,} regression (([^;]{256}){0,}){256};
    • continued admission of the exact-zero {0} and {0,0} controls.
  • CometRegexParitySuite: every pattern in the admitted corpus × ASCII, non-ASCII, control-character, exotic-whitespace, newline, empty, and NULL subjects, with native results compared against java.util.regex.Pattern.find. The subject corpus includes TAB, VT, FF, NUL, DEL, NEL, NBSP, BOM, U+2028, and U+2029. The suite also asserts that EXPLAIN output does not contain JVM codegen dispatcher: rlike. The multi-batch corpus contains 5000 rows with batch size 64. NULL subjects match Spark; a NULL pattern is not part of this native corpus.
  • CometRegExpJvmSuite: routing coverage for:
    • native default and dispatcher default;
    • explicit incompatible opt-in;
    • non-literal and NULL patterns;
    • dispatcher-disabled native execution and fallback;
    • invalid regex handling;
    • Java-only patterns with allowIncompatible=true, including expected native compilation failures for unsupported Rust constructs;
    • Spark 4 collation on the subject and pattern;
    • non-default collation on either operand remains on the JVM dispatcher, including when allowIncompatible=true;
    • Rust-only class operations and Java/Rust class-boundary differences;
    • patterns exceeding native compile limits.
  • The {0,} regression is verified against Spark using non-foldable input, including NULL, and remains on the JVM dispatcher.
  • SQL file rlike_auto_native.sql verifies default-configuration result equality.
  • Validated locally with Spark 3.5.9 and 4.1.3 on JDK 17.
  • After the review follow-up, the targeted CometRegexParitySuite, CometRegExpJvmSuite, and CometRegexSuite completed successfully on Spark 4.1 / Scala 2.13 and Spark 3.5.9 / Scala 2.12:
All tests passed.
BUILD SUCCESS

The two allowIncompatible collation regressions are Spark 4-only (assume(isSpark40Plus)) and are skipped on 3.5.

Other supported Spark and Scala profiles are covered by CI.

Benchmark

CometRegExpBenchmark, 1,048,576 rows, Apple M4, JDK 17.

Native vs Spark for in-subset patterns

After this PR, these patterns use native execution by default.

Pattern Spark Native Speedup vs Spark
[0-9]+ 397 ms 84 ms 4.7X
abc|def|ghi 2304 ms 80 ms 28.8X
[a-zA-Z][0-9]+ 1075 ms 125 ms 8.6X
(ab){2,} 1114 ms 77 ms 14.5X

Default Comet execution before and after this PR

These measurements use identical in-subset queries. Both modes were measured on the PR base commit in the same run: the JVM dispatcher represents the pre-PR default, while allowIncompatible=true selects the same native kernel that this PR now chooses automatically.

Pattern Base: JVM dispatcher Native Speedup vs dispatcher
[0-9]+ 384 ms 84 ms 4.6X
abc|def|ghi 2303 ms 80 ms 28.8X
[a-zA-Z][0-9]+ 1070 ms 125 ms 8.6X
(ab){2,} 1091 ms 77 ms 14.2X

On this ASCII REPEAT workload, the pre-PR dispatcher is essentially Spark-cost. Switching the in-subset default from the dispatcher to native execution is the performance improvement that matters for this PR.

Out-of-subset control

\d+ remains outside the automatic-native subset because Java and Rust differ in their Unicode digit semantics. It uses the JVM dispatcher by default; allowIncompatible=true explicitly selects the native path.

The benchmark SQL preserves the backslash so Spark receives \d+ rather than parsing it as d+.

Pattern Spark Default Comet: JVM dispatcher Opt-in native Native speedup vs dispatcher
\d+ 373 ms 365 ms 77 ms 4.7X

The corrected JVM-dispatcher measurement is close to Spark cost, as expected. The explicitly opted-in native path is approximately 4.7X faster than the dispatcher on this ASCII workload.

The native implementation matches over Arrow buffers and is expected to avoid the dispatcher's per-row toString() / Matcher allocations. This allocation difference was not measured with JFR or a GC log in this PR.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Reviewed the complete 13-file diff from 367ca64be4e62dd10dc4c932de9ca738355a45fa to 86729123819387bbfa132c5c81956f82ea60b193, including the live discussion and CI state. Five independent full-diff passes were followed by independent source and executable verification of the retained cases.

The intended optimization is narrowly scoped to default-native RLike literals. Three pattern-specific P2 regressions remain: Rust character-class set operations change results, leading-] ranges independently change results, and Java-valid patterns can be admitted despite exceeding Rust's compilation limits. Each is detailed inline; none requires opting into incompatible execution.

Prior state and problem

The native RLike kernel already existed, but Java and Rust regex syntax and semantics differ. At the base revision, ordinary literal patterns used Spark's own generated regex implementation through the JVM dispatcher by default; native execution required spark.comet.expression.RLike.allowIncompatible=true. If the dispatcher was disabled, the default expression could fall back to Spark.

This preserves Java behavior but leaves JNI/dispatcher and per-row Java regex costs on patterns that both engines can evaluate equivalently. The PR aims to remove those costs for an automatically recognized subset while keeping other regex expressions and explicitly incompatible execution behavior separate.

Design approach

CometRegex introduces a recursive-descent admission parser rather than a substring-based filter. It recognizes printable ASCII literals, selected character classes, ordinary greedy quantifiers, capturing/non-capturing groups, and alternation. Unrecognized syntax is rejected from automatic native selection; anchors, wildcard dot, shorthand classes, lookaround, backreferences, and flags are intentionally outside the subset.

CometRLike additionally requires a non-null literal and default string collation on both operands for automatic selection. The existing native kernel and Cargo lock are unchanged, and the pattern is serialized without rewriting. Consequently, the admission decision must establish both equivalent Boolean matching and native compilability.

Correctness / compatibility analysis

The retained cases violate two different parts of that admission contract:

  • [a~~b] and [a-z--b] are admitted, but Rust interprets class set operations that Java does not interpret the same way. Concrete non-null inputs produce opposite Booleans.
  • []-a] is also admitted, without either set operator. Java treats the initial ] as a range endpoint; Rust treats this spelling differently. Both positive and negated classes can add or remove matching rows.
  • [^;]{20000}, a{1000000}, nested counted repetitions, and sufficiently nested groups are admitted even though default Rust compilation rejects them. Native plan construction propagates that rejection instead of returning to the dispatcher.

Verification used the unmodified pinned scanner, the exact regex 1.13.1 dependency graph with archive/source checksums matched to Cargo.lock, and real Spark 3.5.9 and 4.0.4 expressions with non-foldable input. The retained cases were checked in both interpreted evaluation and forced generated projections; NULL-subject controls were also checked. These are component/runtime probes plus source-traced integration, not a claimed full Comet/JNI query run.

The unchanged four-test CometRegexSuite passed in isolated builds on Scala 2.12.18 and 2.13.17, and suite-registration and diff-whitespace checks passed. The existing examples do not cover the counterexamples above. CI started during this review: at the 17:28 UTC refresh, CodeQL and Linux/macOS lint checks had succeeded, while native/JVM builds, Rust tests, Java lint, and the benchmark check were still running. No completed full integration-suite result is claimed.

Key design decisions

  • Keep automatic admission distinct from the existing incompatible opt-in. The latter remains an explicit request to use the Rust implementation for applicable literals, not an equivalence guarantee.
  • Check both operand collations for the automatic route. Spark 3.x uses the existing no-collation shim; Spark 4.x uses the existing collation-aware type checks. Explicit opt-in can bypass the automatic equivalence gate.
  • Exclude NULL and non-literal patterns from native applicability. They continue through the dispatcher, or fall back when the dispatcher is unavailable.
  • Preserve the generic expression-enabled gate. Disabling RLike itself still prevents this serializer from converting it; disabling only the dispatcher no longer disables eligible native RLike expressions.

Implementation sketch

The serializer extracts a UTF8String literal, checks the operand types, and calls CometRegex.supportLevel. An admitted pattern becomes Compatible() without a native-opt-in hint. Conversion then emits the existing RLike protobuf through createBinaryExpr; the native builder passes the unchanged literal to RLike::try_new, which uses Regex::new and is_match.

Other applicable literals retain their opt-in hint and default JVM dispatcher route. The patch also adds analyzer and parity suites, expands routing/NULL/collation tests, registers the new suites in both OS workflows, adds SQL cases, and updates documentation and benchmark mode selection. No native engine or dependency change accompanies the newly automatic route.

Behavioral changes worth calling out

For admitted literals, default Comet execution now uses the native engine even with the dispatcher disabled. That is the intended performance change, but it also makes any admission mistake observable without a user accepting regex incompatibility.

Out-of-subset literals continue to use the JVM by default, and non-literal/NULL patterns do not become native. Other regex functions retain their existing routing rules. The compile-limit cases are a change from successful default JVM evaluation to native construction failure; the two character-class cases are silent result changes, not merely different match spans or capture groups.

Suggested improvements

Tighten class admission for both Rust-only set operators and the independent leading-] range boundary. Add positive and negated differential examples, including subjects that distinguish literal operator characters and range interiors, and assert that rejected patterns retain JVM routing by default.

Make native compilability part of the admission contract, with conservative aggregate-expansion/depth bounds or a reliable validation/fallback path. A per-integer bound or source-length limit alone does not cover nested repetition expansion. Add the concrete size/depth cases to routing and error-behavior regression coverage, then verify the full parity/routing suites on the supported Spark profiles as CI completes.

Comment on lines +223 to +225
if (startsWith("&&") || peek == '[') {
return false
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Exclude Rust-only class operators from automatic native routing

This gate rejects && but admits ~~ and subtraction forms such as [a-z--b]. The pinned analyzer returns Compatible for [a~~b]; Spark 3.5.9/4.0.4 return true on subject ~, while the locked Rust regex 1.13.1 returns false because ~~ is symmetric difference. Likewise, [a-z--b] matches b in Spark but not Rust. I verified the Spark results in both interpreted and generated evaluation with a non-foldable input. Since CometRLike.convert now sends these literals to the unchanged native matcher without allowIncompatible=true, existing projections and filters silently change results. Keep these class forms outside automatic admission and add differential/routing regressions before selecting native for them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — the first revision only rejected &&, so [a~~b] / [a-z--b] were wrongly admitted and would have taken the native path without allowIncompatible.

7e51a22 now treats && / ~~ / -- inside a character class as incompatible. Those literals stay on the JVM dispatcher by default. Single ~ in a class (e.g. [a~b]) is still admitted; it is a Java literal, not a Rust set operator.

Regressions:

  • CometRegexSuite: analyzer returns Incompatible for [a~~b], [^a~~b], [a-z--b], [^a-z--b], [a&&b]
  • CometRegExpJvmSuite: EXPLAIN still shows the dispatcher, and checkSparkAnswerAndOperator matches Spark on a non-foldable column (~, a, b, x, and b/a/z/- for the subtraction form), including NULL

No runtime compile-fallback; these stay off automatic native.

Comment on lines +226 to +230
val ranging = lastAtom.isDefined && peek == '-' && peekOffset(1).exists(_ != ']')
if (ranging) {
consume() // '-'
parseClassAtom() match {
case Some(end) if end >= lastAtom.get =>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Reject ranges starting at an unescaped leading closing bracket

The initial literal ] is stored in lastAtom and then accepted as a range endpoint. For the admitted pattern []-a], Java/Spark interpret the range from ] through a, but Rust treats this spelling as the literals ], -, and a. On both Spark 3.5.9 and 4.0.4, interpreted and generated evaluation return true for _ and false for -; the locked native regex returns the opposite results. The negated form also differs. This default-native wrong-answer case contains neither ~~ nor --, so rejecting set operators alone will not fix it. Reject this class-boundary form, or normalize it only after establishing equivalent semantics, and cover both positive and negated variants.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — this is a separate Java/Rust split, not covered by rejecting ~~ / --.

Java treats a leading ] as a literal (so []-a] is the range ]–a), while Rust reads it as the closer and then the literals ], -, a. Same mismatch on the negated form. That’s why _ vs - flip between Spark and native.

7e51a22 now never treats an unescaped ] as a class atom or range endpoint, so []-a] and [^]-a] stay off automatic native. Escaped ] ([\\]]) is still admitted.

Tests:

  • analyzer: Incompatible for []-a], [^]-a], []], [^]]
  • routing: dispatcher + Spark-equal answers on _, -, ], a, z, and NULL

Comment on lines +50 to +52
val scanner = new Scanner(pattern)
if (scanner.parseExpr() && !scanner.remaining) {
Compatible()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve JVM routing for patterns beyond native compile limits

Successful syntax scanning does not ensure Regex::new can compile the pattern. For example, this analyzer admits [^;]{20000} and a{1000000}; real Spark 3.5.9/4.0.4 evaluate them on a column containing a as false, but the locked Rust engine rejects both with Compiled regex exceeds size limit of 10485760 bytes. Nested counted repetitions have the same problem, and 251 nested groups exceed Rust's separate depth limit while succeeding in Spark. The unchanged RLike::try_new and native builder propagate these errors from plan creation, with no dispatcher retry, so previously successful default-config queries now fail without opt-in. Require conservative native size/depth applicability or a reliable compilation-validation/fallback path before returning Compatible.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep — scanning the syntax isn’t the same as “Rust can compile this.” Spark is happy with [^;]{20000} / a{1000000} / nested {n} / deep groups; the locked regex crate then dies at plan time (Compiled regex exceeds size limit / nest depth) and we have no dispatcher retry. That’s a default-config regression.

Didn’t add a compile-then-fallback path here. 7e51a22 just refuses those at plan time:

  • {n} bigger than 256
  • nested counted product over 4096 ((a{100}){100})
  • more than 32 nested groups

They stay on the JVM dispatcher. a{256} and 32-deep groups still go native. Analyzer + EXPLAIN/Spark-equal tests cover the cases you listed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] {0,} still bypasses the compile-size budget

Rechecked at 0f98989095dee698db71b366998e112bdd1e15a6. The aggregate checks address the earlier sibling cases, but (([^;]{256}){0,}){256} is still classified as Compatible. The {0,} branch returns the lower bound 0, so multiplyWithinBudget reduces the inner cost from 256 to 1. Unlike {0} or {0,0}, this is unbounded repetition and the inner expression still has to be compiled.

I reproduced this with the unmodified current scanner on Scala 2.12 and 2.13. Spark 3.5.9 and 4.0.4 both return true for a non-foldable subject containing a, in interpreted evaluation and forced generated projections. The exact locked Rust regex 1.13.1 instead rejects the pattern with Compiled regex exceeds size limit of 10485760 bytes. The equivalent (([^;]{256})*){256} is correctly rejected by the scanner.

Because the admitted form now selects native execution without opt-in and the native builder propagates the compilation error, this still changes a successful default-config query into a failure. Could we retain the inner compilation cost for unbounded {0,} separately from exact-zero repetitions and add this case to the dispatcher-routing regressions? These checks used real Spark expressions and the locked regex engine plus source-traced native routing, not a full Comet/JNI query run.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that was a real gap. {0,} was using the lower bound 0 as the expansion multiplier, so multiplyWithinBudget collapsed the inner cost to 1 and admitted (([^;]{256}){0,}){256}. That pattern is Java-valid but rejected by the locked Rust regex crate, so the native path would fail a query that Spark currently succeeds.

The unbounded {n,} branch now uses math.max(1, n), so {0,} keeps the inner compile cost. {0} and {0,0} stay exact-zero and are still Compatible.

Coverage:

  • analyzer: (([^;]{256}){0,}){256} is Incompatible; {0} / {0,0} remain Compatible
  • routing: the unbounded pattern stays on the JVM codegen dispatcher and matches Spark
  • exact-zero counterparts stay native and match Spark

Pushed in c62b529a7.

}
if (peek == '\\') {
parseEscape(inClass = true)
} else if (isPrintableAscii(peek)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve Spark errors for raw [ range endpoints

The [ guard in parseClass runs before the range branch consumes -, so this helper can still accept an unescaped [ as the range endpoint. The exact scanner returns Compatible for [@-[] (and [^@-[]). With a non-foldable string input, real Spark 3.5.9 and 4.0.4 reject these patterns: interpreted RLIKE raises an invalid-pattern error, and forced generated-projection construction raises PatternSyntaxException for an unclosed character class. The locked Rust regex 1.13.1 instead compiles [@-[] and matches A. Because CometRLike.convert now trusts this admission result without incompatible opt-in, a malformed-pattern failure becomes a successful predicate under the default native route. Please reject raw [ in the range-endpoint helper as well; the escaped [@-\[] control is valid and equivalent in both engines.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — the class-loop [ guard runs before - is consumed, so it never saw [@-[] / [^@-[] as a nested class. Java rejects those as an unclosed class; Rust compiles them and matches A. That’s a default-native “error becomes true/false” case. This commit rejects an unescaped [ in parseClassAtom as well, so it cannot be a range endpoint. Escaped [@-\[] is still admitted and covered by Java/native parity.
Tests:

  • analyzer: Incompatible for [@-[] / [^@-[]; Compatible for [@-\[]
  • routing: dispatcher + Spark regex error on a non-foldable column (@, [, A, NULL)

runInSubsetModes(p.name, v, query)
}
}
outOfSubsetPatterns.foreach { p =>
val query = s"select c1 rlike '${p.pattern}' from parquetV1Table"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the backslash in the out-of-subset benchmark SQL

With the default spark.sql.parser.escapedStringLiterals=false, interpolating the new \d+ pattern here makes Spark parse it as d+ (reproduced with Spark 3.5.9 and 4.0.4). CometRegex admits d+, so the case labelled Comet (Exec, JVM regex) selects the automatic native branch just like the opted-in case. The generated numeric strings also match none of the rows instead of every row. This leaves the new four-way comparison without a JVM-dispatcher measurement. Please preserve the backslash with an escaped or raw SQL literal, as the parity suite already does, before passing this query to runOutOfSubsetModes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in c62b529a7

The out-of-subset pattern is now escaped before being interpolated into the SQL query:

val escapedPattern = p.pattern.replace("\\", "\\\\")
val query = s"select c1 rlike '$escapedPattern' from parquetV1Table"

@andygrove

Copy link
Copy Markdown
Member

Note on this review: this was generated by an LLM (Claude Code) at my request while I worked through a review backlog. I have not verified the individual findings myself. Please treat everything below as suggestions to evaluate rather than as authoritative review feedback, and push back on anything that is wrong or already handled.

This is impressive work. A recursive-descent parser implementing an explicit whitelist is much better than the substring matching this replaces, the rejected list reads like someone actually enumerated where Java and Rust diverge rather than guessing, and CometRegexParitySuite comparing against java.util.regex.Pattern.find over the full admitted corpus is the right shape of test. Being explicit that this is "not a formal proof of equivalence" is honest and appreciated.

Four things.

The whitelist needs fuzzing, not just a corpus

The safety property here is: every pattern the parser admits behaves identically in Rust and Java. A fixed corpus tests the patterns you thought of. The bug that matters is the one where the parser admits something you did not think of, and then the result is silently wrong rather than an error.

Would you add a generative test: produce random patterns from a grammar (including plenty that should be rejected), run them through CometRegex, and for every admitted pattern compare Java's find against the Rust kernel over a set of random subjects including non-ASCII, newlines, and empty strings? A few thousand iterations of that would find parser gaps that no hand-written corpus will. It could run as a nightly rather than on every PR if it is slow.

How much of the real world does the subset cover?

., ^, $, \d, \w, and \s are all rejected, and between them they appear in most regexes anyone actually writes. So it would be useful to know what fraction of realistic patterns the whitelist admits. Even a rough number from a corpus you have to hand, or from the regex-heavy TPC-DS queries, would tell us whether 360 lines of parser is buying meaningful acceleration or mostly covering patterns that are already cheap.

That is not an argument against the PR. It is an argument for putting the number in the description, because it determines whether the follow-up work ((?-u) rewriting to admit . and the classes) is urgent or not.

Where do 256, 4096, and 32 come from?

The compile-budget gates are three magic numbers. regex's own default size limit is 10 MB of compiled program, so presumably these were chosen to stay well under it, but the derivation is not written down. Could the constants carry a comment explaining what each one bounds and roughly what compiled size it corresponds to? Otherwise the next person who wants to raise one has no basis for deciding whether it is safe.

Coupling to the regex crate version

The whitelist encodes assumptions about how the regex crate behaves today. A future regex bump could change the semantics of an admitted pattern, and nothing in the build would notice. CometRegexParitySuite running in both PR workflows is good, but does the parity suite actually exercise the Rust kernel through native execution, or does it compare Java against expected values? If the former, a regex upgrade would break it, which is the behavior we want. If the latter, it would be worth adding at least one test that would fail on a semantics change upstream.

One process note

CometRegExpBenchmark is modified but the description does not include numbers. What is the speedup for an admitted pattern versus the dispatcher? That is the payoff for all of this and it should be in the description.

@sunchao

sunchao commented Aug 27, 2026

Copy link
Copy Markdown
Member

@andygrove, two factual clarifications against c62b529a for these questions:

  • The description already includes author-reported 4.6–28.8× versus the JVM dispatcher for the stated ASCII workload. Those measurements do not establish what fraction of representative real-world patterns the whitelist admits.
  • The parity suite runs Comet-enabled queries against Comet-disabled Spark results, with operator/dispatcher checks; admitted literals route to the native expression builder. It is fixed-corpus coverage, not just Java-versus-expected-values checks, but it cannot guard every possible crate semantic change.

Grammar generation and representative admission-rate measurements would add evidence beyond that coverage. The constants' structural limits are also not a compiled-byte guarantee; the existing compile-budget P2 discussion remains relevant. No tests or benchmarks were rerun for this clarification.

@andygrove

Copy link
Copy Markdown
Member

My main worry going in was that the whitelist only inspects the pattern, so I wanted to know whether the subject data could still drive Java and Rust apart — control characters and exotic whitespace especially. I spent some time testing that rather than guessing, and I think you're fine. I ported CometRegex to Java so I could fuzz it cheaply, then compiled your actual Scala file against stub SupportLevel types and checked both on 300k random strings over a metachar-heavy alphabet to confirm the port was exact (no decision mismatches, ~28k admitted). Running the analyzer-admitted patterns through java.util.regex and the regex crate against subjects built from every C0 control, DEL, NEL, NBSP, soft hyphen, all the Unicode space separators, U+2028/9, BOM, ZWSP, U+FFFD, emoji and U+10FFFF, I got zero divergences over about 68M pattern/subject pairs, and compile status agreed everywhere too, so there's no admitted pattern where Java throws and Rust doesn't or the other way round. The reason it holds looks structural to me: every whitespace and line-terminator disagreement I could find lives in a construct you reject, and the one data-sensitive construct that survives — the negated class — is code-point-based on both sides.

The one thing the pattern whitelist can't reach is invalid UTF-8 in the subject. Rust's regex treats invalid bytes as unmatchable while Java decodes them to U+FFFD first, so [^0-9] against a lone 0xC3 is true on Spark and false natively. cast_binary_to_string does a lossy decode so that path is safe, but I don't see UTF-8 validation in the native Parquet reader. This is pre-existing for every native string kernel and I'm not asking you to fix it here, but rlike is now default-on, so could you add a line to compatibility/regex.md noting that the subset assumes well-formed UTF-8?

Two smaller things. The parity suite's subject list has \n, \r, \r\n, U+0085, U+2028 and U+2029 but not tab, VT, FF, NUL, DEL, NBSP or BOM — the behavior is right, it's just that those are the characters a reader is most likely to want covered, so would you mind adding them? And nativeApplicable got widened from Literal(_, DataTypes.StringType) to Literal(v: UTF8String, _: StringType), which matches any collation. provablyCompatible guards collation but the allowIncompat && nativeApplicable(expr) branch in convert doesn't, so allowIncompatible=true on a UNICODE-collated rlike now goes native where it previously hit the dispatcher. Should the collation guard move into nativeApplicable?

The other thought is that the safety argument rests entirely on regex crate semantics, and the only parity coverage is 30 patterns going through Spark SQL. Would a table of (pattern, subject, expected) triples generated offline from Java, as a unit test in rlike.rs, be worth adding? That would catch a semantics change on a crate upgrade for almost nothing.

Last one, and for a follow-up rather than this PR: leading ^ looks safe to admit. Java's ^ outside MULTILINE and Rust's ^ without (?m) both anchor to input start only — it's $ that differs, since Java matches it before a final line terminator. I fuzzed ^-prefixed variants of all the admitted patterns against the same control-heavy subjects and saw no divergences. Given how common ^prefix is in real workloads, that seems like a decent chunk of coverage left on the table.

Nice work on this. The recursive-descent whitelist instead of substring matching is the right call, and the compile budget errs in the safe direction — collapsing {0} to 1 over-estimates rather than under-estimates what Rust actually expands.

@sunchao

sunchao commented Aug 27, 2026

Copy link
Copy Markdown
Member

Two source-level clarifications on the latest analysis, at c62b529a:

  • The lone-0xC3 example does not yet establish a native false result. RLike passes Arrow string values to regex::Regex as &str, not to the bytes-regex API. The binary-to-string cast uses JVM-compatible lossy decoding. An end-to-end reproducer needs to identify how the malformed bytes reach that kernel; this is not a claim that every malformed-input path is safe.
  • The opt-in collation widening is real: the literal matcher accepts collated string literals, and allowIncompatible=true bypasses the equivalence gate. Default routing still checks both operands' collations. Moving the guard into nativeApplicable would tighten the explicitly incompatible opt-in policy, rather than repair default routing.

These are source checks, not a rerun of the reported fuzz campaign or an executed malformed-input reproduction.

@sam-1112

Copy link
Copy Markdown
Contributor Author

Thanks @andygrove and @sunchao for the detailed review, extensive fuzzing, and clarifications.

I addressed the concrete items in 136ca137:

  • Added the missing control-heavy parity subjects: TAB, VT, FF, NUL, DEL, NBSP, and BOM.
  • Took the conservative collation approach by moving the non-default-collation guard into nativeApplicable. Collated rlike expressions therefore remain on the JVM dispatcher even when allowIncompatible=true. I also added regression tests covering both collated subjects and collated patterns.
  • Documented that the compile-budget constants are conservative structural heuristics, not compiled-byte estimates or guarantees relative to the Rust regex crate's 10 MiB limit.
  • Added cautious compatibility documentation stating that native rlike assumes well-formed UTF-8. I avoided claiming a proven end-to-end malformed-byte divergence because that path has not yet been reproduced through an actual native input path.
  • Updated the PR description to reflect the current implementation and tests, including the collation behavior, additional parity subjects, compile-budget heuristics, UTF-8 note, and dispatcher-versus-native benchmark results. The parity suite exercises the native Rust kernel rather than comparing Java results only against static expected values.

I agree that the collation change tightens the explicit opt-in path rather than fixing the default routing, since the default path was already protected by provablyCompatible.

To keep this PR focused, I suggest leaving the following as follow-up work:

  • grammar-based generative or nightly fuzzing,
  • measuring admission coverage against a representative real-world regex corpus,
  • adding an offline Java-generated (pattern, subject, expected) table to the Rust unit tests,
  • reproducing malformed UTF-8 through an actual native input path, and
  • evaluating support for leading ^.

Thanks again for the thorough review.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] The native compile-budget issue still has an admitted variant at 136ca137:

val q = (1 to 30).foldLeft("a") { (p, _) => s"($p)*" }
val pattern = s"(($q){255}){16}b"

This 105-byte pattern stays at depth 32, counted bound 255, and estimated expansion 4081. The exact HEAD scanner reports Compatible; JDK 21 compiles it and finds b (not c), while locked default regex 1.13.1 rejects it with Compiled regex exceeds size limit of 10485760 bytes.

Group and quantifier accounting omits capture/nullable-state cost. Used as the pattern literal against a nonconstant default-collation string column, it now selects native RLIKE with incompatible opt-in disabled; source tracing propagates the compiler error during first native plan construction without a JVM retry. Requested BASE used JVM dispatch by default. Please conservatively account for this compilation cost or keep such patterns on the JVM.

Verified by exact-source Scala/JDK/locked Rust component execution plus caller/error-path source tracing, not a full Spark/Comet/JNI query run.

@sam-1112

Copy link
Copy Markdown
Contributor Author

Thanks — you were right that the aggregate expansion and group-depth limits still missed compilation cost from deeply nested quantifiers.

I addressed this in two separate commits:

  1. 63c938a3 adds MaxQuantifierNesting = 8. The reported 105-byte pattern is now classified as Incompatible and remains on the JVM dispatcher. The accounting takes the maximum nesting across concatenated siblings rather than summing them, so ordinary sibling quantifiers such as repeated a* are unaffected.

  2. While validating the remaining boundary against the locked regex 1.13.1, I found another admitted CompiledTooBig shape: seven nested capturing * groups over [^;], 16 additional capturing wrappers, and sixteen concatenated {256} copies. da20a2c4 accounts for the CaptureStart / CaptureEnd states added by each capturing group before counted repetition multiplies the inner expression. Non-capturing groups do not add this cost, and *, +, and ? do not multiply it.

Coverage now includes:

  • analyzer and dispatcher-routing regressions for your 105-byte nested-quantifier repro;
  • analyzer and dispatcher-routing regressions for the capture-cost residual;
  • the exact MaxQuantifierNesting boundary and concatenated-sibling control;
  • continued native admission for [^x]{256}, [^;]{256} repeated 16 times, (?:a{64}){64}, (?:foo){256}, and 32 nested groups without counted capture duplication;
  • rejection of the capturing (a{64}){64} counterpart because counted repetition copies its capture states.

I did not add a compile-then-fallback path. These limits remain conservative structural heuristics rather than a formal compiled-byte guarantee, but both concrete admitted CompiledTooBig patterns verified against the locked 1.13.1 engine now stay off automatic native routing.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed da20a2c4. The previous 105-byte nesting and 992-byte capture witnesses are now rejected, but [P2] the existing native compile-budget issue still has an admitted variant:

val cls = """[^ "$&(*,.02468:<>@BDFHJLNPRTVXZ\\\^`bdfhjlnprtvxz|~]"""
val body = (1 to 7).foldLeft(cls)((p, _) => s"(?:$p)*")
val pattern = s"(?:$body){256}" * 16

This 1,552-byte pattern has no captures, quantifier depth 8 and structural size 4096. In the exact-scanner/component check, Java 21.0.12 compiles it and find(0) returns true for all 17 supplied subjects; locked default regex 1.13.1 fails with CompiledTooBig(10485760).

For an ordinary nonfoldable default-binary string column with this literal pattern, the source still selects native RLike without incompatible opt-in and propagates the compile error during first executePlan, without a JVM retry. That query consequence is source-derived; no full Spark/Comet/JNI query was executed. Please keep this shape on the JVM route or otherwise handle native compilation failure safely.

@sam-1112

Copy link
Copy Markdown
Contributor Author

Thanks — confirmed against da20a2c4. The earlier 105-byte nesting and 992-byte capture witnesses were rejected, but this residual pattern still sat exactly on the remaining caps: it has no captures, quantifier depth 8, and structural size 4096 when every character class is counted as size 1. JDK 21 compiles it, while the locked regex 1.13.1 returns CompiledTooBig(10485760). Under the previous routing, the pattern was therefore admitted to default native RLike, where the compilation error would be propagated without a dispatcher retry.

I addressed this in 4b699afb7 using conservative JVM routing rather than compile-then-fallback.

parseClass now charges for each class atom and range endpoint, including escaped literals. The range hyphen and leading ^ negation do not add cost. This cost feeds into the existing MaxExpansion accounting, so the first {256} component of the reported shape already exceeds 4096 and remains off automatic native routing. I did not lower MaxQuantifierNesting or add a blanket multiplier for negated classes.

Ordinary classes remain admitted, including:

  • [^x]{256}
  • [a-zA-Z0-9_]{256}
  • "[^;]{256}" * 16

The analyzer and routing regressions reconstruct the exact reported pattern and assert pattern.length == 1552. The routing test uses the DataFrame col("s").rlike(pattern) API so the SQL parser cannot rewrite the class backslashes. The results match Spark, and EXPLAIN confirms that the expression uses the JVM codegen dispatcher.

This remains a conservative structural heuristic rather than a proof against the Rust engine's 10 MiB compiled-size limit. As before, allowIncompatible=true may explicitly force a default-collation literal onto the native path, including this pattern.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: the compiler-budget issue still applies at 4b699af for uncounted captures. This class-free family remains inside the scanner’s limits:

val q = (1 to 8).foldLeft("a")((p, _) => "(" + p + ")*")
val branch = "(" * 24 + q + ")" * 24
val pattern = Seq.fill(4096)(branch).mkString("|")

It has 303,103 ASCII bytes and 131,072 captures, but scanner size 4096, group depth 32 and quantifier nesting 8. Capture cost is charged only under counted repetition. The locked native compiler retains these captures; its 64-bit source layout gives at least 11,141,120 bytes of intermediate capture/nullable-star states alone, exceeding the 10,485,760-byte construction limit before simplification.

The affected scope is Spark 4.0/JDK21 with an eligible nonfoldable UTF8_BINARY STRING column containing an empty string, native Project/RLike enabled, and spark.comet.expression.RLike.allowIncompatible=false. Hold the public spark.comet.exec.scalaUDF.codegen.enabled=false on both base and head, with ordinary Spark codegen. The base projection stays in Spark; this head selects native and propagates the construction error without Spark retry. This persists from the previous head; the class-cost increment does not affect it.

Evidence is a native/Spark/Janino source trace plus one Java 21 component: Pattern.compile(pattern, 0) succeeded and matcher("").find(0) returned true. Janino splits the long literal into safe string chunks. No Spark, native, scanner or Janino execution is claimed. Please account for captures outside counted repetition or preserve fallback for this family.

@andygrove andygrove added enhancement New feature or request area:expressions Expression evaluation labels Sep 6, 2026
@sam-1112

sam-1112 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed reproducer. I addressed this by accounting for captures outside counted repetition rather than adding a native compile-then-fallback path.

Capture-state cost is now charged when each capturing group is parsed, including groups under *, +, and ?. The complete cost then propagates through concatenation and alternation. Counted repetition multiplies the already-accounted inner size without charging the capturing group again.

I added:

  • boundary coverage for bare, uncounted, counted, concatenated, and alternated captures;
  • the exact 4,096-branch / 303,103-byte ASCII pattern, which the analyzer now classifies as Incompatible before native conversion; and
  • routing and fallback coverage: a 64-branch reduction of the same pattern shape stays on the JVM dispatcher by default, while the full pattern on Spark 4.0/JDK 21 falls back to Spark when the dispatcher is disabled, without reaching native compilation or raising CompiledTooBig.

The focused analyzer, routing/fallback, and parity suites pass locally. PTAL.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went back through the classifier against the constructs that actually diverge and could not break it. Backreferences, lookaround and every other (?...) form, \d / \w / \s, ., the anchors including \A / \Z / \z, possessive and lazy quantifiers, POSIX classes via the nested [ rejection, and non-ASCII pattern characters are all rejected by positive recognition rather than by blocklist. I re-checked the class boundaries on JDK 21 as well. [^a]{2} is false against a single astral code point, so a negated ASCII class is code-point based on both sides, and [a-b-c], [!-&], [a\--b] and [:alpha:] parse identically in the two engines. RLike::try_new compiles the pattern once when the physical expression is built, so there is no per-batch or per-row recompilation.

The generated compatibility page looks wrong now. CometRLike still mixes in NativeOptInAvailable and still returns the unqualified string from getIncompatibleReasons() in strings.scala, so GenerateDocs.writeExpressionCompatNotes will keep emitting its fixed opt-in header, which states that rlike is evaluated in the JVM by default and matches Spark exactly. That lands directly below the new getCompatibleNotes() bullet saying the opposite. The compatible note also renders under the heading for differences that are always present, where a routing rule reads as a divergence. Could the reason string be scoped to out-of-subset patterns, and that header in GenerateDocs.scala around line 407 be made conditional, so the page does not contradict itself?

On the deferred items, grammar-based fuzzing, the admission-rate number, and the offline Java-generated table in rlike.rs are all things I still want, and for a default-on path the safety argument leans on them. Could you file them as issues and link them here rather than leaving them in the thread? The regex crate coupling is the one I care about most, because a crate bump could move an admitted pattern and only a 30-pattern corpus would be watching.

@sam-1112

sam-1112 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for catching the contradiction in the generated compatibility page, and for rechecking the classifier.

I addressed this in afa8823ff. This change only affects generated documentation and its test coverage; it does not change regex routing, the scanner, or the native kernel.

hasConditionalNativeDefault is now documentation metadata on NativeOptInAvailable for expressions where some compatible cases run natively by default, while the remaining applicable cases retain the JVM-default/native-opt-in behavior. CometRLike uses this form.

The generated RLike page now distinguishes the two routes:

  • UTF8_BINARY literal patterns admitted by the plan-time analyzer run natively by default.
  • Other applicable literal patterns use Spark's JVM codegen dispatcher by default, with an explicit native opt-in.

getIncompatibleReasons() is scoped to those remaining applicable literals. The automatic-native note is rendered under “The following cases use Comet's native implementation by default”, rather than under the heading for differences that are always present.

GenerateDocsSuite covers the conditional-native RLike output, including the config key; unchanged output for an ordinary native-opt-in expression; and unchanged headings for ordinary expressions. GenerateDocsSuite passes locally, and python3 dev/ci/check-suites.py passes with the suite registered in both the Linux and macOS PR workflows.

I generated the latest compatibility pages for Spark 3.4, 3.5, 4.0, and 4.1. The unconditional “By default, RLike is evaluated in the JVM” sentence no longer appears in the latest pages. Historical release documentation remains unchanged. Outside the RLike sections, the generated output was byte-for-byte unchanged.

Separately, I filed the deferred items from the earlier review:

Please take another look when you have a chance. Thanks again!!

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation contradiction is fixed properly, and the shape you chose is better than what I asked for. hasConditionalNativeDefault sitting on NativeOptInAvailable with an explicit note that it is documentation metadata only keeps the routing decision in one place, and GenerateDocs now renders the two routes as two statements instead of one statement that denies the other. The compatible note goes under "The following cases use Comet's native implementation by default" rather than the always-present-differences heading, and the incompatible header is scoped to "applicable cases that are not selected for native execution automatically", so the sentence claiming rlike is JVM-evaluated by default and matches Spark exactly is gone. Extracting renderExpressionCompatNotes so a test can assert the rendered Markdown, rather than testing the writer, is the right call, and GenerateDocsSuite is registered in both pr_build_linux.yml and pr_build_macos.yml.

Thanks for filing #5811, #5812 and #5813 rather than leaving them in the thread.

One suggestion on the shortened compatible note. It now says a pattern "admitted by the plan-time compatibility analyzer" runs natively, without saying what that admits. The old wording listed the constructs, and the reason I am not simply asking for it back is that compatibility/regex.md already enumerates the whitelist and the exclusions properly under "When the Rust engine is safe", which is a better home for it than a bullet in a generated table. But nothing on the generated expression page points there, so a user who lands on expressions.md from a search has no way to find out which of their patterns takes the native path. Could the note carry a link to that section?

The dependency I want to be explicit about, since this turns a native regex path on by default: #5813 is the one that matters, and it needs to land before the next regex bump, not before this PR. native/Cargo.toml pins regex = "1.12.3", and a caret bump to 1.13 could move behaviour inside the admitted subset with only the 30-pattern corpus watching. Would you put a comment next to that pin naming #5813, so whoever takes the bump, or reviews the dependabot PR for it, sees that the parity fixtures are the gate? That is cheap and it is the only part of the safety argument that depends on someone remembering something.

Everything else I checked earlier still holds: the classifier rejects by positive recognition rather than blocklist, the capture-state accounting now charges groups under *, + and ? as well as counted repetition, RLike::try_new compiles once per planned expression, and the class-boundary cases I re-ran on JDK 21 agree between the two engines. I could not construct an admitted pattern that diverges.

Approving on that basis. The regex pin comment and the doc link are both small enough to fold in without another round if you agree with them.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed afa8823ff8c118d84d2576ea06a0e0aebd1468be against merge-base 6f25df70b3825b68a3f9eec01e67de36c0e0bcb8, including the full diff, existing discussions, and current CI.

One actionable issue. I found no functional regression in the checks performed.

Finding

[P2] Remove the redundant interpolation prefix — CometRegExpJvmSuite.scala:456

This string contains no interpolation, so the repository’s lint rule rejects its s prefix. Six CI checks fail on this same line: syntactic lint, four Java lint profiles, and benchmark lint. Remove that prefix; retain the next string’s prefix because it interpolates $explain. CI failure.

Validation

  • Parser compatibility: 41,112 admitted patterns, 5,796,792 comparisons, zero Java/Rust differences. Checked JDK 17 and 21 against locked Rust regex 1.13.1.
  • Exact PR revision: compilation, 17 distinct analyzer/documentation tests, 72 routing checks, and 32 support-level checks passed. Generated documentation correctly describes both execution paths.
  • Native integration: 92 tests passed, zero failures, across CometRegexSuite, CometRegexParitySuite, CometRegExpJvmSuite, and GenerateDocsSuite on Spark 4.1.3/JDK 17. One test was skipped because it requires Spark 4.0/JDK 21.

Native integration used the verified CI merge 736ed196cc36576208cdb8e4a850e4fa99f39cbe and its checksum-verified native library. This merge combines the reviewed head with base 424c31aa79d13fddf743ffa29bae3c6f146e6c5e. These are local tests of the CI merge using CI-built native code, not unmerged-head native integration or a local native build. Local native compilation was blocked by a missing dependency in the configured mirror.

CI status

At the publication recheck, CI had 34 successful checks, six failed checks, six skipped checks, and 21 still queued or running. All six failures were independently traced to the redundant interpolation prefix above. Current PR CI run.

Assessment

The restricted native-default routing looks reasonable. NULL, nonliteral, collation, opt-in, and dispatcher-disabled behavior checked out. Earlier parser regressions have rejection coverage.

I would merge after the lint fix and completion of the remaining CI. Compilation-budget limits remain heuristics, and the finite compatibility checks do not prove every admitted pattern equivalent. I did not rerun performance benchmarks.

val explain = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
assert(
!explain.toLowerCase.contains("compiledtoobig"),
s"native compilation must not run for the full uncounted-capture reproducer, " +

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Remove the redundant interpolation prefix to restore CI

This string contains no interpolation, so the repository’s RedundantSyntax rule rejects its s prefix. The current PR run fails syntactic lint, four Java lint profiles, and benchmark lint on this exact line. Remove the prefix here; retain the next string’s prefix because it interpolates $explain.

Failing CI job

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — fixed in bb15f01. I removed the redundant s prefix from the non-interpolated string and retained it on the following string that interpolates $explain.

@sam-1112

Copy link
Copy Markdown
Contributor Author

Thanks @andygrove . I added a link from the generated compatible note to the “When the Rust engine is safe” section in compatibility/regex.md. I also added a comment next to the regex dependency pin identifying #5813 as the parity-fixture gate for the next crate upgrade. Included in bb15f01.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fresh review of bb15f0184f771e02ebaa30e143840f9c8a4e3d34, covering the full 17-file diff against merge-base 6f25df70b3825b68a3f9eec01e67de36c0e0bcb8, the existing discussions, and current CI. The base branch was 424c31aa79d13fddf743ffa29bae3c6f146e6c5e at the final recheck.

No remaining actionable issues found. Approving.

The automatic native route is limited to analyzer-admitted, non-null UTF8_BINARY literals. I rechecked class boundaries, quantifier/capture budgeting, native ownership and error propagation, NULL and dynamic patterns, collation on either operand, incompatible opt-in, and dispatcher-disabled fallback. The previous lint finding is fixed. The documentation now describes both routes and links to the subset definition, and the dependency comment records the parity-fixture follow-up.

Validation

  • Built native code and compiled the JVM code locally at this exact head. The root-reactor run of CometRegexSuite, CometRegexParitySuite, CometRegExpJvmSuite, and GenerateDocsSuite passed 92 tests, with zero failures. One test was skipped because it requires Spark 4.0/JDK 21. This run used Spark 4.1.3/JDK 17 and the freshly built native library, with the packaged library checksum verified.
  • Fresh component differential checks produced 29,284,536 Java/Rust comparisons with zero mismatches against locked regex 1.13.1, including JDK 17/21 and adversarial class, grammar, and Unicode cases. This count includes overlapping corpora and repeated checks across JDKs.
  • 3,177 admitted compile-stress patterns compiled successfully in both engines, including nested counted products reaching the structural expansion limit. Independent planning probes also passed 72 conversion checks and 32 support-level/hint checks.
  • CI has 70 successful checks, eight skips, and no failed or pending checks. I inspected the Spark 4.1 expression log and Spark 4.0/JDK 21 expression log. The latter passes the full capture-pattern regression skipped locally. These CI jobs tested merge c70aa54dfe07929f39e87a32c38a946a77ebb1b3, combining this head with 80c8ec37af422db1952a1128cb23043c9bbfbb72.

The compile budget remains heuristic, and finite differential checks do not prove every admitted pattern equivalent. The new random nested-pattern corpus used subjects of at most four codepoints after longer inputs caused excessive Java backtracking. Malformed UTF-8 and performance benchmarks were not validated in this pass. Follow-ups #5811, #5812, and #5813 remain tracked.

Hung added 4 commits September 16, 2026 11:11
Add a plan-time whitelist so UTF8_BINARY literals the analyzer can prove
equivalent to Java regex take the native path without allowIncompatible.
Out-of-subset literals stay on the JVM dispatcher.
…e budget

Reject Rust-only character-class operators and unescaped ] range endpoints
so incompatible literals stay on the JVM dispatcher. Cap nested counted
expansion and group depth so Java-valid patterns that Rust cannot compile
do not fail native plan construction.
Reject unescaped '[' as a character-class range endpoint so malformed
Java patterns like [@-[] stay on the JVM dispatcher and preserve Spark
regex errors. Apply MaxExpansion across concatenation, alternation, and
counted repetition, including zero-count terms, so sibling groups and
repeated atoms cannot bypass compile-size admission.
Keep `{0,}` from collapsing inner cost so nested patterns like
`(([^;]{256}){0,}){256}` stay on the JVM dispatcher. Escape
backslashes in the rlike benchmark SQL so `\d+` is not parsed as `d+`.
Hung added 7 commits September 16, 2026 11:11
Native rlike does not implement Spark collation, so opt-in must still require
default collation. Document compile-budget constants as structural heuristics
and assume well-formed UTF-8 rather than a confirmed malformed-input split.
Stacked `*`/`+`/`?`/`{n}` do not raise the structural expansion estimate,
so patterns like the 105-byte nested-star repro compiled past the Rust
regex 10 MiB limit. Cap quantifier nesting and keep those literals on the
JVM dispatcher.
Capture start/end states are copied when `{n}` expands an expression.
Add that cost before counted multiplication so residual patterns that
compiled past the Rust 10 MiB limit stay on the JVM dispatcher.
Non-capturing groups and ordinary negated classes such as `[^x]{256}`
remain admitted.
Charge capture-state cost when each capturing group is parsed so it propagates through concatenation, alternation, and uncounted repetition, not only counted copies.

Keep the full uncounted-capture reproducer on the JVM dispatcher by default and fall back to Spark when the dispatcher is disabled.
@sam-1112
sam-1112 force-pushed the rlike-native-compatible-subset-5351 branch from 622db9a to 7ef89f0 Compare September 16, 2026 03:14
@sunchao
sunchao added this pull request to the merge queue Sep 16, 2026
Merged via the queue into apache:main with commit f77cf30 Sep 16, 2026
37 checks passed
@sam-1112
sam-1112 deleted the rlike-native-compatible-subset-5351 branch September 16, 2026 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run rlike natively by default for patterns that are provably Java-regex equivalent

3 participants