Repository navigation
feat: run rlike natively by default for Java-equivalent literal patterns - #5415
Conversation
sunchao
left a comment
There was a problem hiding this comment.
Summary
Reviewed the complete 13-file diff from 367ca64be4e62dd10dc4c932de9ca738355a45fa to 86729123819387bbfa132c5c81956f82ea60b193, including the live discussion and CI state. Five independent full-diff passes were followed by independent source and executable verification of the retained cases.
The intended optimization is narrowly scoped to default-native RLike literals. Three pattern-specific P2 regressions remain: Rust character-class set operations change results, leading-] ranges independently change results, and Java-valid patterns can be admitted despite exceeding Rust's compilation limits. Each is detailed inline; none requires opting into incompatible execution.
Prior state and problem
The native RLike kernel already existed, but Java and Rust regex syntax and semantics differ. At the base revision, ordinary literal patterns used Spark's own generated regex implementation through the JVM dispatcher by default; native execution required spark.comet.expression.RLike.allowIncompatible=true. If the dispatcher was disabled, the default expression could fall back to Spark.
This preserves Java behavior but leaves JNI/dispatcher and per-row Java regex costs on patterns that both engines can evaluate equivalently. The PR aims to remove those costs for an automatically recognized subset while keeping other regex expressions and explicitly incompatible execution behavior separate.
Design approach
CometRegex introduces a recursive-descent admission parser rather than a substring-based filter. It recognizes printable ASCII literals, selected character classes, ordinary greedy quantifiers, capturing/non-capturing groups, and alternation. Unrecognized syntax is rejected from automatic native selection; anchors, wildcard dot, shorthand classes, lookaround, backreferences, and flags are intentionally outside the subset.
CometRLike additionally requires a non-null literal and default string collation on both operands for automatic selection. The existing native kernel and Cargo lock are unchanged, and the pattern is serialized without rewriting. Consequently, the admission decision must establish both equivalent Boolean matching and native compilability.
Correctness / compatibility analysis
The retained cases violate two different parts of that admission contract:
[a~~b]and[a-z--b]are admitted, but Rust interprets class set operations that Java does not interpret the same way. Concrete non-null inputs produce opposite Booleans.[]-a]is also admitted, without either set operator. Java treats the initial]as a range endpoint; Rust treats this spelling differently. Both positive and negated classes can add or remove matching rows.[^;]{20000},a{1000000}, nested counted repetitions, and sufficiently nested groups are admitted even though default Rust compilation rejects them. Native plan construction propagates that rejection instead of returning to the dispatcher.
Verification used the unmodified pinned scanner, the exact regex 1.13.1 dependency graph with archive/source checksums matched to Cargo.lock, and real Spark 3.5.9 and 4.0.4 expressions with non-foldable input. The retained cases were checked in both interpreted evaluation and forced generated projections; NULL-subject controls were also checked. These are component/runtime probes plus source-traced integration, not a claimed full Comet/JNI query run.
The unchanged four-test CometRegexSuite passed in isolated builds on Scala 2.12.18 and 2.13.17, and suite-registration and diff-whitespace checks passed. The existing examples do not cover the counterexamples above. CI started during this review: at the 17:28 UTC refresh, CodeQL and Linux/macOS lint checks had succeeded, while native/JVM builds, Rust tests, Java lint, and the benchmark check were still running. No completed full integration-suite result is claimed.
Key design decisions
- Keep automatic admission distinct from the existing incompatible opt-in. The latter remains an explicit request to use the Rust implementation for applicable literals, not an equivalence guarantee.
- Check both operand collations for the automatic route. Spark 3.x uses the existing no-collation shim; Spark 4.x uses the existing collation-aware type checks. Explicit opt-in can bypass the automatic equivalence gate.
- Exclude NULL and non-literal patterns from native applicability. They continue through the dispatcher, or fall back when the dispatcher is unavailable.
- Preserve the generic expression-enabled gate. Disabling RLike itself still prevents this serializer from converting it; disabling only the dispatcher no longer disables eligible native RLike expressions.
Implementation sketch
The serializer extracts a UTF8String literal, checks the operand types, and calls CometRegex.supportLevel. An admitted pattern becomes Compatible() without a native-opt-in hint. Conversion then emits the existing RLike protobuf through createBinaryExpr; the native builder passes the unchanged literal to RLike::try_new, which uses Regex::new and is_match.
Other applicable literals retain their opt-in hint and default JVM dispatcher route. The patch also adds analyzer and parity suites, expands routing/NULL/collation tests, registers the new suites in both OS workflows, adds SQL cases, and updates documentation and benchmark mode selection. No native engine or dependency change accompanies the newly automatic route.
Behavioral changes worth calling out
For admitted literals, default Comet execution now uses the native engine even with the dispatcher disabled. That is the intended performance change, but it also makes any admission mistake observable without a user accepting regex incompatibility.
Out-of-subset literals continue to use the JVM by default, and non-literal/NULL patterns do not become native. Other regex functions retain their existing routing rules. The compile-limit cases are a change from successful default JVM evaluation to native construction failure; the two character-class cases are silent result changes, not merely different match spans or capture groups.
Suggested improvements
Tighten class admission for both Rust-only set operators and the independent leading-] range boundary. Add positive and negated differential examples, including subjects that distinguish literal operator characters and range interiors, and assert that rejected patterns retain JVM routing by default.
Make native compilability part of the admission contract, with conservative aggregate-expansion/depth bounds or a reliable validation/fallback path. A per-integer bound or source-length limit alone does not cover nested repetition expansion. Add the concrete size/depth cases to routing and error-behavior regression coverage, then verify the full parity/routing suites on the supported Spark profiles as CI completes.
| if (startsWith("&&") || peek == '[') { | ||
| return false | ||
| } |
There was a problem hiding this comment.
[P2] Exclude Rust-only class operators from automatic native routing
This gate rejects && but admits ~~ and subtraction forms such as [a-z--b]. The pinned analyzer returns Compatible for [a~~b]; Spark 3.5.9/4.0.4 return true on subject ~, while the locked Rust regex 1.13.1 returns false because ~~ is symmetric difference. Likewise, [a-z--b] matches b in Spark but not Rust. I verified the Spark results in both interpreted and generated evaluation with a non-foldable input. Since CometRLike.convert now sends these literals to the unchanged native matcher without allowIncompatible=true, existing projections and filters silently change results. Keep these class forms outside automatic admission and add differential/routing regressions before selecting native for them.
There was a problem hiding this comment.
Agreed — the first revision only rejected &&, so [a~~b] / [a-z--b] were wrongly admitted and would have taken the native path without allowIncompatible.
7e51a22 now treats && / ~~ / -- inside a character class as incompatible. Those literals stay on the JVM dispatcher by default. Single ~ in a class (e.g. [a~b]) is still admitted; it is a Java literal, not a Rust set operator.
Regressions:
CometRegexSuite: analyzer returnsIncompatiblefor[a~~b],[^a~~b],[a-z--b],[^a-z--b],[a&&b]CometRegExpJvmSuite: EXPLAIN still shows the dispatcher, andcheckSparkAnswerAndOperatormatches Spark on a non-foldable column (~,a,b,x, andb/a/z/-for the subtraction form), including NULL
No runtime compile-fallback; these stay off automatic native.
| val ranging = lastAtom.isDefined && peek == '-' && peekOffset(1).exists(_ != ']') | ||
| if (ranging) { | ||
| consume() // '-' | ||
| parseClassAtom() match { | ||
| case Some(end) if end >= lastAtom.get => |
There was a problem hiding this comment.
[P2] Reject ranges starting at an unescaped leading closing bracket
The initial literal ] is stored in lastAtom and then accepted as a range endpoint. For the admitted pattern []-a], Java/Spark interpret the range from ] through a, but Rust treats this spelling as the literals ], -, and a. On both Spark 3.5.9 and 4.0.4, interpreted and generated evaluation return true for _ and false for -; the locked native regex returns the opposite results. The negated form also differs. This default-native wrong-answer case contains neither ~~ nor --, so rejecting set operators alone will not fix it. Reject this class-boundary form, or normalize it only after establishing equivalent semantics, and cover both positive and negated variants.
There was a problem hiding this comment.
Good catch — this is a separate Java/Rust split, not covered by rejecting ~~ / --.
Java treats a leading ] as a literal (so []-a] is the range ]–a), while Rust reads it as the closer and then the literals ], -, a. Same mismatch on the negated form. That’s why _ vs - flip between Spark and native.
7e51a22 now never treats an unescaped ] as a class atom or range endpoint, so []-a] and [^]-a] stay off automatic native. Escaped ] ([\\]]) is still admitted.
Tests:
- analyzer:
Incompatiblefor[]-a],[^]-a],[]],[^]] - routing: dispatcher + Spark-equal answers on
_,-,],a,z, and NULL
| val scanner = new Scanner(pattern) | ||
| if (scanner.parseExpr() && !scanner.remaining) { | ||
| Compatible() |
There was a problem hiding this comment.
[P2] Preserve JVM routing for patterns beyond native compile limits
Successful syntax scanning does not ensure Regex::new can compile the pattern. For example, this analyzer admits [^;]{20000} and a{1000000}; real Spark 3.5.9/4.0.4 evaluate them on a column containing a as false, but the locked Rust engine rejects both with Compiled regex exceeds size limit of 10485760 bytes. Nested counted repetitions have the same problem, and 251 nested groups exceed Rust's separate depth limit while succeeding in Spark. The unchanged RLike::try_new and native builder propagate these errors from plan creation, with no dispatcher retry, so previously successful default-config queries now fail without opt-in. Require conservative native size/depth applicability or a reliable compilation-validation/fallback path before returning Compatible.
There was a problem hiding this comment.
Yep — scanning the syntax isn’t the same as “Rust can compile this.” Spark is happy with [^;]{20000} / a{1000000} / nested {n} / deep groups; the locked regex crate then dies at plan time (Compiled regex exceeds size limit / nest depth) and we have no dispatcher retry. That’s a default-config regression.
Didn’t add a compile-then-fallback path here. 7e51a22 just refuses those at plan time:
{n}bigger than 256- nested counted product over 4096 (
(a{100}){100}) - more than 32 nested groups
They stay on the JVM dispatcher. a{256} and 32-deep groups still go native. Analyzer + EXPLAIN/Spark-equal tests cover the cases you listed.
There was a problem hiding this comment.
[P2] {0,} still bypasses the compile-size budget
Rechecked at 0f98989095dee698db71b366998e112bdd1e15a6. The aggregate checks address the earlier sibling cases, but (([^;]{256}){0,}){256} is still classified as Compatible. The {0,} branch returns the lower bound 0, so multiplyWithinBudget reduces the inner cost from 256 to 1. Unlike {0} or {0,0}, this is unbounded repetition and the inner expression still has to be compiled.
I reproduced this with the unmodified current scanner on Scala 2.12 and 2.13. Spark 3.5.9 and 4.0.4 both return true for a non-foldable subject containing a, in interpreted evaluation and forced generated projections. The exact locked Rust regex 1.13.1 instead rejects the pattern with Compiled regex exceeds size limit of 10485760 bytes. The equivalent (([^;]{256})*){256} is correctly rejected by the scanner.
Because the admitted form now selects native execution without opt-in and the native builder propagates the compilation error, this still changes a successful default-config query into a failure. Could we retain the inner compilation cost for unbounded {0,} separately from exact-zero repetitions and add this case to the dispatcher-routing regressions? These checks used real Spark expressions and the locked regex engine plus source-traced native routing, not a full Comet/JNI query run.
There was a problem hiding this comment.
Thanks, that was a real gap. {0,} was using the lower bound 0 as the expansion multiplier, so multiplyWithinBudget collapsed the inner cost to 1 and admitted (([^;]{256}){0,}){256}. That pattern is Java-valid but rejected by the locked Rust regex crate, so the native path would fail a query that Spark currently succeeds.
The unbounded {n,} branch now uses math.max(1, n), so {0,} keeps the inner compile cost. {0} and {0,0} stay exact-zero and are still Compatible.
Coverage:
- analyzer:
(([^;]{256}){0,}){256}is Incompatible;{0}/{0,0}remain Compatible - routing: the unbounded pattern stays on the JVM codegen dispatcher and matches Spark
- exact-zero counterparts stay native and match Spark
Pushed in c62b529a7.
| } | ||
| if (peek == '\\') { | ||
| parseEscape(inClass = true) | ||
| } else if (isPrintableAscii(peek)) { |
There was a problem hiding this comment.
[P2] Preserve Spark errors for raw [ range endpoints
The [ guard in parseClass runs before the range branch consumes -, so this helper can still accept an unescaped [ as the range endpoint. The exact scanner returns Compatible for [@-[] (and [^@-[]). With a non-foldable string input, real Spark 3.5.9 and 4.0.4 reject these patterns: interpreted RLIKE raises an invalid-pattern error, and forced generated-projection construction raises PatternSyntaxException for an unclosed character class. The locked Rust regex 1.13.1 instead compiles [@-[] and matches A. Because CometRLike.convert now trusts this admission result without incompatible opt-in, a malformed-pattern failure becomes a successful predicate under the default native route. Please reject raw [ in the range-endpoint helper as well; the escaped [@-\[] control is valid and equivalent in both engines.
There was a problem hiding this comment.
Good catch — the class-loop [ guard runs before - is consumed, so it never saw [@-[] / [^@-[] as a nested class. Java rejects those as an unclosed class; Rust compiles them and matches A. That’s a default-native “error becomes true/false” case. This commit rejects an unescaped [ in parseClassAtom as well, so it cannot be a range endpoint. Escaped [@-\[] is still admitted and covered by Java/native parity.
Tests:
- analyzer:
Incompatiblefor[@-[]/[^@-[];Compatiblefor[@-\[] - routing: dispatcher + Spark regex error on a non-foldable column (
@,[,A, NULL)
| runInSubsetModes(p.name, v, query) | ||
| } | ||
| } | ||
| outOfSubsetPatterns.foreach { p => | ||
| val query = s"select c1 rlike '${p.pattern}' from parquetV1Table" |
There was a problem hiding this comment.
[P2] Preserve the backslash in the out-of-subset benchmark SQL
With the default spark.sql.parser.escapedStringLiterals=false, interpolating the new \d+ pattern here makes Spark parse it as d+ (reproduced with Spark 3.5.9 and 4.0.4). CometRegex admits d+, so the case labelled Comet (Exec, JVM regex) selects the automatic native branch just like the opted-in case. The generated numeric strings also match none of the rows instead of every row. This leaves the new four-way comparison without a JVM-dispatcher measurement. Please preserve the backslash with an escaped or raw SQL literal, as the parity suite already does, before passing this query to runOutOfSubsetModes.
There was a problem hiding this comment.
Good catch — fixed in c62b529a7
The out-of-subset pattern is now escaped before being interpolated into the SQL query:
val escapedPattern = p.pattern.replace("\\", "\\\\")
val query = s"select c1 rlike '$escapedPattern' from parquetV1Table"
This is impressive work. A recursive-descent parser implementing an explicit whitelist is much better than the substring matching this replaces, the rejected list reads like someone actually enumerated where Java and Rust diverge rather than guessing, and Four things. The whitelist needs fuzzing, not just a corpus The safety property here is: every pattern the parser admits behaves identically in Rust and Java. A fixed corpus tests the patterns you thought of. The bug that matters is the one where the parser admits something you did not think of, and then the result is silently wrong rather than an error. Would you add a generative test: produce random patterns from a grammar (including plenty that should be rejected), run them through How much of the real world does the subset cover?
That is not an argument against the PR. It is an argument for putting the number in the description, because it determines whether the follow-up work ( Where do 256, 4096, and 32 come from? The compile-budget gates are three magic numbers. Coupling to the The whitelist encodes assumptions about how the One process note
|
|
@andygrove, two factual clarifications against
Grammar generation and representative admission-rate measurements would add evidence beyond that coverage. The constants' structural limits are also not a compiled-byte guarantee; the existing compile-budget P2 discussion remains relevant. No tests or benchmarks were rerun for this clarification. |
|
My main worry going in was that the whitelist only inspects the pattern, so I wanted to know whether the subject data could still drive Java and Rust apart — control characters and exotic whitespace especially. I spent some time testing that rather than guessing, and I think you're fine. I ported The one thing the pattern whitelist can't reach is invalid UTF-8 in the subject. Rust's Two smaller things. The parity suite's subject list has The other thought is that the safety argument rests entirely on Last one, and for a follow-up rather than this PR: leading Nice work on this. The recursive-descent whitelist instead of substring matching is the right call, and the compile budget errs in the safe direction — collapsing |
|
Two source-level clarifications on the latest analysis, at
These are source checks, not a rerun of the reported fuzz campaign or an executed malformed-input reproduction. |
|
Thanks @andygrove and @sunchao for the detailed review, extensive fuzzing, and clarifications. I addressed the concrete items in
I agree that the collation change tightens the explicit opt-in path rather than fixing the default routing, since the default path was already protected by To keep this PR focused, I suggest leaving the following as follow-up work:
Thanks again for the thorough review. |
sunchao
left a comment
There was a problem hiding this comment.
[P2] The native compile-budget issue still has an admitted variant at 136ca137:
val q = (1 to 30).foldLeft("a") { (p, _) => s"($p)*" }
val pattern = s"(($q){255}){16}b"This 105-byte pattern stays at depth 32, counted bound 255, and estimated expansion 4081. The exact HEAD scanner reports Compatible; JDK 21 compiles it and finds b (not c), while locked default regex 1.13.1 rejects it with Compiled regex exceeds size limit of 10485760 bytes.
Group and quantifier accounting omits capture/nullable-state cost. Used as the pattern literal against a nonconstant default-collation string column, it now selects native RLIKE with incompatible opt-in disabled; source tracing propagates the compiler error during first native plan construction without a JVM retry. Requested BASE used JVM dispatch by default. Please conservatively account for this compilation cost or keep such patterns on the JVM.
Verified by exact-source Scala/JDK/locked Rust component execution plus caller/error-path source tracing, not a full Spark/Comet/JNI query run.
|
Thanks — you were right that the aggregate expansion and group-depth limits still missed compilation cost from deeply nested quantifiers. I addressed this in two separate commits:
Coverage now includes:
I did not add a compile-then-fallback path. These limits remain conservative structural heuristics rather than a formal compiled-byte guarantee, but both concrete admitted |
sunchao
left a comment
There was a problem hiding this comment.
Re-reviewed da20a2c4. The previous 105-byte nesting and 992-byte capture witnesses are now rejected, but [P2] the existing native compile-budget issue still has an admitted variant:
val cls = """[^ "$&(*,.02468:<>@BDFHJLNPRTVXZ\\\^`bdfhjlnprtvxz|~]"""
val body = (1 to 7).foldLeft(cls)((p, _) => s"(?:$p)*")
val pattern = s"(?:$body){256}" * 16This 1,552-byte pattern has no captures, quantifier depth 8 and structural size 4096. In the exact-scanner/component check, Java 21.0.12 compiles it and find(0) returns true for all 17 supplied subjects; locked default regex 1.13.1 fails with CompiledTooBig(10485760).
For an ordinary nonfoldable default-binary string column with this literal pattern, the source still selects native RLike without incompatible opt-in and propagates the compile error during first executePlan, without a JVM retry. That query consequence is source-derived; no full Spark/Comet/JNI query was executed. Please keep this shape on the JVM route or otherwise handle native compilation failure safely.
|
Thanks — confirmed against I addressed this in
Ordinary classes remain admitted, including:
The analyzer and routing regressions reconstruct the exact reported pattern and assert This remains a conservative structural heuristic rather than a proof against the Rust engine's 10 MiB compiled-size limit. As before, |
sunchao
left a comment
There was a problem hiding this comment.
P2: the compiler-budget issue still applies at 4b699af for uncounted captures. This class-free family remains inside the scanner’s limits:
val q = (1 to 8).foldLeft("a")((p, _) => "(" + p + ")*")
val branch = "(" * 24 + q + ")" * 24
val pattern = Seq.fill(4096)(branch).mkString("|")It has 303,103 ASCII bytes and 131,072 captures, but scanner size 4096, group depth 32 and quantifier nesting 8. Capture cost is charged only under counted repetition. The locked native compiler retains these captures; its 64-bit source layout gives at least 11,141,120 bytes of intermediate capture/nullable-star states alone, exceeding the 10,485,760-byte construction limit before simplification.
The affected scope is Spark 4.0/JDK21 with an eligible nonfoldable UTF8_BINARY STRING column containing an empty string, native Project/RLike enabled, and spark.comet.expression.RLike.allowIncompatible=false. Hold the public spark.comet.exec.scalaUDF.codegen.enabled=false on both base and head, with ordinary Spark codegen. The base projection stays in Spark; this head selects native and propagates the construction error without Spark retry. This persists from the previous head; the class-cost increment does not affect it.
Evidence is a native/Spark/Janino source trace plus one Java 21 component: Pattern.compile(pattern, 0) succeeded and matcher("").find(0) returned true. Janino splits the long literal into safe string chunks. No Spark, native, scanner or Janino execution is claimed. Please account for captures outside counted repetition or preserve fallback for this family.
|
Thanks for the detailed reproducer. I addressed this by accounting for captures outside counted repetition rather than adding a native compile-then-fallback path. Capture-state cost is now charged when each capturing group is parsed, including groups under I added:
The focused analyzer, routing/fallback, and parity suites pass locally. PTAL. |
andygrove
left a comment
There was a problem hiding this comment.
I went back through the classifier against the constructs that actually diverge and could not break it. Backreferences, lookaround and every other (?...) form, \d / \w / \s, ., the anchors including \A / \Z / \z, possessive and lazy quantifiers, POSIX classes via the nested [ rejection, and non-ASCII pattern characters are all rejected by positive recognition rather than by blocklist. I re-checked the class boundaries on JDK 21 as well. [^a]{2} is false against a single astral code point, so a negated ASCII class is code-point based on both sides, and [a-b-c], [!-&], [a\--b] and [:alpha:] parse identically in the two engines. RLike::try_new compiles the pattern once when the physical expression is built, so there is no per-batch or per-row recompilation.
The generated compatibility page looks wrong now. CometRLike still mixes in NativeOptInAvailable and still returns the unqualified string from getIncompatibleReasons() in strings.scala, so GenerateDocs.writeExpressionCompatNotes will keep emitting its fixed opt-in header, which states that rlike is evaluated in the JVM by default and matches Spark exactly. That lands directly below the new getCompatibleNotes() bullet saying the opposite. The compatible note also renders under the heading for differences that are always present, where a routing rule reads as a divergence. Could the reason string be scoped to out-of-subset patterns, and that header in GenerateDocs.scala around line 407 be made conditional, so the page does not contradict itself?
On the deferred items, grammar-based fuzzing, the admission-rate number, and the offline Java-generated table in rlike.rs are all things I still want, and for a default-on path the safety argument leans on them. Could you file them as issues and link them here rather than leaving them in the thread? The regex crate coupling is the one I care about most, because a crate bump could move an admitted pattern and only a 30-pattern corpus would be watching.
6f76686 to
afa8823
Compare
|
Thanks for catching the contradiction in the generated compatibility page, and for rechecking the classifier. I addressed this in
The generated
I generated the latest compatibility pages for Spark 3.4, 3.5, 4.0, and 4.1. The unconditional “By default, RLike is evaluated in the JVM” sentence no longer appears in the latest pages. Historical release documentation remains unchanged. Outside the Separately, I filed the deferred items from the earlier review:
Please take another look when you have a chance. Thanks again!! |
andygrove
left a comment
There was a problem hiding this comment.
The documentation contradiction is fixed properly, and the shape you chose is better than what I asked for. hasConditionalNativeDefault sitting on NativeOptInAvailable with an explicit note that it is documentation metadata only keeps the routing decision in one place, and GenerateDocs now renders the two routes as two statements instead of one statement that denies the other. The compatible note goes under "The following cases use Comet's native implementation by default" rather than the always-present-differences heading, and the incompatible header is scoped to "applicable cases that are not selected for native execution automatically", so the sentence claiming rlike is JVM-evaluated by default and matches Spark exactly is gone. Extracting renderExpressionCompatNotes so a test can assert the rendered Markdown, rather than testing the writer, is the right call, and GenerateDocsSuite is registered in both pr_build_linux.yml and pr_build_macos.yml.
Thanks for filing #5811, #5812 and #5813 rather than leaving them in the thread.
One suggestion on the shortened compatible note. It now says a pattern "admitted by the plan-time compatibility analyzer" runs natively, without saying what that admits. The old wording listed the constructs, and the reason I am not simply asking for it back is that compatibility/regex.md already enumerates the whitelist and the exclusions properly under "When the Rust engine is safe", which is a better home for it than a bullet in a generated table. But nothing on the generated expression page points there, so a user who lands on expressions.md from a search has no way to find out which of their patterns takes the native path. Could the note carry a link to that section?
The dependency I want to be explicit about, since this turns a native regex path on by default: #5813 is the one that matters, and it needs to land before the next regex bump, not before this PR. native/Cargo.toml pins regex = "1.12.3", and a caret bump to 1.13 could move behaviour inside the admitted subset with only the 30-pattern corpus watching. Would you put a comment next to that pin naming #5813, so whoever takes the bump, or reviews the dependabot PR for it, sees that the parity fixtures are the gate? That is cheap and it is the only part of the safety argument that depends on someone remembering something.
Everything else I checked earlier still holds: the classifier rejects by positive recognition rather than blocklist, the capture-state accounting now charges groups under *, + and ? as well as counted repetition, RLike::try_new compiles once per planned expression, and the class-boundary cases I re-ran on JDK 21 agree between the two engines. I could not construct an admitted pattern that diverges.
Approving on that basis. The regex pin comment and the doc link are both small enough to fold in without another round if you agree with them.
sunchao
left a comment
There was a problem hiding this comment.
Reviewed afa8823ff8c118d84d2576ea06a0e0aebd1468be against merge-base 6f25df70b3825b68a3f9eec01e67de36c0e0bcb8, including the full diff, existing discussions, and current CI.
One actionable issue. I found no functional regression in the checks performed.
Finding
[P2] Remove the redundant interpolation prefix — CometRegExpJvmSuite.scala:456
This string contains no interpolation, so the repository’s lint rule rejects its s prefix. Six CI checks fail on this same line: syntactic lint, four Java lint profiles, and benchmark lint. Remove that prefix; retain the next string’s prefix because it interpolates $explain. CI failure.
Validation
- Parser compatibility: 41,112 admitted patterns, 5,796,792 comparisons, zero Java/Rust differences. Checked JDK 17 and 21 against locked Rust regex 1.13.1.
- Exact PR revision: compilation, 17 distinct analyzer/documentation tests, 72 routing checks, and 32 support-level checks passed. Generated documentation correctly describes both execution paths.
- Native integration: 92 tests passed, zero failures, across
CometRegexSuite,CometRegexParitySuite,CometRegExpJvmSuite, andGenerateDocsSuiteon Spark 4.1.3/JDK 17. One test was skipped because it requires Spark 4.0/JDK 21.
Native integration used the verified CI merge 736ed196cc36576208cdb8e4a850e4fa99f39cbe and its checksum-verified native library. This merge combines the reviewed head with base 424c31aa79d13fddf743ffa29bae3c6f146e6c5e. These are local tests of the CI merge using CI-built native code, not unmerged-head native integration or a local native build. Local native compilation was blocked by a missing dependency in the configured mirror.
CI status
At the publication recheck, CI had 34 successful checks, six failed checks, six skipped checks, and 21 still queued or running. All six failures were independently traced to the redundant interpolation prefix above. Current PR CI run.
Assessment
The restricted native-default routing looks reasonable. NULL, nonliteral, collation, opt-in, and dispatcher-disabled behavior checked out. Earlier parser regressions have rejection coverage.
I would merge after the lint fix and completion of the remaining CI. Compilation-budget limits remain heuristics, and the finite compatibility checks do not prove every admitted pattern equivalent. I did not rerun performance benchmarks.
| val explain = new ExtendedExplainInfo().generateExtendedInfo(cometPlan) | ||
| assert( | ||
| !explain.toLowerCase.contains("compiledtoobig"), | ||
| s"native compilation must not run for the full uncounted-capture reproducer, " + |
There was a problem hiding this comment.
[P2] Remove the redundant interpolation prefix to restore CI
This string contains no interpolation, so the repository’s RedundantSyntax rule rejects its s prefix. The current PR run fails syntactic lint, four Java lint profiles, and benchmark lint on this exact line. Remove the prefix here; retain the next string’s prefix because it interpolates $explain.
There was a problem hiding this comment.
Thanks — fixed in bb15f01. I removed the redundant s prefix from the non-interpolated string and retained it on the following string that interpolates $explain.
|
Thanks @andygrove . I added a link from the generated compatible note to the “When the Rust engine is safe” section in |
sunchao
left a comment
There was a problem hiding this comment.
Fresh review of bb15f0184f771e02ebaa30e143840f9c8a4e3d34, covering the full 17-file diff against merge-base 6f25df70b3825b68a3f9eec01e67de36c0e0bcb8, the existing discussions, and current CI. The base branch was 424c31aa79d13fddf743ffa29bae3c6f146e6c5e at the final recheck.
No remaining actionable issues found. Approving.
The automatic native route is limited to analyzer-admitted, non-null UTF8_BINARY literals. I rechecked class boundaries, quantifier/capture budgeting, native ownership and error propagation, NULL and dynamic patterns, collation on either operand, incompatible opt-in, and dispatcher-disabled fallback. The previous lint finding is fixed. The documentation now describes both routes and links to the subset definition, and the dependency comment records the parity-fixture follow-up.
Validation
- Built native code and compiled the JVM code locally at this exact head. The root-reactor run of
CometRegexSuite,CometRegexParitySuite,CometRegExpJvmSuite, andGenerateDocsSuitepassed 92 tests, with zero failures. One test was skipped because it requires Spark 4.0/JDK 21. This run used Spark 4.1.3/JDK 17 and the freshly built native library, with the packaged library checksum verified. - Fresh component differential checks produced 29,284,536 Java/Rust comparisons with zero mismatches against locked
regex 1.13.1, including JDK 17/21 and adversarial class, grammar, and Unicode cases. This count includes overlapping corpora and repeated checks across JDKs. - 3,177 admitted compile-stress patterns compiled successfully in both engines, including nested counted products reaching the structural expansion limit. Independent planning probes also passed 72 conversion checks and 32 support-level/hint checks.
- CI has 70 successful checks, eight skips, and no failed or pending checks. I inspected the Spark 4.1 expression log and Spark 4.0/JDK 21 expression log. The latter passes the full capture-pattern regression skipped locally. These CI jobs tested merge
c70aa54dfe07929f39e87a32c38a946a77ebb1b3, combining this head with80c8ec37af422db1952a1128cb23043c9bbfbb72.
The compile budget remains heuristic, and finite differential checks do not prove every admitted pattern equivalent. The new random nested-pattern corpus used subjects of at most four codepoints after longer inputs caused excessive Java backtracking. Malformed UTF-8 and performance benchmarks were not validated in this pass. Follow-ups #5811, #5812, and #5813 remain tracked.
Add a plan-time whitelist so UTF8_BINARY literals the analyzer can prove equivalent to Java regex take the native path without allowIncompatible. Out-of-subset literals stay on the JVM dispatcher.
…e budget Reject Rust-only character-class operators and unescaped ] range endpoints so incompatible literals stay on the JVM dispatcher. Cap nested counted expansion and group depth so Java-valid patterns that Rust cannot compile do not fail native plan construction.
Reject unescaped '[' as a character-class range endpoint so malformed Java patterns like [@-[] stay on the JVM dispatcher and preserve Spark regex errors. Apply MaxExpansion across concatenation, alternation, and counted repetition, including zero-count terms, so sibling groups and repeated atoms cannot bypass compile-size admission.
Keep `{0,}` from collapsing inner cost so nested patterns like
`(([^;]{256}){0,}){256}` stay on the JVM dispatcher. Escape
backslashes in the rlike benchmark SQL so `\d+` is not parsed as `d+`.
Native rlike does not implement Spark collation, so opt-in must still require default collation. Document compile-budget constants as structural heuristics and assume well-formed UTF-8 rather than a confirmed malformed-input split.
Stacked `*`/`+`/`?`/`{n}` do not raise the structural expansion estimate,
so patterns like the 105-byte nested-star repro compiled past the Rust
regex 10 MiB limit. Cap quantifier nesting and keep those literals on the
JVM dispatcher.
Capture start/end states are copied when `{n}` expands an expression.
Add that cost before counted multiplication so residual patterns that
compiled past the Rust 10 MiB limit stay on the JVM dispatcher.
Non-capturing groups and ordinary negated classes such as `[^x]{256}`
remain admitted.
Charge capture-state cost when each capturing group is parsed so it propagates through concatenation, alternation, and uncounted repetition, not only counted copies. Keep the full uncounted-capture reproducer on the JVM dispatcher by default and fall back to Spark when the dispatcher is disabled.
622db9a to
7ef89f0
Compare
Which issue does this PR close?
Closes #5351.
Rationale for this change
rlikealready has a native kernel, but the Rust regex implementation is treated as potentially incompatible with Java regex and is therefore opt-in. Before this PR, literal patterns ran through the JVM codegen dispatcher by default unlessspark.comet.expression.RLike.allowIncompatible=true.#4310 correctly concluded that the Rust engine cannot be made fully compatible with Java regex, so this PR does not change the engine globally. Instead, it introduces a deliberately restricted, conservative whitelist for plan-time
UTF8_BINARYliteral patterns.Patterns admitted by the whitelist use native execution by default and are covered by Java-versus-Rust differential tests. Out-of-subset non-null literals preserve the existing dispatcher /
allowIncompatiblebehavior. Non-literal and NULL patterns use the dispatcher; NULL is never considered a usable native pattern.What changes are included in this PR?
CometRegex, a recursive-descent parser implementing a conservative whitelist rather than relying on substring matching. Unrecognized constructs areIncompatible.*+?{n}{n,}{n,m}, capturing and non-capturing groups ((?:…)), alternation, and escaped metacharacters, provided counted and nested forms remain within conservative native compile-size and depth budgets.^$.\d\w\s, lookaround, backrefs, possessive/lazy quantifiers, inline flags,\p/\u/\0, nested classes, Rust-only class set operators (&&,~~,--), unescaped]class atoms or range endpoints, raw[range endpoints, counted or nested patterns exceeding the conservative compile budget, non-ASCII patterns, and non-default Spark 4 collation on either operand.{n,}, a lower bound of zero still retains the inner expression's compilation cost; exact-zero{0}and{0,0}repetitions remain distinct.CometRLikeconsults the analyzer. In-subset literals becomeCompatible()with nonativeOptInhint and convert to native without opt-in.Compatible(nativeOptIn = …)and convert to the dispatcher unlessallowIncompatible=true.literalPatternonly matches a non-nullUTF8Stringliteral).allowIncompatiblecan force native execution for a non-null default-collation literal. Non-default collations remain on the JVM dispatcher because the native kernel does not implement Spark collation semantics.rlikeoperates on Arrow string values and assumes well-formed UTF-8; behavior outside Arrow's StringArray invariants has not been established.compatibility/regex.md,expressions.md(rlike/regexp/regexp_like), and therlikeexpression-audit note.(?-u), propagate collation into the native kernel, or changeregexp_replace,split,regexp_extract, orregexp_extract_all.A conservative whitelist identifies a deliberately restricted subset for which Java and Rust
findsemantics are expected to agree and are covered by differential parity tests. It is not a formal proof of equivalence.How are these changes tested?
CometRegexSuite: admit / reject coverage, including:[(?=],\\d, and[.];\A,\Z,{2,1}, and an unclosed(;]and raw[class-range boundaries;{0,}regression(([^;]{256}){0,}){256};{0}and{0,0}controls.CometRegexParitySuite: every pattern in the admitted corpus × ASCII, non-ASCII, control-character, exotic-whitespace, newline, empty, and NULL subjects, with native results compared againstjava.util.regex.Pattern.find. The subject corpus includes TAB, VT, FF, NUL, DEL, NEL, NBSP, BOM, U+2028, and U+2029. The suite also asserts that EXPLAIN output does not containJVM codegen dispatcher: rlike. The multi-batch corpus contains 5000 rows with batch size 64. NULL subjects match Spark; a NULL pattern is not part of this native corpus.CometRegExpJvmSuite: routing coverage for:allowIncompatible=true, including expected native compilation failures for unsupported Rust constructs;allowIncompatible=true;{0,}regression is verified against Spark using non-foldable input, including NULL, and remains on the JVM dispatcher.rlike_auto_native.sqlverifies default-configuration result equality.CometRegexParitySuite,CometRegExpJvmSuite, andCometRegexSuitecompleted successfully on Spark 4.1 / Scala 2.13 and Spark 3.5.9 / Scala 2.12:The two
allowIncompatiblecollation regressions are Spark 4-only (assume(isSpark40Plus)) and are skipped on 3.5.Other supported Spark and Scala profiles are covered by CI.
Benchmark
CometRegExpBenchmark, 1,048,576 rows, Apple M4, JDK 17.Native vs Spark for in-subset patterns
After this PR, these patterns use native execution by default.
[0-9]+abc|def|ghi[a-zA-Z][0-9]+(ab){2,}Default Comet execution before and after this PR
These measurements use identical in-subset queries. Both modes were measured on the PR base commit in the same run: the JVM dispatcher represents the pre-PR default, while
allowIncompatible=trueselects the same native kernel that this PR now chooses automatically.[0-9]+abc|def|ghi[a-zA-Z][0-9]+(ab){2,}On this ASCII
REPEATworkload, the pre-PR dispatcher is essentially Spark-cost. Switching the in-subset default from the dispatcher to native execution is the performance improvement that matters for this PR.Out-of-subset control
\d+remains outside the automatic-native subset because Java and Rust differ in their Unicode digit semantics. It uses the JVM dispatcher by default;allowIncompatible=trueexplicitly selects the native path.The benchmark SQL preserves the backslash so Spark receives
\d+rather than parsing it asd+.\d+The corrected JVM-dispatcher measurement is close to Spark cost, as expected. The explicitly opted-in native path is approximately 4.7X faster than the dispatcher on this ASCII workload.
The native implementation matches over Arrow buffers and is expected to avoid the dispatcher's per-row
toString()/Matcherallocations. This allocation difference was not measured with JFR or a GC log in this PR.