Repository navigation
Conversation
andygrove
left a comment
There was a problem hiding this comment.
Thanks for picking this up @mizulun. The fixture design lines up well with what #5813 asked for. Failures report the pattern, subject, expected and actual answers, and I like that CometRegexSuite re-checks the stored answers against whatever JDK CI runs on, so a Java behavior change surfaces too.
I checked out the branch and ran a few things locally. Regenerating on JDK 17.0.17 produced byte-identical cases, with only the jdk_runtime_version line differing. cargo test -p datafusion-comet-spark-expr --lib rlike passed 7/7, and the fixture test takes about 5s in a debug build.
A couple of things that don't fit on a diff line:
- Would you add a short comment near the limit constants in
CometRegex.scalapointing atdev/GenerateRegexFixtures.javaand the new doc page? If someone admits a new construct or raisesMaxExpansionorMaxCountedBound, the fixtures keep passing but no longer probe the new edge. A pointer next to the caps seems like the most likely place for that person to see it. - Could you update the testing section of the description once
CometRegexSuiteandCometRegexParitySuitehave run against the corrected fixtures? Right now it says that run is pending.
| Use `git push` for normal updates to a PR branch. If a rebase or amend requires a force push, | ||
| use `git push --force-with-lease`, never `--force` or `-f`, to reduce the risk of overwriting | ||
| another maintainer's commits. If the lease check rejects the push, inspect and integrate the | ||
| remote changes before retrying; do not bypass it with `--force`. See | ||
| [Submitting a Pull Request](docs/source/contributor-guide/development.md#submitting-a-pull-request). |
There was a problem hiding this comment.
The --force-with-lease guidance here and in development.md looks unrelated to the regex fixtures. Could you move it to its own PR? AGENTS.md is the shared agent guidance file, so changes there are worth discussing on their own, and it keeps this PR focused on #5813.
| java dev/GenerateRegexFixtures.java spark/src/test/resources/regex/rlike-java-fixtures.json | ||
| java dev/GenerateRegexFixtures.java /tmp/rlike-java-fixtures.json | ||
| cmp spark/src/test/resources/regex/rlike-java-fixtures.json /tmp/rlike-java-fixtures.json |
There was a problem hiding this comment.
The first command overwrites the committed JSON before the cmp, so the comparison is between two fresh runs rather than against the committed file. Could you split this into two steps? First, generate to /tmp and cmp against the committed file to check it is current. Second, regenerate in place only when you mean to change the fixtures. That also makes it clearer which step to run after a JDK or generator change.
| "[a-]", | ||
| "[-a]", | ||
| "[a\\-z]", | ||
| "[@-\\[]", |
There was a problem hiding this comment.
The whitelist admits ranges whose start is an escaped character, such as [\.-9], [\--/] and [\\-a], but the fixtures only cover an escaped end ([@-\[]). Could you add a few of those? I checked them against regex 1.13.1 and they agree with Java today. Since the point of the table is to catch a future crate change, it would be good to have every admitted class shape in it.
| } | ||
| add("group-depth-32", "(".repeat(32) + "a" + ")".repeat(32), "", "a", "b"); | ||
| add("counted-bound-256", "a{256}", "", "a".repeat(255), "a".repeat(256)); | ||
| add("quantifier-depth-8", "(?:".repeat(8) + "a" + "){1}".repeat(8), "", "a", "b"); |
There was a problem hiding this comment.
quantifier-depth-8 nests {1}, which a regex engine can optimize away. CometRegex.MaxQuantifierNesting, however, is intended to bound stacked unbounded quantifiers, whose nullable compile state is not captured by the structural expansion estimate.
CometRegexSuite checks only that nestedStars(8) is admitted and nestedStars(9) is rejected. It does not compile and evaluate the admitted boundary in Rust. Could you add nestedStars(8) to the Java/native fixture corpus so test_rlike_java_fixtures exercises it against the Java oracle?
((((((((a)*)*)*)*)*)*)*)*
Please include subjects such as "", "a", "aa", and "b" to cover its empty-match and repeated-input behavior.
sunchao
left a comment
There was a problem hiding this comment.
Summary
- Prior state and problem: The existing regex corpus could miss semantic or compilation changes after a Rust
regexupgrade. - Design approach: Add 10,781 committed Java-generated fixtures shared by Rust and Scala tests.
- Correctness / compatibility analysis: Checked Spark implementations and tests for 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0. The oracle matches
RLikesearch semantics for the default collation covered by native admission. - Key design decisions: Bounded generation, recorded JDK provenance and one shared fixture file keep the design straightforward. Added evaluation is test-only and introduces no query execution overhead.
- Implementation sketch: Java generates expected answers, Rust tests scalar and UTF-8 array
RLikeevaluation, and Scala checks admission, Java results and nesting boundaries. - Behavioral changes worth calling out: Production matching and admission remain unchanged. Earlier requests concerning escaped ranges, nested stars and regeneration instructions are addressed.
- Suggested improvements: No introduced P1/P2 issues found within this review. No substantiated existing P1/P2 blockers remain.
Reviewed the entire eight-file diff from base 67803a7a422c44de07af1e5d25c1dbeae8df68d4 to head 726a77f2cb8583a410d07c2d759dba1fd0b781a6. Routed skills: review-comet-pr and review-comet-expression-pr.
Validation: All fixtures regenerated byte-for-byte with the recorded JDK under C and UTF-8 locales. An isolated harness using the locked regex engine versions passed all 10,781 cases. An isolated Scala harness using unchanged CometRegex.scala and stubbed support types passed fixture admission, Java results and 36 nesting-boundary checks on JDK 17 and 21. git diff --check passed.
Exact-head CI: The labeling job passed. Comet CI and CodeQL report action_required, providing no build/test verdict.
Validation limits: The repository Rust test command stopped before compilation because its offline workspace dependency iceberg-rust was uncached. Maven Central access was blocked. Full native/ScalaTest integration suites and release RAT were not run. The isolated checks do not validate Arrow/DataFusion integration.
|
|
||
| ```shell | ||
| java dev/GenerateRegexFixtures.java /tmp/rlike-java-fixtures.json | ||
| cmp spark/src/test/resources/regex/rlike-java-fixtures.json /tmp/rlike-java-fixtures.json |
There was a problem hiding this comment.
This cmp will report a difference on any JDK build other than the recorded Eclipse Adoptium 17.0.20.1+1, because lines 2 and 3 of the JSON hold jdk_vendor and jdk_runtime_version. I ran the generator from the PR head with JDK 17.0.19 (Apple build) and the only differing lines were those two, but cmp still exits non-zero. A contributor on another vendor or patch release would read that as a stale fixture file. Could the check skip the header, for example diff <(sed 1,3d spark/src/test/resources/regex/rlike-java-fixtures.json) <(sed 1,3d /tmp/rlike-java-fixtures.json), and could the text say that the two header lines are expected to differ?
| ))) | ||
| .expect("valid Java regex fixtures"); | ||
| assert!(!fixtures.cases.is_empty()); | ||
| for fixture in fixtures.cases { |
There was a problem hiding this comment.
Every failure path in this loop panics, so a regex upgrade that changes results reports only the first affected pattern and the developer has to fix, rerun and repeat to see the rest. For an upgrade the useful output is the whole set of affected (pattern, subject) pairs. Could the loop push each context string into a Vec<String> and assert it is empty after the loop, printing the count and the first N entries? The compile error case can follow the same route by matching on RLike::try_new instead of unwrapping.
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
I re-read the PR against the current head and my earlier comments. The AGENTS.md and development.md change is gone from the diff. The regeneration steps in regex-fixtures.md now generate to /tmp and compare before anything overwrites the committed file. The matrix covers escaped range starts such as [\.-9], [\--/] and [\\-a], and it goes further by covering every admitted escape as a start, an end and all ordered pairs. The limit constants in CometRegex.scala now point at the generator and the doc page. The star-depth-8 case also covers the stacked-stars boundary that sam-1112 asked for.
The two open threads from comphead on regex-fixtures.md (the cmp including the jdk_vendor and jdk_runtime_version header lines) and on rlike.rs (collecting every mismatch before asserting) still look worth resolving. Of the two, I think collecting all failures matters most. A regex upgrade is the exact situation this test exists for, and seeing the whole affected set of (pattern, subject) pairs in one run is much more useful than fixing the first panic and rerunning.
I did not find other concerns. Compared with the latest release branch, production code is unchanged apart from a comment in CometRegex.scala, so there is no behavior change. I read the fixture test in rlike.rs, the new CometRegexSuite tests, and the generator, and checked that serde_json and serde are already dependencies of the crate and that MaxQuantifierNesting is visible to the suite.
andygrove
left a comment
There was a problem hiding this comment.
I regenerated the fixtures on JDK 17.0.10 and got the same 10,781 cases. I also flipped one expected value and confirmed test_rlike_java_fixtures fails with the pattern, subject and actual result. This looks good to me.
Nit, optional: comphead's two threads are still open, and I think the first matters most. Could the loop in test_rlike_java_fixtures collect each failing (pattern, subject) and assert once at the end? After a regex upgrade, the full affected set in one run is the useful output. The cmp in regex-fixtures.md could also skip the first three lines, since the JDK header differs on any other build.
Which issue does this PR close?
Closes #5813.
Rationale for this change
The
CometRegexwhitelist depends on the behavior of the Rustregexcrate. A dependency upgrade could change matching results or reject an admitted pattern without the existing corpus detecting it.Committed Java-generated fixtures provide a stable baseline for detecting these changes directly in Rust tests.
What changes are included in this PR?
Pattern.matcher(subject).find()as the oracle.RLike, reporting fixture details on compilation errors or result mismatches.CometRegexSuite.The Rust fixture test does not start a JVM. Production execution and whitelist behavior are unchanged.
How are these changes tested?
cargo test -p datafusion-comet-spark-expr --lib rlike: all 7 tests passed against the updated fixtures../mvnw test -Dtest=none -Dsuites=org.apache.comet.expressions.CometRegexSuite,org.apache.comet.CometRegexParitySuite: all 19 tests passed on the default Spark 4.1 profile with JDK 17.make core: passed../mvnw spotless:check apache-rat:check: passed.git diff --check: passed.