Repository navigation
feat: include field paths and row locations in JSON conversion errors - #11085
Conversation
|
Rebased onto current main (e5b2fb9) to resolve the conflict in |
61a471b to
6ba2c34
Compare
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The rebase keeps the JSON implementation and write tests unchanged and retains both integration-test modules. No merge-blocking findings; the diagnostics preserve JSON encoding, null behavior, and parser errors.
Focused JSON conversion tests passed on this revision.
Closes #10996.
A JSON text value that fails to parse during
convert_json_columns(the conversion every dataset write runs on Arrow JSON columns) used to fail with onlyFailed to encode JSON: <parser error>. The error now names the field path, the row within the batch being converted and, for a value inside lists or maps, its position in each of them, outermost first:The leaf records the failing index in the array it converts, and
convert_json_arrayadds context while the error propagates up: a struct adds the child's name, and a list, large list, fixed-size list or map maps the child index to its own row through its offsets and records the element position. List item and map entry fields add no name, so a map value readsattrs.value. The row is the index within the converted batch, not a dataset row ID; theconvert_json_columnsdoc comment says so. A value of a sliced list that no row references is reported asin list values not referenced by any input batch rowrather than given a wrong row.The parser error is kept as is, nulls are still skipped without parsing, and the error is still
ArrowError::InvalidArgumentError(a LanceError::Arrowon the write path).JsonArray::try_fromon a string array keeps the oldFailed to encode JSON: ...message, since it has no field to name. Context is only built on the error path, so successful conversions do the same work as before.How was this patch tested
test_convert_json_columns_invalid_json_location(rstest, replaces the twois_err()-only invalid JSON tests) asserts the full message for a top-level Utf8 column with a null before the bad row, LargeUtf8, a struct child, list of struct with an empty row before the failing one, nested large lists, a fixed-size list, a map value and an unreferenced value of a sliced list. On main all eight cases fail with the bare message.test_json_array_invalid_json_keeps_parser_errorpins the unchangedJsonArray::try_frommessage.rust/lance/tests/json_writewrites a dataset throughDataset::writewith an invalid value in a top-level JSON column (a sliced array, so the batch row is not the array's original index) and inlist<struct<json>>, and checks theError::Arrowmessage.Commands run:
cargo test --profile ci --locked -p lance-arrow: 255 passed, 6 doctests passedcargo test --profile ci --locked -p lance --test integration_tests: 114 passedcargo clippy --locked -p lance-arrow --tests --benches -- -D warningsandcargo clippy --locked -p lance --test integration_tests -- -D warnings: cleancargo fmt --all -- --check: cleanpython3 ci/check_module_size.py origin/main HEAD: no findingsThe workspace-wide
cargo clippy --all --tests --benchesand thelancelib test build were not completed locally: the machine I ran this on ran out of memory compiling thelancelib test target, so the write-path test lives in the integration test binary and clippy covered the two targets above.