Skip to content

feat: include field paths and row locations in JSON conversion errors - #11085

Merged
Xuanwo merged 1 commit into
lance-format:mainfrom
shu-chen-dot-ca:fix-10996-json-error-context
Nov 18, 2026
Merged

Xuanwo merged 1 commit into
lance-format:mainfrom
shu-chen-dot-ca:fix-10996-json-error-context

Conversation

@shu-chen-dot-ca

Copy link
Copy Markdown
Contributor

Closes #10996.

A JSON text value that fails to parse during convert_json_columns (the conversion every dataset write runs on Arrow JSON columns) used to fail with only Failed to encode JSON: <parser error>. The error now names the field path, the row within the batch being converted and, for a value inside lists or maps, its position in each of them, outermost first:

Failed to encode JSON in field 'data' at input batch row 2: expected ident, pos 3
Failed to encode JSON in field 'media.extra' at input batch row 2, list element positions [1]: expected ident, pos 3
Failed to encode JSON in field 'matrix' at input batch row 1, list element positions [1, 1]: expected ident, pos 3

The leaf records the failing index in the array it converts, and convert_json_array adds context while the error propagates up: a struct adds the child's name, and a list, large list, fixed-size list or map maps the child index to its own row through its offsets and records the element position. List item and map entry fields add no name, so a map value reads attrs.value. The row is the index within the converted batch, not a dataset row ID; the convert_json_columns doc comment says so. A value of a sliced list that no row references is reported as in list values not referenced by any input batch row rather than given a wrong row.

The parser error is kept as is, nulls are still skipped without parsing, and the error is still ArrowError::InvalidArgumentError (a Lance Error::Arrow on the write path). JsonArray::try_from on a string array keeps the old Failed to encode JSON: ... message, since it has no field to name. Context is only built on the error path, so successful conversions do the same work as before.

How was this patch tested

  • test_convert_json_columns_invalid_json_location (rstest, replaces the two is_err()-only invalid JSON tests) asserts the full message for a top-level Utf8 column with a null before the bad row, LargeUtf8, a struct child, list of struct with an empty row before the failing one, nested large lists, a fixed-size list, a map value and an unreferenced value of a sliced list. On main all eight cases fail with the bare message.
  • test_json_array_invalid_json_keeps_parser_error pins the unchanged JsonArray::try_from message.
  • rust/lance/tests/json_write writes a dataset through Dataset::write with an invalid value in a top-level JSON column (a sliced array, so the batch row is not the array's original index) and in list<struct<json>>, and checks the Error::Arrow message.

Commands run:

  • cargo test --profile ci --locked -p lance-arrow: 255 passed, 6 doctests passed
  • cargo test --profile ci --locked -p lance --test integration_tests: 114 passed
  • cargo clippy --locked -p lance-arrow --tests --benches -- -D warnings and cargo clippy --locked -p lance --test integration_tests -- -D warnings: clean
  • cargo fmt --all -- --check: clean
  • python3 ci/check_module_size.py origin/main HEAD: no findings

The workspace-wide cargo clippy --all --tests --benches and the lance lib test build were not completed locally: the machine I ran this on ran out of memory compiling the lance lib test target, so the write-path test lives in the integration test binary and clippy covered the two targets above.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added the enhancement New feature or request label Nov 15, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 15, 2026
@shu-chen-dot-ca

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (e5b2fb9) to resolve the conflict in rust/lance/tests/integration_tests.rs: main added mod label_list_optimize; where this PR adds mod json_write;, so both are kept in alphabetical order. No other changes. cargo test --profile ci --locked -p lance-arrow and -p lance --test integration_tests (126 passed) pass on the rebased branch, along with fmt, lance-arrow clippy and the module size check.

@shu-chen-dot-ca
shu-chen-dot-ca force-pushed the fix-10996-json-error-context branch from 61a471b to 6ba2c34 Compare November 15, 2026 23:15
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 15, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The rebase keeps the JSON implementation and write tests unchanged and retains both integration-test modules. No merge-blocking findings; the diagnostics preserve JSON encoding, null behavior, and parser errors.

Focused JSON conversion tests passed on this revision.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 15, 2026
@Xuanwo
Xuanwo merged commit 5e9cc43 into lance-format:main Nov 18, 2026
44 of 48 checks passed
@shu-chen-dot-ca
shu-chen-dot-ca deleted the fix-10996-json-error-context branch November 18, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature: include field paths and row locations in JSON conversion errors

2 participants