Skip to content

[Bug]: JSONLinesFileReader drops rows containing U+2028, U+2029 or U+0085 (uses str.splitlines) #2594

Open

Description

Do you need to file an issue?

  • I have searched the existing issues and this bug is not already filed.
  • My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here. (Not applicable: the bug is in the input reader, before any model call.)
  • I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.

Describe the bug

JSONLinesFileReader.read_file splits the file with text.splitlines() (jsonl.py#L38). Python's str.splitlines() also breaks on U+2028 (LINE SEPARATOR), U+2029 (PARAGRAPH SEPARATOR) and U+0085 (NEXT LINE). JSON Lines uses only \n as the record separator, and those three characters are valid unescaped inside JSON strings. A valid row containing one of them is therefore cut into two fragments, neither of which parses. Each fragment is logged as Skipping malformed JSONL row in <path> at line N, so that row is dropped from the input (the rest of the file is still read), and the only trace is a warning.

(Side note: splitlines() also splits on \x0b, \x0c, \x1c, \x1d and \x1e. Those are control characters that must be escaped in JSON, so they cannot occur unescaped in a valid row and are not part of this bug.)

Steps to reproduce

File x.jsonl, where the first row contains a raw U+2028 between a and b (shown here as <LS>):

{"text":"a<LS>b"}
{"text":"c"}
docs = await JSONLinesFileReader(storage=storage).read_files()
print([d.text for d in docs])   # ['c']   (expected ['a
b', 'c'])
# log: Skipping malformed JSONL row in x.jsonl at line 1
#      Skipping malformed JSONL row in x.jsonl at line 2

json.loads('{"text":"a
b"}') parses fine on its own. The same happens with U+2029 and U+0085.

Expected Behavior

Both rows are read. Split on "\n" only and strip a trailing "\r" for CRLF files.

GraphRAG Config Used

input:
  type: jsonl

Logs and screenshots

Skipping malformed JSONL row in x.jsonl at line 1
Skipping malformed JSONL row in x.jsonl at line 2

Additional Information

Suggested fix: for line_number, line in enumerate(text.split("\n"), start=1): line = line.rstrip("\r"), plus a test with U+2028 and U+0085 inside a string value.

Activity

  1. nefayran commented on Oct 10, 2026

    @nefayran

    PRs here are limited to collaborators, so the fix is on a branch instead: main...nefayran:graphrag:fix/jsonl-unicode-line-separators. It splits on "\n" only (a trailing \r is whitespace to json.loads), adds a test with U+2028, U+2029 and U+0085 rows in a CRLF file, and a patch entry under .semversioner/next-release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions