You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
when cell content exceeds cell boundaries, next cell gets messed up (exmples) #538
Hi @shula Unfortunately this is expected behavior for a PDF with this kind of problem. The "extra"/unexpected characters (for example AL YPT in line 1068) are present, but under the text for the next cell to the left. So Tabula is correctly extracting the characters.
I'm running into this problem too, attempting to "free" data locked inside some legal filings.
@jeremybmerrill Can you comment as to whether the data still exists in uncorrupted form inside the PDF, and where in the code I might attempt to patch the extraction logic? Or barring that, suppress the overflow text so it doesn't corrupt its neighbors? I'm experienced in Java, but not PDF or this particular codebase. Thanks.
@apgove It's difficult to generalize about PDFs; they're a bit like Dostoyevsky's idea of families. Happy PDFs are all alike, but unhappy PDFs are unhappy in their own ways. Except it's you that's unhappy, not the PDF.
Unfortunately, there's likely not a generalizable way to "fix" this problem. There's no sense of a table; it's just a bunch of lines that happen to be arranged in a way that humans perceive as a table. Those lines comprising the table are stored totally separately from the text. So the text doesn't know what cell it's in; the cells don't know what text they are supposed to contain. From the PDF's perspective, the only sense in which the text is "in" a cell is whether it is contained within the bounding box created by the lines, and, thus, if text appears to overflow into the "next" cell, it IS in the next cell.
You could perhaps do something funny with detecting overlapping text and then using language modeling to predict which overlapping character belongs in the "previous" cell, but that would be nondeterministic and probably wouldn't work! In certain cases, you could be lucky enough that you could heuristically determine based on (for instance) text size or Y-position which overlapping characters belong in which cell for your specific document, but that wouldn't generalize.
Thanks for the explanation @jeremybmerrill , it's much appreciated to avoid wasting [even more] time on a wild goose chase. I'm guessing, given that the overlapping characters get intermingled in the output, that each glyph is represented independently, not as strings, making this quite challenging.
Just spitballing (please forgive the uninformed speculation, I've been on the receiving end and it can be really annoying!), but there must be some sort of z-order information stored somewhere in order for the overrunning characters to get hidden; I wonder whether that could be used as a signal that the hidden characters belong in the previous cell, not the one they're contained by. Of course, that would still fail if it overran more than 1 column, but it could be a big improvement. Or... Maybe if we have enough information to know that a character is being suppressed due to overlap, it can just be skipped over and not added to the output for any table cell. Information would be lost, but it would at least avoid corruption and accurately reflect the visual representation of the original PDF.
Anyways, maybe I'll dive down this rabbit hole later, but for now I'll pursue other leads.
@apgove Feel free to give it a try! I don't think there's likely to be a free lunch in the general case, but it's possible that a there may be heuristics for the PDFs in your use case that would resolve your problem.
TextPosition is the underlying representation of a character (or a string of a few characters) in the underlying PDFBox library. Some PDFs have one TextPosition per glyph; others have a few characters in a single TextPosition. Because PDFs are primarily a display format not a data transfer format, different PDF generators do different things (and I don't think standards, at least not early ones, defined a right way to do it).
There's no explicit Z-index, although there's an underlying ordering that might correspond to what you're looking for, for your specific documents. Only way to know is to try.
I should add this caveat: Unless you've got thousands of PDFs or need to build a pipeline that ingests PDFs into the future, you might spend more time investigating this than it would take to fix the misbehaving cells manually!
Activity
jeremybmerrill commentedon Feb 20, 2024
Hi @shula Unfortunately this is expected behavior for a PDF with this kind of problem. The "extra"/unexpected characters (for example
AL YPTin line 1068) are present, but under the text for the next cell to the left. So Tabula is correctly extracting the characters.apgove commentedon Dec 14, 2024
I'm running into this problem too, attempting to "free" data locked inside some legal filings.
@jeremybmerrill Can you comment as to whether the data still exists in uncorrupted form inside the PDF, and where in the code I might attempt to patch the extraction logic? Or barring that, suppress the overflow text so it doesn't corrupt its neighbors? I'm experienced in Java, but not PDF or this particular codebase. Thanks.
jeremybmerrill commentedon Dec 15, 2024
@apgove It's difficult to generalize about PDFs; they're a bit like Dostoyevsky's idea of families. Happy PDFs are all alike, but unhappy PDFs are unhappy in their own ways. Except it's you that's unhappy, not the PDF.
Unfortunately, there's likely not a generalizable way to "fix" this problem. There's no sense of a table; it's just a bunch of lines that happen to be arranged in a way that humans perceive as a table. Those lines comprising the table are stored totally separately from the text. So the text doesn't know what cell it's in; the cells don't know what text they are supposed to contain. From the PDF's perspective, the only sense in which the text is "in" a cell is whether it is contained within the bounding box created by the lines, and, thus, if text appears to overflow into the "next" cell, it IS in the next cell.
You could perhaps do something funny with detecting overlapping text and then using language modeling to predict which overlapping character belongs in the "previous" cell, but that would be nondeterministic and probably wouldn't work! In certain cases, you could be lucky enough that you could heuristically determine based on (for instance) text size or Y-position which overlapping characters belong in which cell for your specific document, but that wouldn't generalize.
apgove commentedon Dec 15, 2024
Thanks for the explanation @jeremybmerrill , it's much appreciated to avoid wasting [even more] time on a wild goose chase. I'm guessing, given that the overlapping characters get intermingled in the output, that each glyph is represented independently, not as strings, making this quite challenging.
Just spitballing (please forgive the uninformed speculation, I've been on the receiving end and it can be really annoying!), but there must be some sort of z-order information stored somewhere in order for the overrunning characters to get hidden; I wonder whether that could be used as a signal that the hidden characters belong in the previous cell, not the one they're contained by. Of course, that would still fail if it overran more than 1 column, but it could be a big improvement. Or... Maybe if we have enough information to know that a character is being suppressed due to overlap, it can just be skipped over and not added to the output for any table cell. Information would be lost, but it would at least avoid corruption and accurately reflect the visual representation of the original PDF.
Anyways, maybe I'll dive down this rabbit hole later, but for now I'll pursue other leads.
jeremybmerrill commentedon Dec 16, 2024
@apgove Feel free to give it a try! I don't think there's likely to be a free lunch in the general case, but it's possible that a there may be heuristics for the PDFs in your use case that would resolve your problem.
TextPosition is the underlying representation of a character (or a string of a few characters) in the underlying PDFBox library. Some PDFs have one TextPosition per glyph; others have a few characters in a single TextPosition. Because PDFs are primarily a display format not a data transfer format, different PDF generators do different things (and I don't think standards, at least not early ones, defined a right way to do it).
There's no explicit Z-index, although there's an underlying ordering that might correspond to what you're looking for, for your specific documents. Only way to know is to try.
I should add this caveat: Unless you've got thousands of PDFs or need to build a pipeline that ingests PDFs into the future, you might spend more time investigating this than it would take to fix the misbehaving cells manually!