You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have compiled a text file that contains erroneous copyright detection values. I have removed quote characters and separated each copyright value by several lines. bad-copyright-detections.txt
We could refine the candidate detection when in binaries such that we are more strict in that case?
We could improve the crude/dump "Named Entity Recognition"(NER) we have today to skip some. There are features in NLTK and we could give a shot to https://github.com/explosion/spaCy and several other NLP tools that help there
Note that the adoption of https://github.com/nexB/pygmars/ as a replacement for NLTK should allow the easier reuse and integration of other libraries in the lexing process including NER and giberish detection. In pygmars, a tokenization rule can now be an arbitrary callable that behaves like re.match so it may enable using other tools as part of token recognition aka. lexing
I still think we need better performance to integrate, i was looking into the other implementation which is a library. There are some additional steps there so I'll try that with the same data. Also thing to note is this only uses positive training, i.e. only trains on good non gibberish values, so if we could do some negative boosting for legalese gibberish that could improve the performance.
I did extend the character set to include a-z, A-Z, number and other characters.
Let's be mindful not to get too much into the weeds as this can be hairy and yield only small improvements.
It may be best to start using this on copyright before applying to license?
We now have code that performs gibberish detection on strings using a 2-character markov chain, which has been merged in #4610
At a high level, it detects gibberish strings by seeing if a sequence of letters is part or a whole word, two letters at a time. It does this by checking how likely it is to go from one letter to another. The probabilities of going from one letter to another are determined by a model that has been trained on a large set of valid text, where it counts each transition between letters and computes a probability based off of that. These probabilities and thresholds are stored in a model that is saved to a Python pickle.
Activity
pombredanne commentedon Feb 16, 2021
That's an interesting class of errors!
I guess that they all come from binaries? And because it is useful, we cannot stop detecting in binaries.
Some remarks:
These are at first considered as candidates by the copyright detector because they contains
(c)or©(which is normalized to(c)). Then they are somehow matching some of the lexing rules in our part of speech tagger and are tagged as being recognized names somehow, and as names they end up recognized finally by the grammar.That's a gibberish detector.
pombredanne commentedon Jul 7, 2021
Note that the adoption of https://github.com/nexB/pygmars/ as a replacement for NLTK should allow the easier reuse and integration of other libraries in the lexing process including NER and giberish detection. In pygmars, a tokenization rule can now be an arbitrary callable that behaves like re.match so it may enable using other tools as part of token recognition aka. lexing
pombredanne commentedon Jul 10, 2021
Another candidate for gibberish that works quite well is https://github.com/domanchi/gibberish-detector
pombredanne commentedon Aug 19, 2021
See
pombredanne commentedon Sep 17, 2021
I ran this with bad-copyright-detections.txt
pip install gibberish-detectorgibberish-detector train examples/big.txt > big.modelI then loaded this in libreoffice to do some evaluation. Here are the results:
This is pretty good and I would expect even better from using a proper training set.
AyanSinhaMahapatra commentedon Aug 2, 2022
I was testing out the very basic gibberish detector at https://github.com/rrenaud/Gibberish-Detector, which https://github.com/domanchi/gibberish-detector (mentioned above is based on), as it is much easier to integrate.
I used our scancode license texts and rules (minus the test set) as the training data and false positives and some of the license tags as test data.
It is pretty good at detecting non gibberish but is not so good when these are ambigious.
attaching the results of the test here for reference, here the probability is of the text being non-gibberish.
bad-> false positivesgood-> license tagsgibberish-test.csv
pombredanne commentedon Aug 3, 2022
very nice! what's your take on applicability to license then? Did you apply some boosting to legalese words?
AyanSinhaMahapatra commentedon Aug 3, 2022
I still think we need better performance to integrate, i was looking into the other implementation which is a library. There are some additional steps there so I'll try that with the same data. Also thing to note is this only uses positive training, i.e. only trains on good non gibberish values, so if we could do some negative boosting for legalese gibberish that could improve the performance.
I did extend the character set to include a-z, A-Z, number and other characters.
Do you think it's worth spending time on this?
pombredanne commentedon Aug 3, 2022
Let's be mindful not to get too much into the weeds as this can be hairy and yield only small improvements.
It may be best to start using this on copyright before applying to license?
[-]Collection of erroneous copyright detection values[/-][+]Improve gibberish license/copyright detection[/+]pombredanne commentedon Jun 28, 2024
Some samples for copyrights:
pombredanne commentedon Oct 7, 2025
Here is the list of gibberish originally attached:
Explore use of nostril to check copyrights #2402
27 remaining items
Create command to train gibberish detector model #2402
Add arguments to provide custom training data to gibberish detector t…
Run train-gibberish-model during builds #2402
Do not commit gibberish training model #2402
Add docs for new command #2402
Use io.open to handle file operations in gibberish.py #2402
Use io.open to handle file operations in gibberish.py #2402
Update comment #2402
Update comment and revert change to gibberish.py #2402
Update comment and revert change to gibberish.py #2402
Move junk copyrights test into new directory #2402
Move junk copyrights test into new directory #2402
Merge pull request #4610 from aboutcode-org/2402-detect-gibberish-cop…
JonoYang commentedon Jan 13, 2026
We now have code that performs gibberish detection on strings using a 2-character markov chain, which has been merged in #4610
At a high level, it detects gibberish strings by seeing if a sequence of letters is part or a whole word, two letters at a time. It does this by checking how likely it is to go from one letter to another. The probabilities of going from one letter to another are determined by a model that has been trained on a large set of valid text, where it counts each transition between letters and computes a probability based off of that. These probabilities and thresholds are stored in a model that is saved to a Python pickle.
These copyrights are no longer reported.