Repository navigation
Testing of distributions #357
Description
We should add some testing for distributions. Unfortunately since distribution samples are random, this is inherently difficult. Original issue dhardy#72.
See for some direction rust-lang/rust#10084, and the random-tests.
Metadata
Metadata
Assignees
Labels
No labels
Activity
vks commentedon Mar 30, 2018
Also see #290.
dhardy commentedon Aug 4, 2018
@vks has some plans for histogram-based testing, though possibly only after PDFs are added.
Separately, we should add some tests of value-stability.
vks commentedon Aug 4, 2018
See
tests/uniformity.rs, which could be generalized to arbitrary distributions (with given PDFs).dhardy commentedon Nov 6, 2019
@vks could I interest you (or potentially someone else) in adding at utility to
rand_distr(perhaps an example) which plots histograms of each distribution from actual samples (not PDFs)? This would at least let us eye-ball that distributions are doing roughly the right thing.vks commentedon Nov 6, 2019
I was thinking about using sparklines for that. I'm interested, but I can't make promises that I'll get to it before the end of the year.
vks commentedon May 15, 2021
Resolving this issue was started in #1121, more work is needed to extend it to the other distributions.
vks commentedon Apr 26, 2024
I think we should either make this issue more actionable or close it.
For which distributions do we want histogram tests? Sometimes, calculating the PDF is non-trivial and requires special functions.
dhardy commentedon Apr 27, 2024
I suggest the following approach:
This should allow fast and robust testing (within the limits of reproducibility), with the caveat that any value-breaking change must again be visually confirmed.
It should be applicable to all distributions, though may be hard to visualise for multi-dimensional outputs.
It cannot confirm perfect accuracy, especially in degenerate cases.
jambochen commentedon Sep 30, 2024
I am a university student with a background in statistics and would love to contribute to this repository. I have a few questions regarding this issue. I noticed that histograms are used in the tests for visual inspection by comparing the expected curve with the sample curve.
Would it be possible to use a chi-square test for validation? The process should be similar, but hypothesis testing might provide more convincing results.
I wrote a simple test for this: https://github.com/JamboChen/distr_test. The code is rough, and I'm unsure if this is the right approach to use here, I'd appreciate any feedback.
dhardy commentedon Sep 30, 2024
@jambochen feel free to take a look at #1494 by @benjamin-lieser implementing a Kolmogorov-Smirnov test.
dhardy commentedon Oct 17, 2024
Implemented in #1504 thanks to @jambochen and @benjamin-lieser.
Failure cases:
(100, 50, 49): Hypergeo fix #15101e91.844e19dhardy commentedon Nov 13, 2024
We have a PR for the hypergeo issue and #1515 for the Poisson issue, so this can be closed.