Repository navigation
fix(data): harden CSV health checks - #2388
OKAN TIKTAS (OKAN-SOFTWARE) wants to merge 4 commits into
Conversation
Allow CSV-only health checks without a Qlib directory, report missing fields and missing factors without crashing, support CSV date columns for large price-change reports, and include missing-data-only failures in the summary. Add seven self-contained regression tests using synthetic CSV data. Refs microsoft#854.
|
@microsoft-github-policy-service agree |
|
Ran the PR head's
|
|
Thanks Arhan Canli (@arhancanli) for catching this and providing a clear reproduction! Fixed in I also added two regression tests covering missing values in the factor column and unrelated extra columns. Validation: 9 tests passed, and Appreciate the careful review! |
|
Read the diff of |
|
Arhan Canli (@arhancanli) Thanks for the follow-up! Completely missing OHLCV columns are already handled by I kept that separate from The regression test I think keeping those responsibilities separate makes the diagnostics clearer, but I'm happy to adjust if you'd prefer a unified report. |
|
Checked the diff: |
|
Thanks for the suggestion! I've updated I also added a regression test covering a CSV with both a missing required column and NaN values, verifying the summary order. Updated in Validation:
Thanks again for the careful review! |
|
Checked 1afa8f8: the required-columns block now prints before the missing-data block, and the new test builds the right input (CSV without |
|
Thanks for the careful review! I've strengthened the regression test to compare the actual rendered tables rather than relying on a column-name substring. I also verified the test against both implementations:
This confirms that the regression test detects the original ordering problem. The implementation itself remains unchanged. Thanks again for the helpful suggestion! |
|
Checked 566c039: the test now builds both rendered tables and asserts |
|
Thanks for confirming the ordering fix! I checked the current implementation. That's a worthwhile additional health check. I'd prefer to keep it separate from this PR's current fixes so we don't introduce unrelated behavior without dedicated tests. A follow-up could validate duplicate and non-monotonic dates in CSV mode, while handling Qlib's MultiIndex dates appropriately. Thanks for pointing out the gap! :) |
|
Fair to keep it out of this PR. For the follow-up, two details are worth putting in its tests: in pandas, Is the duplicate case something you have hit in real CSV dumps, and if so, did it come from merged downloads or from a single vendor file? That would tell me which fixture to model. |
|
Thanks for the additional details! Good point about monotonicity not detecting duplicate dates. I don't have a specific real-world CSV dump I can confidently attribute to merged downloads or a single vendor, so I'd keep the initial regression fixtures source-neutral. For the follow-up, I'd cover duplicate dates and out-of-order dates separately, with per-instrument validation for Qlib MultiIndex data. That should give us deterministic coverage without making assumptions about the source of the duplicates. Appreciate the guidance! |
Description
This PR improves the reliability of Qlib's data health checker when validating CSV-based financial datasets.
Changes
Motivation and Context
Incomplete or malformed financial datasets can cause unexpected errors during health checks. These improvements make the existing checker more resilient and help developers identify data-quality problems.
Related issue: #854.
How Has This Been Tested?
Seven regression tests have been added in
tests/test_data_health_checker.py.The tests cover CSV initialization, missing fields, missing factor data, large price changes, date handling, and summary generation.
Test command:
pytest -q tests/test_data_health_checker.pyExecution results are pending verification. GitHub Actions requires maintainer approval.
Types of Changes