Skip to content

fix(data): harden CSV health checks - #2388

Open
OKAN TIKTAS (OKAN-SOFTWARE) wants to merge 4 commits into
microsoft:mainfrom
OKAN-SOFTWARE:fix/data-health-checker-csv-854
Open

OKAN TIKTAS (OKAN-SOFTWARE) wants to merge 4 commits into
microsoft:mainfrom
OKAN-SOFTWARE:fix/data-health-checker-csv-854

Conversation

@OKAN-SOFTWARE

@OKAN-SOFTWARE OKAN TIKTAS (OKAN-SOFTWARE) commented Oct 26, 2026 •

Copy link
Copy Markdown

Description

This PR improves the reliability of Qlib's data health checker when validating CSV-based financial datasets.

Changes

  • Support CSV-only validation without requiring a Qlib data directory.
  • Handle missing OHLCV columns without crashing.
  • Detect missing or empty factor columns.
  • Improve reporting of missing values.
  • Support CSV date columns when detecting large price changes.
  • Preserve compatibility with Qlib MultiIndex dates.
  • Include missing-data findings in the final health-check summary.

Motivation and Context

Incomplete or malformed financial datasets can cause unexpected errors during health checks. These improvements make the existing checker more resilient and help developers identify data-quality problems.

Related issue: #854.

How Has This Been Tested?

Seven regression tests have been added in tests/test_data_health_checker.py.

The tests cover CSV initialization, missing fields, missing factor data, large price changes, date handling, and summary generation.

Test command:

pytest -q tests/test_data_health_checker.py

Execution results are pending verification. GitHub Actions requires maintainer approval.

Types of Changes

  • Fix bugs
  • Add new feature
  • Update documentation

Allow CSV-only health checks without a Qlib directory, report missing fields and missing factors without crashing, support CSV date columns for large price-change reports, and include missing-data-only failures in the summary.

Add seven self-contained regression tests using synthetic CSV data.

Refs microsoft#854.
@OKAN-SOFTWARE

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

@arhancanli

Copy link
Copy Markdown

Ran the PR head's scripts/check_data_health.py (pandas, qlib stubbed since CSV mode no longer needs it) on one CSV with valid OHLCV for 6 rows and an all-empty factor column:

check_missing_data():
             open  high  low  close  volume
a.csv           0     0    0      0       0
check_missing_factor():  missing_factor_col False, missing_factor_data True

check_missing_data now flags a file via (missing_counts > self.missing_data_num).any(), which looks at every column (factor, date, anything extra), but the report only prints the five OHLCV counts. So a factor-only gap produces a row of all zeros under "missing data" and, since the summary now also keys off check_missing_data_result, a summary entry that points at nothing. Either restrict the .any() to the five reported columns (the factor case is already covered by check_missing_factor), or add a column for the others. A test with one NaN-only extra column would pin it. The check_data fix (check_missing_data_result replacing the duplicated large-step term) looks right to me.

@OKAN-SOFTWARE

Copy link
Copy Markdown
Author

Thanks Arhan Canli (@arhancanli) for catching this and providing a clear reproduction!

Fixed in 4ef761a4. check_missing_data() now checks only the five reported OHLCV columns, while factor validation remains separate.

I also added two regression tests covering missing values in the factor column and unrelated extra columns.

Validation: 9 tests passed, and git diff --check is clean.

Appreciate the careful review!

@arhancanli

Copy link
Copy Markdown

Read the diff of 4ef761a: the per-file test now looks only at open, high, low, close and volume, and missing_counts.get(column, 0) keeps a CSV that lacks one of those columns from raising a KeyError. The all-empty factor case from my repro no longer lands in the OHLCV table and still shows up in check_missing_factor(), which the second new test pins down. One question: do you want a file with a column entirely missing (not just empty) reported by check_missing_factor-style logic for OHLCV too, or is that left to the instrument-level checks?

@OKAN-SOFTWARE

Copy link
Copy Markdown
Author

Arhan Canli (@arhancanli) Thanks for the follow-up! Completely missing OHLCV columns are already handled by check_required_columns(), which reports each missing column per instrument and includes the findings in the final check_data() summary.

I kept that separate from check_missing_data(), which checks null values in existing OHLCV columns, and check_missing_factor(), which handles factor-specific validation.

The regression test test_missing_factor_and_multiple_required_columns_are_reported covers this behavior.

I think keeping those responsibilities separate makes the diagnostics clearer, but I'm happy to adjust if you'd prefer a unified report.

@arhancanli

Copy link
Copy Markdown

Checked the diff: check_required_columns() is untouched by the missing-values path and test_missing_factor_and_multiple_required_columns_are_reported does pin the per-column report, so the separation holds and I would keep it. A unified report is not needed. One thing the three checks do not show together: a file with a required column absent and the remaining columns also containing NaNs gets two separate tables. Is check_data() meant to print them in a fixed order (required columns first), so a reader sees the structural problem before the null counts? A test asserting that order would make the summary stable.

@OKAN-SOFTWARE

Copy link
Copy Markdown
Author

Thanks for the suggestion! I've updated check_data() to report missing required OHLCV columns before missing-value counts.

I also added a regression test covering a CSV with both a missing required column and NaN values, verifying the summary order.

Updated in 1afa8f8c.

Validation:

  • 10/10 regression tests passed
  • git diff --check passed
  • Existing detection logic remains unchanged

Thanks again for the careful review!

@arhancanli

Copy link
Copy Markdown

Checked 1afa8f8: the required-columns block now prints before the missing-data block, and the new test builds the right input (CSV without high, NaN in open). One thing about the assertion: output.index("volume") only proves order if "volume" first appears inside the missing-data table. The CSV has a volume column, so if the required-columns table (or any earlier line) prints it too, index("missing_col") < index("volume") passes even with the old order. Could the test anchor on a string only the NaN table prints (for example the header or the open count row), and does the old order fail it when you revert the swap locally? That red/green check is the quickest way to show the test pins the change.

@OKAN-SOFTWARE

Copy link
Copy Markdown
Author

Thanks for the careful review! I've strengthened the regression test to compare the actual rendered tables rather than relying on a column-name substring.

I also verified the test against both implementations:

  • GREEN: 10 tests passed with the corrected reporting order.
  • RED: The ordering test failed against the previous implementation (4ef761a4), as expected.

This confirms that the regression test detects the original ordering problem. The implementation itself remains unchanged.

Thanks again for the helpful suggestion!

@arhancanli

Copy link
Copy Markdown

Checked 566c039: the test now builds both rendered tables and asserts output.index(required_table) < output.index(missing_data_table), which a column-name substring could not do, and the red run on 4ef761a is the evidence I wanted. No further changes from my side on the ordering. One edge the CSV-only path may not cover yet: a file with duplicated or unsorted date rows. Does check_data() flag those in CSV mode, or only when a Qlib data directory is present?

@OKAN-SOFTWARE

Copy link
Copy Markdown
Author

Thanks for confirming the ordering fix!

I checked the current implementation. check_data() does not explicitly detect duplicated or unsorted date rows in either CSV or Qlib-directory mode. CSV dates are currently used for reporting large-step changes, not for validating date integrity.

That's a worthwhile additional health check. I'd prefer to keep it separate from this PR's current fixes so we don't introduce unrelated behavior without dedicated tests.

A follow-up could validate duplicate and non-monotonic dates in CSV mode, while handling Qlib's MultiIndex dates appropriately.

Thanks for pointing out the gap! :)

@arhancanli

Copy link
Copy Markdown

Fair to keep it out of this PR. For the follow-up, two details are worth putting in its tests: in pandas, index.is_monotonic_increasing returns True for repeated dates, so the unsorted check and the duplicate check (index.duplicated().any()) have to be separate. On the MultiIndex, (instrument, datetime) needs the sorted and duplicate checks per instrument, because a frame sorted overall can still repeat a date inside one instrument.

Is the duplicate case something you have hit in real CSV dumps, and if so, did it come from merged downloads or from a single vendor file? That would tell me which fixture to model.

@OKAN-SOFTWARE

Copy link
Copy Markdown
Author

Thanks for the additional details! Good point about monotonicity not detecting duplicate dates.

I don't have a specific real-world CSV dump I can confidently attribute to merged downloads or a single vendor, so I'd keep the initial regression fixtures source-neutral.

For the follow-up, I'd cover duplicate dates and out-of-order dates separately, with per-instrument validation for Qlib MultiIndex data.

That should give us deterministic coverage without making assumptions about the source of the duplicates. Appreciate the guidance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants