Skip to content

fix: return the last observation from PandasQuote - #2404

Open
朱甲文(Jiawen Zhu) (andone-07) wants to merge 1 commit into
microsoft:mainfrom
andone-07:fix/pandas-quote-last-observation
Open

朱甲文(Jiawen Zhu) (andone-07) wants to merge 1 commit into
microsoft:mainfrom
andone-07:fix/pandas-quote-last-observation

Conversation

@andone-07

Copy link
Copy Markdown

Description

Handle method="last" in PandasQuote.get_data with positional selection after the requested time window has been sliced. Preserve a final NaN, matching NumpyQuote, and document the distinction from ts_data_last.

Motivation and Context

BaseQuote.get_data documents last as a supported aggregation returning the final observation. PandasQuote currently passes that string to resam_ts_data, which invokes Series.last() without its required date offset. A three-day quote containing [1.0, 2.0, 3.0] raises TypeError: NDFrame.last() missing 1 required positional argument: 'offset', while NumpyQuote returns 3.0 for the same timestamp-bounded query.

The fix is limited to the optional Pandas quote backend. It does not change general resampling behavior or the existing default Numpy backend. last retains the final observation even when it is NaN; ts_data_last continues to select the last non-NaN observation.

How Has This Been Tested?

  • Baseline regression: 12 failed, 5 passed in 1.51s.
  • Fixed regression: 17 passed in 1.38s.
  • cd tests && python -m pytest . -m "not slow" --durations=20: 708 passed, 1 skipped, 13 deselected, 495 warnings in 1139.58s (0:18:59).
  • python -m pytest tests/test_all_pipeline.py -q: 3 passed, 42 warnings in 152.75s (0:02:32).
  • pip check, full make pylint, flake8, mypy, documentation build, new-test flake8 and sequential Black checks on 350 Python files pass.

Validation scope: Linux/Python 3.12.14 with pandas 2.3.3. The high-frequency test is skipped by upstream. Of the 13 deselected slow tests, three pipeline tests passed separately and the other ten were not run. The standard Black/nbqa entrypoints are blocked by the environment's multiprocessing socket restriction; source formatting was checked with the same Black API sequentially. Notebook and cross-platform CI checks are not claimed.

The 17 regression cases cover backend parity, sorted/reversed input, ordinary/interior/trailing/all NaN values, bounded/open-ended/single-point/empty windows, and unchanged mean/sum/ts_data_last/no-aggregation behavior. Inputs are deterministic synthetic quote data exercised through the real APIs, not mocks; they demonstrate API behavior, not investment performance.

Types of changes

  • Fix bugs
  • Add new feature
  • Update documentation

Prepared with AI assistance.

Copilot AI balanced review requested due to automatic review settings November 15, 2026 11:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change to last-observation selection.

No open issues 🎉

What changed in this PR

Fixes PandasQuote.get_data(method="last") to return the final observation in the requested time window, matching NumpyQuote without changing general resampling behavior.

Changes:

  • Uses positional selection for "last", preserving a trailing NaN.
  • Documents the distinction between "last" and "ts_data_last".
  • Adds 17 regression cases covering backend parity, input ordering, time windows, NaN values, and unchanged aggregation behavior.
File Description
tests/​backtest/​test_pandas_quote.py Adds regression coverage for last-observation selection and existing behavior.
qlib/​backtest/​high_performance_ds.py Implements positional "last" selection and documents NaN handling.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants