Skip to content

feat: Add deephaven-sample-data package with vendored datasets - #1448

Draft
mofojed wants to merge 5 commits into
deephaven:mainfrom
mofojed:agents/split-deephaven-sample-data-package
Draft

mofojed wants to merge 5 commits into
deephaven:mainfrom
mofojed:agents/split-deephaven-sample-data-package

Conversation

@mofojed

@mofojed mofojed commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

Adds a new, independently published deephaven-sample-data package (deephaven.sample_data) with deterministic sample datasets that any plugin can use. Each dataset ticks by default; pass ticking=False for a static table. The package only depends on Deephaven (deephaven-core, jpy, plus pandas/numpy, which come with deephaven-core). It does not depend on plotly.

Changes

  • New package plugins/sample-data (deephaven-sample-data, 0.1.0.dev0) with a README and tests that build every dataset, static and ticking.
    • All plotly-express datasets: iris, jobs, marketing, stocks, tips, election, wind, gapminder, fish_market, outages, flights, plus the OUTAGE_CENTER/FLIGHT_CENTER constants.
    • All TradingView Lightweight datasets: ohlc, stock_trades (formerly tvl.data.stocks), volume, yields, options_chain, values, large_prices.
  • Vendored open data: the source data for iris, tips, election, wind and gapminder used to come from plotly.express.data. It is now vendored as CSVs in deephaven/sample_data/_data/ (~115 KB), exported unmodified from plotly 6.9.0. The generators add ticking rows on top, as before. NOTICE records each dataset's source and license and includes Plotly's MIT license; it ships in the wheel and sdist.
  • plotly-express depends on deephaven-sample-data. dx.data.*, from deephaven.plot.express.data import ... and dx.data.data_generators.* still work for the original 13 names but forward to deephaven.sample_data with a DeprecationWarning. Newer datasets (e.g. ohlc) are only available from deephaven.sample_data. Importing deephaven.plot.express itself does not warn.
  • TradingView Lightweight depends on deephaven-sample-data. deephaven.plot.tradingview_lightweight.data forwards to it with a DeprecationWarning, mapping stocks to sample_data.stock_trades. The DEEPHAVEN_PLUGINS_STATIC_DATA=1 override still works.
  • Release wiring: cog.toml, tools/update_version.sh, sample-data-v* tag trigger, and CI change detection (sample-data changes also re-run the plotly-express and TradingView tests).
  • Local resolution of the unreleased package: tox, the docs workflow, the Dockerfile and plugin_builder.py install the in-repo sample-data. plugin_builder.py builds and installs it automatically when building plotly-express or TradingView Lightweight.
  • Docs: the plotly-express and TradingView example-data.md pages use deephaven.sample_data and note the deprecation. Other dx.data.* usages in docs are left for a follow-up.
  • Bug fix: flights(ticking=False) previously failed because FlightIdx/Timestamp were only added in the ticking branch.

Licensing

Dataset Source License
iris UCI Machine Learning Repository CC BY 4.0
election Ville de Montréal CC BY 4.0
gapminder Gapminder.org CC BY 4.0
wind Plotly MIT
tips Bryant & Smith (1995) No explicit license; widely redistributed (seaborn, R, plotly)

Reviewers: please confirm that redistributing tips is acceptable.

Release notes

  • sample-data must be released (tag sample-data-v0.1.0) before the next plotly-express or TradingView Lightweight release, since their published versions will pull it from PyPI.
  • After publishing, add deephaven-sample-data to pyright's additional_dependencies in .pre-commit-config.yaml and drop the reportMissingImports ignores in the plotly-express and TradingView data shims.

Testing

  • tox -e py3.12 passes in plugins/sample-data (8 tests, including vendored row counts and a check that plotly isn't imported), plugins/plotly-express (213 tests, including the deprecation tests) and plugins/tradingview-lightweight (732 tests).
  • Checked that the wheel and sdist include the CSVs and NOTICE.
  • In a fresh venv with only deephaven-server and deephaven-sample-data, every vendored dataset loads with the expected row counts, and plotly is neither installed nor imported.
  • pre-commit (black, pyright, ruff) passes.

Move the plotly-express data generators into a new independently
published deephaven-sample-data package (deephaven.sample_data).
plotly-express now depends on it and deephaven.plot.express.data
forwards to it with a DeprecationWarning.

- Add plugins/sample-data with tests and release wiring (cog, tags,
  update_version.sh, change detection)
- Install in-repo sample-data in tox, docs builds, Docker, and
  plugin_builder so unreleased versions resolve locally
- Fix flights(ticking=False) failing due to missing columns

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 7, 2026 20:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Separate pip invocations cause fresh-environment installation failures for unpublished sample-data, while the compatibility shims break legacy exports and wildcard imports and recommend an invalid migration target.

4 open issues
What changed in this PR

Extracts sample datasets from Plotly Express into the independently published deephaven-sample-data package, making them available through deephaven.sample_data.

Changes:

  • Moves dataset generators into the new package, adds tests, and fixes static flight generation.
  • Adds deprecated forwarding modules and updates example-data documentation.
  • Updates release automation, CI, and local dependency installation for the package split.
File Description
tools/​update_version.sh Supports sample-data version updates and Python-only releases.
tools/​plugin_builder.py Batches matching wheels into one installation command.
plugins/​sample-data/​tox.ini Configures Python test environments.
plugins/​sample-data/​test/​deephaven/​sample_data/​test_sample_data.py Tests static and ticking datasets, determinism, and map centers.
plugins/​sample-data/​test/​deephaven/​sample_data/​__init__.py Enables test discovery.
plugins/​sample-data/​test/​deephaven/​__init__.py Enables test discovery.
plugins/​sample-data/​test/​__init__.py Initializes the JVM for tests.
plugins/​sample-data/​src/​deephaven/​sample_data/​_generators.py Houses migrated generators and fixes static flights.
plugins/​sample-data/​src/​deephaven/​sample_data/​__init__.py Exports datasets and map-center constants.
plugins/​sample-data/​setup.cfg Defines package metadata and dependencies.
plugins/​sample-data/​README.md Documents installation, datasets, and migration.
plugins/​sample-data/​pyproject.toml Configures the Python build backend.
plugins/​sample-data/​LICENSE Adds the Apache 2.0 license.
plugins/​sample-data/​CHANGELOG.md Initializes the package changelog.
plugins/​sample-data/​.gitignore Excludes generated and local files.
plugins/​plotly-express/​tox.ini Installs sample-data from the repository for tests.
plugins/​plotly-express/​test/​deephaven/​plot/​express/​test_data_deprecation.py Tests forwarding and deprecation warnings.
plugins/​plotly-express/​src/​deephaven/​plot/​express/​data/​data_generators.py Replaces generators with lazy compatibility forwarding.
plugins/​plotly-express/​src/​deephaven/​plot/​express/​data/​_deprecation.py Centralizes lazy imports and warnings.
plugins/​plotly-express/​src/​deephaven/​plot/​express/​data/​__init__.py Provides deprecated dataset access.
plugins/​plotly-express/​src/​deephaven/​plot/​express/​__init__.py Imports the compatibility data package.
plugins/​plotly-express/​setup.cfg Adds the sample-data dependency.
plugins/​plotly-express/​docs/​example-data.md Uses the new API and explains deprecation.
Dockerfile Installs local wheels together for dependency resolution.
cog.toml Registers sample-data for releases.
.github/​workflows/​modified-plugin.yml Adds the sample-data release-tag trigger.
.github/​workflows/​build-docs.yml Installs local sample-data before documentation builds.
.github/​actions/​get-changes/​action.yml Detects sample-data changes and triggers dependent checks.

Review effort: Balanced

Comment thread tools/plugin_builder.py Resolved
- Restore the deephaven.plot.express.data_generators alias
- Provide __all__ lazily in deprecated data shims so wildcard imports work
- Point deprecation warnings at replacements that actually expose each name
- Install all selected plugins in one pip call in plugin_builder
- Regenerate example-data docs snapshots

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 9, 2026 14:25
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

plotly-express docs preview (Available for 14 days)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Targeted plotly-express installs omit the local sample-data build and installation, causing reliance on PyPI or a missing dependency with --reinstall.

1 open issue
4 resolved since last review

Review effort: Balanced

Comment thread tools/plugin_builder.py Resolved

@dsmmcken dsmmcken left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trading view has some additional generators too that are slightly different to meet the needs of showing of that library.

…n_builder

Selecting plotly-express now also builds and installs the in-repo
sample-data package, so targeted --install/--reinstall use the local
version instead of PyPI (or leaving it missing with --no-deps).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 9, 2026 15:34
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

plotly-express docs preview (Available for 14 days)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain to block the sample-data package extraction.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

deephaven-sample-data now depends only on Deephaven. The plotly-backed datasets (iris, tips, election, wind, gapminder) stay in plotly-express, which re-exports the Deephaven-only datasets from deephaven.sample_data without deprecation.

Move the TradingView Lightweight example tables into deephaven.sample_data, renaming stocks to stock_trades, and deprecate tvl.data in favor of it. Lower the deephaven-core floor to 41.1 so TradingView Lightweight can depend on it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 18:37
Move all plotly-express datasets into deephaven-sample-data. The iris, tips, election, wind and gapminder source data is vendored as CSVs with attribution in NOTICE, so sample-data no longer depends on plotly. plotly-express depends on sample-data, and dx.data forwards its original names to it with a DeprecationWarning.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mofojed mofojed changed the title feat: Split sample data into deephaven-sample-data package feat: Add deephaven-sample-data package with vendored datasets Nov 11, 2026
@github-actions

Copy link
Copy Markdown

plotly-express docs preview (Available for 14 days)

@github-actions

Copy link
Copy Markdown

tradingview-lightweight docs preview (Available for 14 days)

@github-actions

Copy link
Copy Markdown

plotly-express docs preview (Available for 14 days)

@github-actions

Copy link
Copy Markdown

tradingview-lightweight docs preview (Available for 14 days)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Eager imports break deephaven.plot.express when sample-data is absent, and both legacy data namespaces omit the promised deprecation warnings.

1 open issue

Review effort: Balanced


# Datasets that only need Deephaven live in the deephaven-sample-data package
# Unresolvable by pre-commit's pyright until deephaven-sample-data is published to PyPI
from deephaven.sample_data import ( # pyright: ignore[reportMissingImports]
Copilot AI balanced review requested due to automatic review settings November 11, 2026 18:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Redistribution rights for tips.csv remain unconfirmed because NOTICE states there is no explicit data license, requiring human/legal review before publishing the vendored dataset.

1 open issue

Review effort: Balanced

@mofojed
mofojed marked this pull request as draft November 12, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants