Skip to content

build(python): make numpy an optional dependency - #11084

Open
ZephyrYWZhou wants to merge 2 commits into
lance-format:mainfrom
ZephyrYWZhou:make-numpy-optional
Open

ZephyrYWZhou wants to merge 2 commits into
lance-format:mainfrom
ZephyrYWZhou:make-numpy-optional

Conversation

@ZephyrYWZhou

@ZephyrYWZhou ZephyrYWZhou commented Nov 14, 2026 •

Copy link
Copy Markdown

Closes #1216.

numpy was a required dependency of pylance mainly because pyarrow required it. pyarrow dropped its required numpy dependency in 18.0 and has kept it optional through the current 26.x, so pylance can make numpy optional too.

Changes

  • pyproject.toml: move numpy out of required dependencies into a new numpy optional extra. Bump the pyarrow floor to >=18, the first release that does not pull numpy in.
  • Route the last two direct import numpy statements (indices/builder.py, _dataset/scanner.py) through the existing lazy _LazyModule helper in lance/dependencies.py, matching how pandas/polars/torch are already handled. Most modules were already lazy.
  • Gate the isinstance/issubclass numpy checks in _dataset/scanner.py and pydantic.py behind _check_for_numpy, so they don't trigger the lazy proxy when numpy is absent.
  • Add test_numpy_optional.py: import lance works with numpy absent, and a numpy-only path raises a clear error.

Genuine numpy paths (vector math, sampling, kmeans) still need numpy and will raise a helpful error if it's missing. Install with pip install pylance[numpy] to get it.

One open question

pyproject previously allowed pyarrow>=14, but numpy-free only holds on pyarrow>=18. This PR bumps the floor to 18. Keeping >=14 and instead documenting that numpy-free needs pyarrow 18+ is also an option, if the community prefers that.

Testing

  • New test_numpy_optional.py (blocks numpy import in a subprocess).
  • The full Python suite should run in CI. Feedback on the pyarrow floor would be welcome.

pyarrow dropped its required numpy dependency in 18.0, so pylance no longer needs numpy as a hard dependency.

- Move numpy out of required deps into a new `numpy` optional extra, and raise the pyarrow floor to >=18 (the first release where pyarrow does not pull numpy in).
- Route the two remaining direct `import numpy` statements (indices/builder.py, _dataset/scanner.py) through the existing lazy `_LazyModule` helper, matching pandas/polars/torch.
- Gate the isinstance/issubclass numpy checks in _dataset/scanner.py and pydantic.py behind `_check_for_numpy` so they do not trigger the lazy proxy when numpy is absent.
- Add a test asserting `import lance` works with numpy absent and that numpy-only paths raise a clear error.
@github-actions github-actions Bot added A-python Python bindings A-deps Dependency updates chore labels Nov 14, 2026
@ZephyrYWZhou
ZephyrYWZhou marked this pull request as ready for review November 15, 2026 23:36

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 16, 2026
… test

take() chose the allowed integer types from the first index only, so mixed Python and NumPy indices were rejected depending on order. Check each index on its own. The numpy-absent test only blocked import, so find_spec still saw numpy. Block it with a meta_path finder instead.
@lance-gatekeeper lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Nov 16, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

Both earlier findings are fixed. The lazy dependency approach supports NumPy-free Arrow workflows while keeping NumPy-backed features available through pylance[numpy].

Please mark this PR with the breaking-change label: python/pyproject.toml raises the PyArrow minimum from 14 to 18 and removes the default NumPy dependency, so clients pinned to PyArrow 14–17 must upgrade and callers of NumPy-backed operations must install pylance[numpy].

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Nov 16, 2026
@ZephyrYWZhou

Copy link
Copy Markdown
Author

Hey @wjones127, this one's ready for a look whenever you get a chance. Since it's from a fork, the Python CI needs a maintainer to approve the run.

The gatekeeper also suggested a breaking-change label because of the pyarrow floor bump to 18. Happy to go back to >=14 if you'd rather avoid that.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates A-python Python bindings chore K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make numpy optional pylance dependency

1 participant