Repository navigation
build(python): make numpy an optional dependency - #11084
ZephyrYWZhou wants to merge 2 commits into
Conversation
pyarrow dropped its required numpy dependency in 18.0, so pylance no longer needs numpy as a hard dependency. - Move numpy out of required deps into a new `numpy` optional extra, and raise the pyarrow floor to >=18 (the first release where pyarrow does not pull numpy in). - Route the two remaining direct `import numpy` statements (indices/builder.py, _dataset/scanner.py) through the existing lazy `_LazyModule` helper, matching pandas/polars/torch. - Gate the isinstance/issubclass numpy checks in _dataset/scanner.py and pydantic.py behind `_check_for_numpy` so they do not trigger the lazy proxy when numpy is absent. - Add a test asserting `import lance` works with numpy absent and that numpy-only paths raise a clear error.
… test take() chose the allowed integer types from the first index only, so mixed Python and NumPy indices were rejected depending on order. Check each index on its own. The numpy-absent test only blocked import, so find_spec still saw numpy. Block it with a meta_path finder instead.
There was a problem hiding this comment.
✅ Gate recommendation: approve.
Both earlier findings are fixed. The lazy dependency approach supports NumPy-free Arrow workflows while keeping NumPy-backed features available through pylance[numpy].
Please mark this PR with the breaking-change label: python/pyproject.toml raises the PyArrow minimum from 14 to 18 and removes the default NumPy dependency, so clients pinned to PyArrow 14–17 must upgrade and callers of NumPy-backed operations must install pylance[numpy].
|
Hey @wjones127, this one's ready for a look whenever you get a chance. Since it's from a fork, the Python CI needs a maintainer to approve the run. The gatekeeper also suggested a |
Closes #1216.
numpy was a required dependency of pylance mainly because pyarrow required it. pyarrow dropped its required numpy dependency in 18.0 and has kept it optional through the current 26.x, so pylance can make numpy optional too.
Changes
pyproject.toml: movenumpyout of required dependencies into a newnumpyoptional extra. Bump the pyarrow floor to>=18, the first release that does not pull numpy in.import numpystatements (indices/builder.py,_dataset/scanner.py) through the existing lazy_LazyModulehelper inlance/dependencies.py, matching how pandas/polars/torch are already handled. Most modules were already lazy.isinstance/issubclassnumpy checks in_dataset/scanner.pyandpydantic.pybehind_check_for_numpy, so they don't trigger the lazy proxy when numpy is absent.test_numpy_optional.py:import lanceworks with numpy absent, and a numpy-only path raises a clear error.Genuine numpy paths (vector math, sampling, kmeans) still need numpy and will raise a helpful error if it's missing. Install with
pip install pylance[numpy]to get it.One open question
pyproject previously allowed
pyarrow>=14, but numpy-free only holds onpyarrow>=18. This PR bumps the floor to 18. Keeping>=14and instead documenting that numpy-free needs pyarrow 18+ is also an option, if the community prefers that.Testing
test_numpy_optional.py(blocks numpy import in a subprocess).