Repository navigation
Resolve object-store URLs in the FFI via the cloud registry - #9558
Conversation
|
Should we do this at a lower level (can object store not do this?) |
|
Agreed — that's the better home for it. Reworking to resolve URLs in core instead, in the |
f8f244e to
9ff11a0
Compare
|
Reworked as discussed — resolution now happens in core: |
9ff11a0 to
59e6785
Compare
Merging this PR will degrade performance by 9.47%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_neon[16384] |
12.4 µs | 20.4 µs | -38.91% |
| ❌ | Simulation | decompress[u64, (4000, 1024)] |
71.4 µs | 86.8 µs | -17.69% |
| ❌ | WallTime | words_gather_scalar_avx2[65536] |
8.2 µs | 9.4 µs | -12.04% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.4 µs | 17.7 µs | +21.18% |
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing balicat:ffi-object-store-urls (0d3dcac) with develop (d82de0e)
Footnotes
-
218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
59e6785 to
900b956
Compare
|
CI fixes pushed:
On CodSpeed: the one regressed benchmark ( |
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
900b956 to
84fa9bd
Compare
|
Rebased onto current develop (c93f5a9) and re-ran the checks locally: vortex-file (with Could a maintainer approve the CI run and take a look when convenient? The rework follows @joseph-isaacs's suggestion to resolve URLs in core rather than in the FFI, so the FFI diff is now just the feature flag plus a test. |
|
Hi @balicat. We've taken the route for the integrations to use their own globbing. For example, vortex-duckdb and vortex-datafusion both use query engines' implementation for that and operate on single files only. MultiFileDataSource is used only in FFI, JNI, and one benchmark integration, and we plan to phase it out eventually. |
84fa9bd to
94cf0c5
Compare
|
Thanks, that makes sense. I've moved the resolution out of core. The whole change now lives in vortex-ffi: it parses each incoming path and, for an
|
|
LGTM. Let's shorten the comments, make the tests pass, update the PR description and name, and I'll merge this. |
The C FFI builds a MultiFileDataSource with every glob passed as fs: None, so a bare object-store URL (s3://, gs://, az://, https://, ...) was treated as a local path and failed, even though the Python and Java bindings resolve their own stores and the DuckDB / DataFusion integrations glob through their query engines. Resolve object-store URLs in the FFI itself: each incoming path that parses as one is routed through the shared vortex-cloud Registry (the same resolution as the Python binding's resolve_store), wrapped in Compat and an ObjectStoreFileSystem, and handed to with_glob as an explicit filesystem. Parse failures, file URLs, and single-character schemes (Windows drive paths) stay local globs. Core (vortex-file, vortex, vortex-cloud) is unchanged, so the resolution moves with the FFI if MultiFileDataSource is later retired. A cfg(not(windows)) test asserts an unconfigured az:// URL fails in the registry's store builder rather than as an unmatched local glob. Signed-off-by: David Linton <e.david.linton@gmail.com>
94cf0c5 to
0d3dcac
Compare
|
Done: comments shortened (the "why" moved out), the taplo formatting on |
Summary
The C FFI builds a
MultiFileDataSourcewith every glob passed asfs: None, so abare object-store URL (
s3://,gs://,az://,https://, ...) was treated as alocal path and failed, even though the Python and Java bindings resolve their own
stores and the DuckDB / DataFusion integrations glob through their query engines.
Following review, the resolution lives entirely in
vortex-ffi(an earlier revisionput it in core): the FFI owns its globbing the way the other integrations own theirs,
and it moves with the FFI if
MultiFileDataSourceis later retired.Changes
vortex-ffi, each incoming path that parses as an object-store URL is resolvedthrough the shared
vortex-cloudRegistry(the same resolution as the Pythonbinding's
resolve_store), wrapped inCompatand anObjectStoreFileSystem, andhanded to
with_globas an explicitFileSystemRef.file://URLs, and single-character schemes (Windows drive pathslike
C:/data.vortex) stay ordinary local globs.vortex-ffigainsobject_store+urldependencies and enables the existingvortex/object_store_registryfeature. Core (vortex-file,vortex,vortex-cloud) is unchanged.A
cfg(not(windows))test asserts an unconfiguredaz://URL fails inside theregistry's store builder rather than as an unmatched local glob - a deterministic,
network-free check that the registry path was taken.
Validated end-to-end from a .NET binding over the C FFI against Cloudflare R2 on the
public internet: a single-series predicate scan of an 18 MB / 4.5 M-row file completes
in ~2 s cold via both
https://and signeds3://, byte-identical to the ranges thesame plan issues against a local file; the local-file path is regression-identical to
a stock build.
Claude Code assisted.