Skip to content

Resolve object-store URLs in the FFI via the cloud registry - #9558

Merged
myrrc merged 1 commit into
vortex-data:developfrom
balicat:ffi-object-store-urls
Sep 14, 2026
Merged

myrrc merged 1 commit into
vortex-data:developfrom
balicat:ffi-object-store-urls

Conversation

@balicat

@balicat balicat commented Aug 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The C FFI builds a MultiFileDataSource with every glob passed as fs: None, so a
bare object-store URL (s3://, gs://, az://, https://, ...) was treated as a
local path and failed, even though the Python and Java bindings resolve their own
stores and the DuckDB / DataFusion integrations glob through their query engines.

Following review, the resolution lives entirely in vortex-ffi (an earlier revision
put it in core): the FFI owns its globbing the way the other integrations own theirs,
and it moves with the FFI if MultiFileDataSource is later retired.

Changes

  • In vortex-ffi, each incoming path that parses as an object-store URL is resolved
    through the shared vortex-cloud Registry (the same resolution as the Python
    binding's resolve_store), wrapped in Compat and an ObjectStoreFileSystem, and
    handed to with_glob as an explicit FileSystemRef.
  • Parse failures, file:// URLs, and single-character schemes (Windows drive paths
    like C:/data.vortex) stay ordinary local globs.
  • vortex-ffi gains object_store + url dependencies and enables the existing
    vortex/object_store_registry feature. Core (vortex-file, vortex,
    vortex-cloud) is unchanged.

A cfg(not(windows)) test asserts an unconfigured az:// URL fails inside the
registry's store builder rather than as an unmatched local glob - a deterministic,
network-free check that the registry path was taken.

Validated end-to-end from a .NET binding over the C FFI against Cloudflare R2 on the
public internet: a single-series predicate scan of an 18 MB / 4.5 M-row file completes
in ~2 s cold via both https:// and signed s3://, byte-identical to the ranges the
same plan issues against a local file; the local-file path is regression-identical to
a stock build.


Claude Code assisted.

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Should we do this at a lower level (can object store not do this?)

@balicat

balicat commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Agreed — that's the better home for it. Reworking to resolve URLs in core instead, in the fs: None path of MultiFileDataSource next to where the local filesystem is auto-created, so every consumer gets it and the FFI change all but disappears. Will update this PR.

@balicat
balicat force-pushed the ffi-object-store-urls branch from f8f244e to 9ff11a0 Compare August 24, 2026 16:03
@balicat balicat changed the title FFI: route object-store URLs through env-configured filesystems Resolve object-store URLs in MultiFileDataSource via the cloud registry Aug 24, 2026
@balicat

balicat commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Reworked as discussed — resolution now happens in core: with_glob keeps URLs intact and build() routes (url, None) sources through the existing vortex-cloud::Registry, gated behind a vortex-file/object_store_registry feature threaded through the facade. The FFI diff is down to a feature flag plus a test. Title and description updated to match.

@joseph-isaacs
joseph-isaacs requested a review from myrrc August 24, 2026 17:09
@balicat
balicat force-pushed the ffi-object-store-urls branch from 9ff11a0 to 59e6785 Compare August 24, 2026 21:03
@robert3005 robert3005 added the changelog/fix A bug fix label Aug 25, 2026
@codspeed

codspeed Bot commented Aug 25, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 9.47%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 3 regressed benchmarks
✅ 2192 untouched benchmarks
⏩ 218 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime arrow_checked_add_u32_neon[16384] 12.4 µs 20.4 µs -38.91%
❌ Simulation decompress[u64, (4000, 1024)] 71.4 µs 86.8 µs -17.69%
❌ WallTime words_gather_scalar_avx2[65536] 8.2 µs 9.4 µs -12.04%
⚡ WallTime arrow_checked_add_u32_avx2[16384] 21.4 µs 17.7 µs +21.18%
⚡ Simulation allocate_drop_arrow[0] 456.9 ns 402.7 ns +13.45%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing balicat:ffi-object-store-urls (0d3dcac) with develop (d82de0e)

Open in CodSpeed

Footnotes

  1. 218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@balicat
balicat force-pushed the ffi-object-store-urls branch from 59e6785 to 900b956 Compare August 25, 2026 17:18
@balicat

balicat commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

CI fixes pushed:

  • taplo fmt on the two manifests it flagged.
  • Removed an intra-doc link to a function-locally imported type that broke the docs build.
  • The publish dry-run failure was real: this PR's vortex-file → vortex-cloud dependency closes a cycle with vortex-cloud's dev-dependency on vortex-file, and publish ordering broke. Fixed by making that dev-dependency path-only (no version) so it drops out of the publish graph — same pattern as vortex-alp in the fastlanes encoding.

On CodSpeed: the one regressed benchmark (compress_fsst[(500, 64, 8)], −10.7%) is a code path this PR doesn't touch, and the same benchmark family moved −11.6% on #9614 and +10.9% on #9616 — looks like runner noise. Happy for someone with dashboard access to acknowledge it.

@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Sep 10, 2026
@balicat
balicat force-pushed the ffi-object-store-urls branch from 900b956 to 84fa9bd Compare September 11, 2026 06:49
@balicat

balicat commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current develop (c93f5a9) and re-ran the checks locally: vortex-file (with object_store_registry) and vortex-ffi test suites green, clippy with -D warnings on both crates, cargo fmt --check on the nightly the CI pins, taplo on the manifests, and the lockfile under --locked.

Could a maintainer approve the CI run and take a look when convenient? The rework follows @joseph-isaacs's suggestion to resolve URLs in core rather than in the FFI, so the FFI diff is now just the feature flag plus a test.

@myrrc myrrc self-assigned this Sep 11, 2026
@myrrc myrrc removed the stale This PR is stale and will be auto-closed soon label Sep 11, 2026
@myrrc

myrrc commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Hi @balicat. We've taken the route for the integrations to use their own globbing. For example, vortex-duckdb and vortex-datafusion both use query engines' implementation for that and operate on single files only.

MultiFileDataSource is used only in FFI, JNI, and one benchmark integration, and we plan to phase it out eventually.

@balicat
balicat force-pushed the ffi-object-store-urls branch from 84fa9bd to 94cf0c5 Compare September 11, 2026 10:24
@balicat

balicat commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, that makes sense. I've moved the resolution out of core.

The whole change now lives in vortex-ffi: it parses each incoming path and, for an
object-store URL, resolves it through the shared vortex-cloud registry (the same
resolve the Python binding's resolve_store uses) and hands MultiFileDataSource
an explicit FileSystemRef. Local paths, file://, and Windows drive paths stay
ordinary local globs. So the FFI now owns its globbing the way vortex-duckdb and
vortex-datafusion own theirs, and if MultiFileDataSource is later phased out the
resolution moves with the FFI rather than being stranded in core.

vortex-file, vortex-cloud, and vortex are back to their develop state; the
core diff and the publish-graph feature threading are gone. The change is ~30 lines
plus a test in vortex-ffi.

@myrrc

myrrc commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

LGTM. Let's shorten the comments, make the tests pass, update the PR description and name, and I'll merge this.
Also, please disclose LLM usage, if any, in the PR description.

Comment thread vortex-ffi/src/data_source.rs Resolved
@myrrc
myrrc removed the request for review from robert3005 September 11, 2026 12:29
@myrrc myrrc added the lang/c Relates to the Vortex C (FFI) API label Sep 11, 2026
The C FFI builds a MultiFileDataSource with every glob passed as fs: None, so a
bare object-store URL (s3://, gs://, az://, https://, ...) was treated as a local
path and failed, even though the Python and Java bindings resolve their own stores
and the DuckDB / DataFusion integrations glob through their query engines.

Resolve object-store URLs in the FFI itself: each incoming path that parses as one
is routed through the shared vortex-cloud Registry (the same resolution as the
Python binding's resolve_store), wrapped in Compat and an ObjectStoreFileSystem,
and handed to with_glob as an explicit filesystem. Parse failures, file URLs, and
single-character schemes (Windows drive paths) stay local globs. Core (vortex-file,
vortex, vortex-cloud) is unchanged, so the resolution moves with the FFI if
MultiFileDataSource is later retired.

A cfg(not(windows)) test asserts an unconfigured az:// URL fails in the registry's
store builder rather than as an unmatched local glob.

Signed-off-by: David Linton <e.david.linton@gmail.com>
@balicat
balicat force-pushed the ffi-object-store-urls branch from 94cf0c5 to 0d3dcac Compare September 12, 2026 09:12
@balicat balicat changed the title Resolve object-store URLs in MultiFileDataSource via the cloud registry Resolve object-store URLs in the FFI via the cloud registry Sep 12, 2026
@balicat

balicat commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Done: comments shortened (the "why" moved out), the taplo formatting on vortex-ffi/Cargo.toml fixed (that was the Linters and Tests failure), and the title and description updated. LLM usage disclosed in the description. The CMake / C++ jobs on the previous run failed on an apt-fetch network hiccup, unrelated to the change, and should pass on a fresh run.

@myrrc
myrrc merged commit 56c550c into vortex-data:develop Sep 14, 2026
115 of 119 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix lang/c Relates to the Vortex C (FFI) API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants