Skip to content

Session-scoped object-store registration for FFI consumers #11509

Open

Description

@balicat

#9558 made the C FFI resolve object-store URLs: vx_data_source_new routes each URL glob through a vortex-cloud Registry (registry_filesystem in vortex-ffi/src/data_source.rs, a function-local static REGISTRY: LazyLock<Registry>), which builds stores from the process environment. That covers the common case. For FFI consumers, though, the environment is often the wrong channel. The library is embedded in a host process the embedder doesn't own; in my case it's a .NET Excel add-in, so the process is EXCEL.EXE. Credentials set with setenv are process-global, visible to every other plugin in the process, and racy to change once threads are running. And because the shared registry caches each store it builds, by URL prefix, credentials changed after the first read of a prefix are never picked up until the process restarts.

The registry already has what's needed. Registry::with_vars builds stores from an explicit set of config variables instead of the environment, and ObjectStoreRegistry::register installs a store at a URL prefix that resolve consults before building one, so one bucket can serve prefixes with different credentials. What's missing is a way to reach them from the FFI.

The design question first: should registration be session-scoped, or is registering into the shared process-wide registry acceptable?

Since #9558 kept resolution in vortex-ffi, and registry_filesystem already receives the session, the session-scoped shape looks FFI-only: a session var holding a Registry (via VortexSession::register / with_some), consulted before the shared static, which stays the fallback. Two sessions in one process could then hold different credentials and drop them with the session. Core (vortex-cloud, vortex-file) would not change.

FFI surface, sketched for the session-scoped shape:

/* Build a store for `url` from explicit key/value config (the
   `parse_url_opts` / `with_vars` vocabulary) and register it at that URL
   prefix in the session's registry. */
void vx_session_register_store(vx_session *session,
                               vx_view url,
                               const vx_view *config_keys,
                               const vx_view *config_values,
                               size_t config_len,
                               vx_error **error_out);

Concrete use case, already working end-to-end over the FFI against Cloudflare R2 from a .NET binding: a spreadsheet add-in reading s3://bucket/… datasets, where credentials come from the add-in's own key manager rather than the process environment, and where a public prefix of the same bucket should read anonymously.

Happy to implement whichever shape you prefer.

Claude Code assisted.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions