Skip to content

An asynchronous version of CatalogList/CatalogProvider/SchemaProvider #3777

Closed

Description

@v0y4g3r

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I'm trying to implement a CatalogList/CatalogProvider/SchemaProvider that based on some remote storage service like ZooKeeper/Etcd.

Describe the solution you'd like
But current CatalogList/CatalogProvider/SchemaProvider trait is in a sync manner.

https://github.com/apache/arrow-datafusion/blob/e54110fb592e03704da5f6ebd832b8fe1c51123b/datafusion/core/src/catalog/catalog.rs#L29-L47

That makes asynchronously accessing remote storage service impossible while implementing CatalogList/CatalogProvider/SchemaProvider.

Describe alternatives you've considered

Maybe change these traits to async or provide an async version?

Additional context
I see projects like influxdb-iox did some hack to get around this problem.

Activity

  1. JanKaul commented on Nov 9, 2022

    @JanKaul
    Contributor

    I would also be interested in this in this feature. I tried using futures::executor::block_on but this leads to my execution hanging as described in tokio-rs/tokio#2376.

  2. alamb commented on Nov 9, 2022

    @alamb
    Contributor

    Hi @JanKaul and @v0y4g3r -- I wonder if the (very) newly added TableProviderFactories may be helpful / fulfill this usecase #4126

  3. v0y4g3r commented on Nov 10, 2022

    @v0y4g3r
    Author

    I would also be interested in this in this feature. I tried using futures::executor::block_on but this leads to my execution hanging as described in tokio-rs/tokio#2376.

    I found a workaround. In this case you can spawn a thread dedicating to block:

    use std::collections::HashMap;
    use std::sync::Arc;
    use tokio::sync::RwLock;
    
    trait CatalogList {
        fn catalog(&self, name: &str) -> Option<Arc<CatalogProvider>>;
    }
    
    struct CatalogProvider;
    
    #[derive(Default)]
    struct CatalogListImpl {
        map: Arc<RwLock<HashMap<String, Arc<CatalogProvider>>>>,
    }
    
    impl CatalogList for CatalogListImpl {
        fn catalog(&self, name: &str) -> Option<Arc<CatalogProvider>> {
            let map = self.map.clone();
            let name = name.to_string();
            std::thread::spawn(move || {
                futures::executor::block_on(async {
                    map.read().await.get(&name).cloned()
                })
            }).join().unwrap()
        }
    }
    
    #[tokio::main]
    async fn main() {
        let catalog_list = CatalogListImpl::default();
        assert!(catalog_list.catalog("hello").is_none());
    }
  4. JanKaul commented on Nov 10, 2022

    @JanKaul
    Contributor

    Hi @JanKaul and @v0y4g3r -- I wonder if the (very) newly added TableProviderFactories may be helpful / fulfill this usecase #4126

    Hi, thanks for the quick reply. I'm trying to implement an iceberg catalog for datafusion. It functions much like a common catalog with schemas(namespaces) and table identifiers. From what I understood about the TableProviderFactory, it wouldn't really fit the use case. One could use the url field to pass in a table identifer like "my_catalog.my_schema.my_table" and get the TableProvider from that. But I don't think this is what it is intended for.

    I found a workaround. In this case you can spawn a thread dedicating to block:

    use std::collections::HashMap;
    use std::sync::Arc;
    use tokio::sync::RwLock;
    
    trait CatalogList {
        fn catalog(&self, name: &str) -> Option<Arc<CatalogProvider>>;
    }
    
    struct CatalogProvider;
    
    #[derive(Default)]
    struct CatalogListImpl {
        map: Arc<RwLock<HashMap<String, Arc<CatalogProvider>>>>,
    }
    
    impl CatalogList for CatalogListImpl {
        fn catalog(&self, name: &str) -> Option<Arc<CatalogProvider>> {
            let map = self.map.clone();
            let name = name.to_string();
            std::thread::spawn(move || {
                futures::executor::block_on(async {
                    map.read().await.get(&name).cloned()
                })
            }).join().unwrap()
        }
    }
    
    #[tokio::main]
    async fn main() {
        let catalog_list = CatalogListImpl::default();
        assert!(catalog_list.catalog("hello").is_none());
    }

    Awesome, thanks for the heads up. It is probably not the cleanest thing to span another thread while the rest of the application is executed with tokio, but it should work for the time being.

  5. JanKaul commented on Nov 10, 2022

    @JanKaul
    Contributor

    I had a look at the code and async versions of CatalogList/CatalogProvider/SchemaProvider would have a huge impact on the code base and also the external API. A lot would need to be async. I understand that that is not desirable.

    The solution with std::thread doesn't work for me because the async function that I'm calling uses hyper which expects an async Context from tokio. I could probably create a new Tokio runtime and spawn a thread from there. But it seems not right to create a second tokio runtime.

    Investigating the futures::executor::block_on hang that I was experiencing, it looks like the problem comes from calling a synchronous function using block_on from inside an async tokio context. If block on is called from other synchronous code it works. The solution to the problem would be to wrap the call to the blocking synchronous function from an async context with a spawn_blocking.

    I'm not sure if that is a route to go, but I could create a branch with the changes that would need to be included in datafusion.

  6. alamb commented on Nov 10, 2022

    @alamb
    Contributor

    I am not 100% familiar but I know there has been a bunch of work recently to support delta-lake style catalogs in DataFusion using ListingTableProviders and ListingSchemaProviders and TableProviderFactories. So my pattern matching mind says "can iceberg use the same approach"

    Maybe someone like @avantgardnerio @timvw @milenkovicm who worked on these features more directly has some other suggestions

    Some related PRs:
    #4125
    #4112
    #4095

    Here is
    https://github.com/delta-io/delta-rs/blob/main/rust/src/delta_datafusion.rs

    The stuff in IOx predates any of these new interfaces, so I would likely not recommend it going forward

    cc @crepererum

  7. crepererum commented on Nov 10, 2022

    @crepererum
    Contributor

    IMHO it makes sense to make the catalog interfaces async.

    FWIW IOx gets around this because we "discover" the whole catalog/tables/schemas before we build the DataFusion data structures. This mostly works because our APIs are scoped in a way that allows that. However midterm I think even IOx could be improved by async schema exploration, because this would lower our the amount of data we need to unconditionally prefetech/prepare (e.g. we would only need to construct schemas for tables that are actually touched by a query).

  8. milenkovicm commented on Nov 10, 2022

    @milenkovicm
    Contributor

    IMHO it makes sense to make the catalog interfaces async.

    I tend to agree with @crepererum, most of the factory methods are async so this would not be a strange change of signature, it would be backward incompatible, but I guess thats not usually show stopper.

    change may not be a small one, as it will end up changing context.rs methods, ContextProvider ...

  9. timvw commented on Nov 10, 2022

    @timvw
    Contributor

    In https://github.com/datafusion-contrib/datafusion-catalogprovider-glue we also first "discover/fetch" the tables/schemas and as of then it's "static/cached".. Async would enable a whole set of use-cases (eg: refreshing catalog(parts)).

  10. v0y4g3r commented on Nov 10, 2022

    @v0y4g3r
    Author

    Did an attempt to refactor CatalogList/CatalogProvider/... stuff to async trait, turned out to be not that easy, lots of method defintions need to change.

  11. avantgardnerio commented on Nov 10, 2022

    @avantgardnerio
    Contributor

    Maybe someone like @avantgardnerio

    I'm not opposed, as long as we are mindful not to break Ballista (which I don't think we will).

    FWIW, when we needed this for Ballista, we just added the async refresh() method and called it at convenient times:

  12. crepererum commented on Nov 10, 2022

    @crepererum
    Contributor

    The issue with refresh() though is that it only enables a small subset of features, namely async full discovery. It doesn't allow for partial schema discovery. For example, if you have a hive dataset and you know the tables by scanning one directory, but there might be thousands of them (I'm exaggerating here) and you wanna spare yourself reading all the _common_metadata files, because the user may only query a single table. We in IOx have a similar situation where it would be nice if we could get away with only constructing the schema for tables we actually need (it's not a major blocker, but I nice to have).

  13. avantgardnerio commented on Nov 10, 2022

    @avantgardnerio
    Contributor

    The issue with refresh()

    Makes sense. Also refresh() is just "kicking the can down the road". Someone has to call it, the caller needs to be async anyway, so creating a proper async interface would clean things up.

  14. JanKaul commented on Nov 10, 2022

    @JanKaul
    Contributor

    If there is a consensus that an async interface is preferable, I can try to make the necessary changes.

  15. 14 remaining items

  16. JanKaul commented on Nov 16, 2022

    @JanKaul
    Contributor

    Does it make sense to make the interface async at both the CatalogProvider and SchemaProvider level? I haven't really looked through the concrete use cases but it seems like we could make SchemaProvider async but still have CatalogProvider be sync. If a particular implementation needed to prefetch metadata (meta-metadata?) then that could be something handled at construction for that implementation.

    That's also a good point. I guess this would work for most use cases. On the other hand I think the biggest burden is to change the public interface of datafusion and update the dependencies. If the dependiencies have to be updated due to changes in SchemaProvider, I think also updating the interface for CatalogProvider is just a minor issue. And one avoids a potential breaking change in the future, in case there are use cases for an async CatalogProvider. Overall, I think having an async interface is a cleaner solution.

  17. JanKaul commented on Nov 16, 2022

    @JanKaul
    Contributor

    It looks like no one was opposed to the proposal. I guess I can start the implementation.

  18. JanKaul commented on Nov 17, 2022

    @JanKaul
    Contributor

    Does it make sense to make the interface async at both the CatalogProvider and SchemaProvider level? I haven't really looked through the concrete use cases but it seems like we could make SchemaProvider async but still have CatalogProvider be sync. If a particular implementation needed to prefetch metadata (meta-metadata?) then that could be something handled at construction for that implementation.

    After a more detailed look at the source code I would like to come back to your suggestion @thinkharderdev. If we use async traits for CatalogProvider and CatalogListProvider the new constructors of SessionState and SessionContext will be async. Maybe that was obvious, but I didn't think of it.

    I would like to get your opinion on which of the following two solutions you prefer:

    1. SchemaProvider, CatalogProvider and CatalogListProvider async
      => SessionContext::new() and SessionState::new() are async

    2. only SchemaProvider async
      => less flexibility with CatalogProvider and CatalogListProvider

    React with:
    🎉 for solution 1 and
    🚀 for solution 2

  19. tustvold commented on Nov 28, 2022

    @tustvold
    Contributor

    where the information cannot simply be stored in memory.

    Looking at the interface of SchemaProvider the only interface it needs is to provide access to TableProvider by name, it doesn't actually need any more information than this.

    The constraint then becomes, what information is needed to construct a TableProvider, which boils down to what information TableProvider needs to be able to provide. Currently this is just the schema, there is support for statistics but I'm not sure this is exploited anywhere.

    My question is therefore, are there use-cases where the number of tables exceeds what can be stored in memory? If not I don't see a compelling reason to make SchemaProvider async, we can potentially make TableProvider methods async to allow deferred loading of metadata, but SchemaProvider itself I think can remain sync?

    A similar argument could be made for CatalogProvider, where you only need to know the schema names up front.

    Edit: Apologies for not commenting sooner I've had too many plates lately to keep track of all these things, but seeing the complexity explosion this results in, with self-recursive async functions etc..., I'm a little bit apprehensive about this...

    Edit Edit: Making TableProvider::schema async runs into the same issue of ending up making the planner async. I'll keep playing around

  20. JanKaul commented on Nov 29, 2022

    @JanKaul
    Contributor

    I think the biggest issue with the making SchemaProvider async is that it makes query planning async. The async gets introduced to the query planner in the ContextProvider implementation of SessionState:

    impl ContextProvider for SessionState {
        fn get_table_provider(&self, name: TableReference) -> Result<Arc<dyn TableSource>> {
            let resolved_ref = self.resolve_table_ref(name);
            match self.schema_for_ref(resolved_ref) {
                Ok(schema) => {
                    let provider = schema.table(resolved_ref.table).ok_or_else(|| { // <= async would be introduced here
                        DataFusionError::Plan(format!(
                            "table '{}.{}.{}' not found",
                            resolved_ref.catalog, resolved_ref.schema, resolved_ref.table
                        ))
                    })?;
                    Ok(provider_as_source(provider))
                }
                Err(e) => Err(e),
            }
        }
        ...
    }

    And I don't see a clean solution to make ContextProvider sync while SchemaProvider being async. If you are pre-fetching the tables you could do it already in the SchemaProvider.

    One approach could be to have an in memory mirror of the SchemaProvider that stores only the essential data. And everytime its state is changed it schedules an async task to update the actual storage without awaiting the response. This however could lead to data races if multiple users access the storage simultaneously.

    Actually, after rethinking my earlier comment, I think it could make sense to make SchemaProvider partially async and ContextProvider sync. One could realize the SchemaProvider as an in memory mirror with a "synchronisation" operation with the underlying storage service. If we look at the SchemaProvider trait it has methods for reading and writing its state. One could make the reading methods sync, by reading from the in memory mirror and make the writing methods async, making sure the changes are propagated to the underlying storage service. Since the reading methods are sync, the ContextProvider trait could also be sync, simplifying the query planning step.

    One problem with this is that external changes to the underlying storage system are not automatically reflected in the in memory mirror, thus one would need to make sure to trigger a "synchronisation" operation before using the trait.

    Additionally, one could provide sync and async versions for the reading methods, allowing operations that want to be sure the mirror is up to date to use the async versions.

  21. tustvold commented on Dec 12, 2022

    @tustvold
    Contributor

    Sorry for the delay, I plan to spend some time on this this week and try to come up with a proposal.

    I think the key thing is to make it easy and obvious how to hook in a remote catalog, without requiring all of planning to become async. I'll see what I can come up with

  22. tustvold commented on Jan 3, 2023

    @tustvold
    Contributor

    #4607 is now ready for review, PTAL. Hopefully we can get this in to make the release at the end of this week 😄

  23. added a commit that references this issue on Jan 5, 2023
  24. JanKaul commented on Jan 6, 2023

    @JanKaul
    Contributor

    Thank you for your efforts!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions