You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
An asynchronous version of CatalogList/CatalogProvider/SchemaProvider #3777
I would also be interested in this in this feature. I tried using futures::executor::block_on but this leads to my execution hanging as described in tokio-rs/tokio#2376.
I would also be interested in this in this feature. I tried using futures::executor::block_on but this leads to my execution hanging as described in tokio-rs/tokio#2376.
I found a workaround. In this case you can spawn a thread dedicating to block:
Hi @JanKaul and @v0y4g3r -- I wonder if the (very) newly added TableProviderFactories may be helpful / fulfill this usecase #4126
Hi, thanks for the quick reply. I'm trying to implement an iceberg catalog for datafusion. It functions much like a common catalog with schemas(namespaces) and table identifiers. From what I understood about the TableProviderFactory, it wouldn't really fit the use case. One could use the url field to pass in a table identifer like "my_catalog.my_schema.my_table" and get the TableProvider from that. But I don't think this is what it is intended for.
I found a workaround. In this case you can spawn a thread dedicating to block:
Awesome, thanks for the heads up. It is probably not the cleanest thing to span another thread while the rest of the application is executed with tokio, but it should work for the time being.
I had a look at the code and async versions of CatalogList/CatalogProvider/SchemaProvider would have a huge impact on the code base and also the external API. A lot would need to be async. I understand that that is not desirable.
The solution with std::thread doesn't work for me because the async function that I'm calling uses hyper which expects an async Context from tokio. I could probably create a new Tokio runtime and spawn a thread from there. But it seems not right to create a second tokio runtime.
Investigating the futures::executor::block_on hang that I was experiencing, it looks like the problem comes from calling a synchronous function using block_on from inside an async tokio context. If block on is called from other synchronous code it works. The solution to the problem would be to wrap the call to the blocking synchronous function from an async context with a spawn_blocking.
I'm not sure if that is a route to go, but I could create a branch with the changes that would need to be included in datafusion.
I am not 100% familiar but I know there has been a bunch of work recently to support delta-lake style catalogs in DataFusion using ListingTableProviders and ListingSchemaProviders and TableProviderFactories. So my pattern matching mind says "can iceberg use the same approach"
IMHO it makes sense to make the catalog interfaces async.
FWIW IOx gets around this because we "discover" the whole catalog/tables/schemas before we build the DataFusion data structures. This mostly works because our APIs are scoped in a way that allows that. However midterm I think even IOx could be improved by async schema exploration, because this would lower our the amount of data we need to unconditionally prefetech/prepare (e.g. we would only need to construct schemas for tables that are actually touched by a query).
IMHO it makes sense to make the catalog interfaces async.
I tend to agree with @crepererum, most of the factory methods are async so this would not be a strange change of signature, it would be backward incompatible, but I guess thats not usually show stopper.
change may not be a small one, as it will end up changing context.rs methods, ContextProvider ...
Did an attempt to refactor CatalogList/CatalogProvider/... stuff to async trait, turned out to be not that easy, lots of method defintions need to change.
The issue with refresh() though is that it only enables a small subset of features, namely async full discovery. It doesn't allow for partial schema discovery. For example, if you have a hive dataset and you know the tables by scanning one directory, but there might be thousands of them (I'm exaggerating here) and you wanna spare yourself reading all the _common_metadata files, because the user may only query a single table. We in IOx have a similar situation where it would be nice if we could get away with only constructing the schema for tables we actually need (it's not a major blocker, but I nice to have).
Makes sense. Also refresh() is just "kicking the can down the road". Someone has to call it, the caller needs to be async anyway, so creating a proper async interface would clean things up.
Does it make sense to make the interface async at both the CatalogProvider and SchemaProvider level? I haven't really looked through the concrete use cases but it seems like we could make SchemaProvider async but still have CatalogProvider be sync. If a particular implementation needed to prefetch metadata (meta-metadata?) then that could be something handled at construction for that implementation.
That's also a good point. I guess this would work for most use cases. On the other hand I think the biggest burden is to change the public interface of datafusion and update the dependencies. If the dependiencies have to be updated due to changes in SchemaProvider, I think also updating the interface for CatalogProvider is just a minor issue. And one avoids a potential breaking change in the future, in case there are use cases for an async CatalogProvider. Overall, I think having an async interface is a cleaner solution.
Does it make sense to make the interface async at both the CatalogProvider and SchemaProvider level? I haven't really looked through the concrete use cases but it seems like we could make SchemaProvider async but still have CatalogProvider be sync. If a particular implementation needed to prefetch metadata (meta-metadata?) then that could be something handled at construction for that implementation.
After a more detailed look at the source code I would like to come back to your suggestion @thinkharderdev. If we use async traits for CatalogProvider and CatalogListProvider the new constructors of SessionState and SessionContext will be async. Maybe that was obvious, but I didn't think of it.
I would like to get your opinion on which of the following two solutions you prefer:
SchemaProvider, CatalogProvider and CatalogListProvider async
=> SessionContext::new() and SessionState::new() are async
only SchemaProvider async
=> less flexibility with CatalogProvider and CatalogListProvider
where the information cannot simply be stored in memory.
Looking at the interface of SchemaProvider the only interface it needs is to provide access to TableProvider by name, it doesn't actually need any more information than this.
The constraint then becomes, what information is needed to construct a TableProvider, which boils down to what information TableProvider needs to be able to provide. Currently this is just the schema, there is support for statistics but I'm not sure this is exploited anywhere.
My question is therefore, are there use-cases where the number of tables exceeds what can be stored in memory? If not I don't see a compelling reason to make SchemaProvider async, we can potentially make TableProvider methods async to allow deferred loading of metadata, but SchemaProvider itself I think can remain sync?
A similar argument could be made for CatalogProvider, where you only need to know the schema names up front.
Edit: Apologies for not commenting sooner I've had too many plates lately to keep track of all these things, but seeing the complexity explosion this results in, with self-recursive async functions etc..., I'm a little bit apprehensive about this...
Edit Edit: Making TableProvider::schema async runs into the same issue of ending up making the planner async. I'll keep playing around
I think the biggest issue with the making SchemaProvider async is that it makes query planning async. The async gets introduced to the query planner in the ContextProvider implementation of SessionState:
implContextProviderforSessionState{fnget_table_provider(&self,name:TableReference) -> Result<Arc<dynTableSource>>{let resolved_ref = self.resolve_table_ref(name);matchself.schema_for_ref(resolved_ref){Ok(schema) => {let provider = schema.table(resolved_ref.table).ok_or_else(|| {// <= async would be introduced hereDataFusionError::Plan(format!("table '{}.{}.{}' not found",
resolved_ref.catalog, resolved_ref.schema, resolved_ref.table
))})?;Ok(provider_as_source(provider))}Err(e) => Err(e),}}
...
}
And I don't see a clean solution to make ContextProvider sync while SchemaProvider being async. If you are pre-fetching the tables you could do it already in the SchemaProvider.
One approach could be to have an in memory mirror of the SchemaProvider that stores only the essential data. And everytime its state is changed it schedules an async task to update the actual storage without awaiting the response. This however could lead to data races if multiple users access the storage simultaneously.
Actually, after rethinking my earlier comment, I think it could make sense to make SchemaProvider partially async and ContextProvider sync. One could realize the SchemaProvider as an in memory mirror with a "synchronisation" operation with the underlying storage service. If we look at the SchemaProvider trait it has methods for reading and writing its state. One could make the reading methods sync, by reading from the in memory mirror and make the writing methods async, making sure the changes are propagated to the underlying storage service. Since the reading methods are sync, the ContextProvider trait could also be sync, simplifying the query planning step.
One problem with this is that external changes to the underlying storage system are not automatically reflected in the in memory mirror, thus one would need to make sure to trigger a "synchronisation" operation before using the trait.
Additionally, one could provide sync and async versions for the reading methods, allowing operations that want to be sure the mirror is up to date to use the async versions.
Sorry for the delay, I plan to spend some time on this this week and try to come up with a proposal.
I think the key thing is to make it easy and obvious how to hook in a remote catalog, without requiring all of planning to become async. I'll see what I can come up with
Activity
JanKaul commentedon Nov 9, 2022
I would also be interested in this in this feature. I tried using
futures::executor::block_onbut this leads to my execution hanging as described in tokio-rs/tokio#2376.alamb commentedon Nov 9, 2022
Hi @JanKaul and @v0y4g3r -- I wonder if the (very) newly added
TableProviderFactoriesmay be helpful / fulfill this usecase #4126v0y4g3r commentedon Nov 10, 2022
I found a workaround. In this case you can spawn a thread dedicating to block:
JanKaul commentedon Nov 10, 2022
Hi, thanks for the quick reply. I'm trying to implement an iceberg catalog for datafusion. It functions much like a common catalog with schemas(namespaces) and table identifiers. From what I understood about the
TableProviderFactory, it wouldn't really fit the use case. One could use theurlfield to pass in a table identifer like "my_catalog.my_schema.my_table" and get theTableProviderfrom that. But I don't think this is what it is intended for.Awesome, thanks for the heads up. It is probably not the cleanest thing to span another thread while the rest of the application is executed with tokio, but it should work for the time being.
JanKaul commentedon Nov 10, 2022
I had a look at the code and async versions of
CatalogList/CatalogProvider/SchemaProviderwould have a huge impact on the code base and also the external API. A lot would need to be async. I understand that that is not desirable.The solution with
std::threaddoesn't work for me because the async function that I'm calling useshyperwhich expects an async Context from tokio. I could probably create a new Tokio runtime and spawn a thread from there. But it seems not right to create a second tokio runtime.Investigating the
futures::executor::block_onhang that I was experiencing, it looks like the problem comes from calling a synchronous function usingblock_onfrom inside an async tokio context. Ifblock onis called from other synchronous code it works. The solution to the problem would be to wrap the call to the blocking synchronous function from an async context with aspawn_blocking.I'm not sure if that is a route to go, but I could create a branch with the changes that would need to be included in datafusion.
alamb commentedon Nov 10, 2022
I am not 100% familiar but I know there has been a bunch of work recently to support delta-lake style catalogs in DataFusion using
ListingTableProvidersandListingSchemaProvidersandTableProviderFactories. So my pattern matching mind says "can iceberg use the same approach"Maybe someone like @avantgardnerio @timvw @milenkovicm who worked on these features more directly has some other suggestions
Some related PRs:
#4125
#4112
#4095
Here is
https://github.com/delta-io/delta-rs/blob/main/rust/src/delta_datafusion.rs
The stuff in IOx predates any of these new interfaces, so I would likely not recommend it going forward
cc @crepererum
crepererum commentedon Nov 10, 2022
IMHO it makes sense to make the catalog interfaces async.
FWIW IOx gets around this because we "discover" the whole catalog/tables/schemas before we build the DataFusion data structures. This mostly works because our APIs are scoped in a way that allows that. However midterm I think even IOx could be improved by async schema exploration, because this would lower our the amount of data we need to unconditionally prefetech/prepare (e.g. we would only need to construct schemas for tables that are actually touched by a query).
milenkovicm commentedon Nov 10, 2022
I tend to agree with @crepererum, most of the factory methods are
asyncso this would not be a strange change of signature, it would be backward incompatible, but I guess thats not usually show stopper.change may not be a small one, as it will end up changing
context.rsmethods,ContextProvider...timvw commentedon Nov 10, 2022
In https://github.com/datafusion-contrib/datafusion-catalogprovider-glue we also first "discover/fetch" the tables/schemas and as of then it's "static/cached".. Async would enable a whole set of use-cases (eg: refreshing catalog(parts)).
v0y4g3r commentedon Nov 10, 2022
Did an attempt to refactor
CatalogList/CatalogProvider/... stuff to async trait, turned out to be not that easy, lots of method defintions need to change.avantgardnerio commentedon Nov 10, 2022
I'm not opposed, as long as we are mindful not to break Ballista (which I don't think we will).
FWIW, when we needed this for Ballista, we just added the
async refresh()method and called it at convenient times:crepererum commentedon Nov 10, 2022
The issue with
refresh()though is that it only enables a small subset of features, namely async full discovery. It doesn't allow for partial schema discovery. For example, if you have a hive dataset and you know the tables by scanning one directory, but there might be thousands of them (I'm exaggerating here) and you wanna spare yourself reading all the_common_metadatafiles, because the user may only query a single table. We in IOx have a similar situation where it would be nice if we could get away with only constructing the schema for tables we actually need (it's not a major blocker, but I nice to have).avantgardnerio commentedon Nov 10, 2022
Makes sense. Also
refresh()is just "kicking the can down the road". Someone has to call it, the caller needs to be async anyway, so creating a proper async interface would clean things up.JanKaul commentedon Nov 10, 2022
If there is a consensus that an async interface is preferable, I can try to make the necessary changes.
14 remaining items
JanKaul commentedon Nov 16, 2022
That's also a good point. I guess this would work for most use cases. On the other hand I think the biggest burden is to change the public interface of datafusion and update the dependencies. If the dependiencies have to be updated due to changes in
SchemaProvider, I think also updating the interface forCatalogProvideris just a minor issue. And one avoids a potential breaking change in the future, in case there are use cases for an asyncCatalogProvider. Overall, I think having an async interface is a cleaner solution.JanKaul commentedon Nov 16, 2022
It looks like no one was opposed to the proposal. I guess I can start the implementation.
JanKaul commentedon Nov 17, 2022
After a more detailed look at the source code I would like to come back to your suggestion @thinkharderdev. If we use async traits for
CatalogProviderandCatalogListProviderthenewconstructors ofSessionStateandSessionContextwill be async. Maybe that was obvious, but I didn't think of it.I would like to get your opinion on which of the following two solutions you prefer:
SchemaProvider,CatalogProviderandCatalogListProviderasync=>
SessionContext::new()andSessionState::new()are asynconly
SchemaProviderasync=> less flexibility with
CatalogProviderandCatalogListProviderReact with:
🎉 for solution 1 and
🚀 for solution 2
tustvold commentedon Nov 28, 2022
Looking at the interface of
SchemaProviderthe only interface it needs is to provide access toTableProviderby name, it doesn't actually need any more information than this.The constraint then becomes, what information is needed to construct a
TableProvider, which boils down to what informationTableProviderneeds to be able to provide. Currently this is just the schema, there is support for statistics but I'm not sure this is exploited anywhere.My question is therefore, are there use-cases where the number of tables exceeds what can be stored in memory? If not I don't see a compelling reason to make
SchemaProviderasync, we can potentially makeTableProvidermethods async to allow deferred loading of metadata, butSchemaProvideritself I think can remain sync?A similar argument could be made for
CatalogProvider, where you only need to know the schema names up front.Edit: Apologies for not commenting sooner I've had too many plates lately to keep track of all these things, but seeing the complexity explosion this results in, with self-recursive async functions etc..., I'm a little bit apprehensive about this...
Edit Edit: Making TableProvider::schema async runs into the same issue of ending up making the planner async. I'll keep playing around
JanKaul commentedon Nov 29, 2022
I think the biggest issue with the making
SchemaProviderasync is that it makes query planning async. The async gets introduced to the query planner in theContextProviderimplementation ofSessionState:And I don't see a clean solution to makeContextProvidersync whileSchemaProviderbeing async. If you are pre-fetching the tables you could do it already in theSchemaProvider.One approach could be to have an in memory mirror of theSchemaProviderthat stores only the essential data. And everytime its state is changed it schedules an async task to update the actual storage without awaiting the response. This however could lead to data races if multiple users access the storage simultaneously.Actually, after rethinking my earlier comment, I think it could make sense to make
SchemaProviderpartially async andContextProvidersync. One could realize theSchemaProvideras an in memory mirror with a "synchronisation" operation with the underlying storage service. If we look at theSchemaProvidertrait it has methods for reading and writing its state. One could make the reading methods sync, by reading from the in memory mirror and make the writing methods async, making sure the changes are propagated to the underlying storage service. Since the reading methods are sync, theContextProvidertrait could also be sync, simplifying the query planning step.One problem with this is that external changes to the underlying storage system are not automatically reflected in the in memory mirror, thus one would need to make sure to trigger a "synchronisation" operation before using the trait.
Additionally, one could provide sync and async versions for the reading methods, allowing operations that want to be sure the mirror is up to date to use the async versions.
tustvold commentedon Dec 12, 2022
Sorry for the delay, I plan to spend some time on this this week and try to come up with a proposal.
I think the key thing is to make it easy and obvious how to hook in a remote catalog, without requiring all of planning to become async. I'll see what I can come up with
Async catalog support (apache#3777)
Async catalog support (apache#3777)
Make SchemaProvider async (apache#3777)
Make SchemaProvider async (apache#3777)
tustvold commentedon Jan 3, 2023
#4607 is now ready for review, PTAL. Hopefully we can get this in to make the release at the end of this week 😄
Make SchemaProvider::table async (#4607)
JanKaul commentedon Jan 6, 2023
Thank you for your efforts!
chore(deps): bump urllib3 from 2.6.3 to 2.7.0 (apache#22109)