You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Browser-accessible official DataFusion playground / DataFusion fiddle #13818
Is your feature request related to a problem or challenge?
It would be useful to be able to quickly test queries in current DataFusion (latest release or nightly snapshot), without having to check out main branch and running the build.
It would lower the barrier of entry to verifying repro steps in issues, etc.
Describe the solution you'd like
a simple, official website hosted by the DataFusion project allowing to run queries using WASM build of DataFusion
Describe alternatives you've considered
cargo run
Not doing this and requiring contributors to instead
git checkout main && git pull &&cd datafusion-cli &&
cargo run
This is time consuming, and discourages issue reporters from checking their repro steps.
Or project maintainers from verifying things in current main.
docker image
Having datafusion docker image and using that.
This would be pretty convenient for maintainers, but not less accessible for the contributors or users. Browser-based would be more accessible
This should greatly reduce the bar of trying out DataFusion 👍
Actually I have tried the "black terminal" style before, as the first version:
It turns out that this style requires a lots of logic in the frontend side to simulate a terminal, which is why I change it to the datafusion-wasm-playground looks like now. But this is not the most important thing, just to mention it.
parquet viewer is super cool, especially when a user starts with some data in the form of parquet.
I was thinking about something more SQL centric. Most discussions in issues and PRs revolve around SELECT with perhaps CREATE TABLE, etc. It would be nice to be able to run those on current main or last released version, without having to create Parquet file first.
@XiangpengHao Do you think this can be evolved into simple query editor with SELECT and CREATE TABLE capabilities, where Parquet upload is an option to create a table?
currently parquet viewer is completely in Rust, and we create js binding when we need to interact with browser, so it can be quite challenging to incorporate to many fancy frontend lib, like monaco editor etc...
If we want some cool editor experience, it's better to create a binding of DataFusion and develop from the frontend side, like what @waynexia has done for datafusion-wasm-bindings.
But my frontend knowledge can barely understand what llm generates...
If people like it, I can donate it to a more official-looking repo and make it a starting point of datafusion online (or something similar)
For the donate part, how about making it the new datafusion-playground. Parquet viewer has implemented a great function set with a more beautiful appearance. Actually I don't want to maintain the old frontend project after seeing the viewer 🙈
changed the title [-]Browser-accessible official DataFusion playground[/-][+]Browser-accessible official DataFusion playground / DataFusion fiddle[/+]on Feb 4, 2025
Made it a while ago and still use it for quickly trying out stuff, there are lots of low hanging fruits for improving it (local storage based query history, syntax/error highlighting, a proper open source table component, etc...). If there's something I can do to help please let me know.
I'm imaging a highly encapsulated JS component like
<DataFusionconfig={config},interactive=true/>
and the user who wants an interactive terminal can render a DataFusion webpage with one line, who wants to use DataFusion WASM to do computation can invoke it as a JS function
Hi @XiangpengHao and @waynexia is there a way to make the a website like in parquet-viewer without reading the whole file from s3 like you implemented into the memory or are you saving it into filesystem?
because as far as I know its not possible to read and write to filesystem in wasm as of yet?
Hi @XiangpengHao and @waynexia is there a way to make the a website like in parquet-viewer without reading the whole file from s3 like you implemented into the memory or are you saving it into filesystem? because as far as I know its not possible to read and write to filesystem in wasm as of yet?
Parquet viewer does not read the whole file; it only reads the metadata and data relevant to your query.
Disk cache is done automatically by the browser, so it may or may not work depends on many things (request size, range, browser settings, freshness, etc)
I've opened a discussion to gather ideas on the design and features we’d like to see in the official DataFusion playground.
Would love to hear your thoughts—feel free to jump in and share your ideas. Thanks in advance! 😊
Note the site currently features v45.0.0. Would be great to have new fusion versions auto-deployed there.
However, I think it's not something DF project has currently bandwidth to take responsibility for.
Note the site currently features v45.0.0. Would be great to have new fusion versions auto-deployed there. However, I think it's not something DF project has currently bandwidth to take responsibility for.
Makes sense, I'd be happy to move it there. The app is hosted in vercel now connected to my Github account, not sure if there's something else to be done about that apart from just putting it into datafusion-contrib.
Makes sense, I'd be happy to move it there. The app is hosted in vercel now connected to my Github account, not sure if there's something else to be done about that apart from just putting it into datafusion-contrib.
Let's start just by moving the code into datafusion-contrib (I sent you an invite) and if it gets more attention and people want to help move it to some other hosting platform (like github pages or something) that might be a nice follow on contribution
Note: deployments won't work unless a datafusion-contrib owner accepts linking datafusion-contrib/datafusion-fiddle to my vercel account (a request should have already been sent) or a new vercel account is created and linked to the project. I'd be happy to still host the project in my vercel account, but if you think there's a better way that also works.
Note: deployments won't work unless a datafusion-contrib owner accepts linking datafusion-contrib/datafusion-fiddle to my vercel account (a request should have already been sent) or a new vercel account is created and linked to the project. I'd be happy to still host the project in my vercel account, but if you think there's a better way that also works.
Activity
alamb commentedon Dec 17, 2024
💯 Thank you @findepi
Here is some inspiration:
https://duckdb.org/docs/api/wasm/overview.html
It would be amazing to have something similar in the datafuion doc site. Something like this:
https://datafusion.apache.org/

alamb commentedon Dec 17, 2024
As @findepi points out there are at least two implementations of something similar already on #9834
These from @waynexia :
These from @XiangpengHao
waynexia commentedon Dec 18, 2024
This should greatly reduce the bar of trying out DataFusion 👍
Actually I have tried the "black terminal" style before, as the first version:
It turns out that this style requires a lots of logic in the frontend side to simulate a terminal, which is why I change it to the
datafusion-wasm-playgroundlooks like now. But this is not the most important thing, just to mention it.XiangpengHao commentedon Dec 19, 2024
That's exactly why I started parquet-viewer -- to view paruqet files and experiment with queries without using the cli tool.
If people like it, I can donate it to a more official-looking repo and make it a starting point of datafusion online (or something similar)
findepi commentedon Dec 19, 2024
parquet viewer is super cool, especially when a user starts with some data in the form of parquet.
I was thinking about something more SQL centric. Most discussions in issues and PRs revolve around SELECT with perhaps CREATE TABLE, etc. It would be nice to be able to run those on current main or last released version, without having to create Parquet file first.
@XiangpengHao Do you think this can be evolved into simple query editor with SELECT and CREATE TABLE capabilities, where Parquet upload is an option to create a table?
XiangpengHao commentedon Dec 19, 2024
Maybe we are looking for something like this? https://sql-workbench.com
currently parquet viewer is completely in Rust, and we create js binding when we need to interact with browser, so it can be quite challenging to incorporate to many fancy frontend lib, like monaco editor etc...
If we want some cool editor experience, it's better to create a binding of DataFusion and develop from the frontend side, like what @waynexia has done for datafusion-wasm-bindings.
But my frontend knowledge can barely understand what llm generates...
waynexia commentedon Dec 19, 2024
For the donate part, how about making it the new datafusion-playground. Parquet viewer has implemented a great function set with a more beautiful appearance. Actually I don't want to maintain the old frontend project after seeing the viewer 🙈
[-]Browser-accessible official DataFusion playground[/-][+]Browser-accessible official DataFusion playground / DataFusion fiddle[/+]alamb commentedon Feb 4, 2025
I just game across one from @gabotechs : https://datafusion-fiddle.vercel.app/
Repo: https://github.com/gabotechs/datafusion-fiddle
gabotechs commentedon Feb 5, 2025
Made it a while ago and still use it for quickly trying out stuff, there are lots of low hanging fruits for improving it (local storage based query history, syntax/error highlighting, a proper open source table component, etc...). If there's something I can do to help please let me know.
findepi commentedon Feb 5, 2025
This looks very cool, @gabotechs! And even better than the CLI itself (#13286) :)
I took the privilege to file the first enhancement idea - gabotechs/datafusion-fiddle#1.
waynexia commentedon Feb 25, 2025
I'm imaging a highly encapsulated JS component like
and the user who wants an interactive terminal can render a DataFusion webpage with one line, who wants to use DataFusion WASM to do computation can invoke it as a JS function
pranavJibhakate commentedon Mar 14, 2025
Hi @XiangpengHao and @waynexia is there a way to make the a website like in parquet-viewer without reading the whole file from s3 like you implemented into the memory or are you saving it into filesystem?
because as far as I know its not possible to read and write to filesystem in wasm as of yet?
XiangpengHao commentedon Mar 14, 2025
Parquet viewer does not read the whole file; it only reads the metadata and data relevant to your query.
Try this example: https://parquet-viewer.xiangpeng.systems/?url=https%3A%2F%2Fhuggingface.co%2Fdatasets%2FHuggingFaceFW%2Ffineweb%2Fresolve%2Fmain%2Fdata%2FCC-MAIN-2014-42%2F000_00005.parquet
You almost instantly get the query results even if the parquet file is a few GBs large.
Edit: caching is done by the browser
pranavJibhakate commentedon Mar 14, 2025
Thanks a lot I will look into the implementation of parquet-viewer about how the caching is done.
XiangpengHao commentedon Mar 14, 2025
This is the in-memory cache implementation: https://github.com/XiangpengHao/parquet-viewer/blob/main/src/object_store_cache.rs
Disk cache is done automatically by the browser, so it may or may not work depends on many things (request size, range, browser settings, freshness, etc)
qstommyshu commentedon Apr 9, 2025
Hi everyone,
I've opened a discussion to gather ideas on the design and features we’d like to see in the official DataFusion playground.
Would love to hear your thoughts—feel free to jump in and share your ideas. Thanks in advance! 😊
findepi commentedon Jun 25, 2025
https://datafusion-fiddle.vercel.app/ exists and is awesome. I am closing the issue.
Note the site currently features v45.0.0. Would be great to have new fusion versions auto-deployed there.
However, I think it's not something DF project has currently bandwidth to take responsibility for.
alamb commentedon Jun 25, 2025
Looks like @gabotechs is the owner of https://github.com/gabotechs/datafusion-fiddle
If you want to move the app into the
datafusion-contriborganization https://github.com/datafusion-contrib @gabotechs I would be happy to give you the permissions to do soMoving it into that organization might make it easier for other people to find and contribute to
gabotechs commentedon Jun 26, 2025
Makes sense, I'd be happy to move it there. The app is hosted in vercel now connected to my Github account, not sure if there's something else to be done about that apart from just putting it into datafusion-contrib.
alamb commentedon Jun 26, 2025
Let's start just by moving the code into datafusion-contrib (I sent you an invite) and if it gets more attention and people want to help move it to some other hosting platform (like github pages or something) that might be a nice follow on contribution
I'll invite you to the org
gabotechs commentedon Jun 28, 2025
Moved, it's already on https://github.com/datafusion-contrib/datafusion-fiddle. I made a small revamp before moving it, so it should look a bit better now.
Note: deployments won't work unless a datafusion-contrib owner accepts linking datafusion-contrib/datafusion-fiddle to my vercel account (a request should have already been sent) or a new vercel account is created and linked to the project. I'd be happy to still host the project in my vercel account, but if you think there's a better way that also works.
alamb commentedon Jun 28, 2025
Nice! it is also at DataFusion 48
Done!
Thank you