Skip to content

Add optional Blosc2 compression support #10986

Open

Description

@niltecedu

Problem or motivation

I’ve been working with scientific data, including numerical weather data stored with Blosc2 compression, and I’m interested in whether Vortex could support it too. At the moment, converting this data to Vortex means using pcodec which is nice and all however pcodec suffers a lot of smaller data sizes or nulls. Atleast based on my experience with vortex compact

Proposed solution

To add blosc2 byte shuffle + lz4 and blosc2 byte shuffle+ zstd1 as one of the compression levels, or something more configurable like how hdf5plugin does it. I can try to do a POC of it.

Additional context

https://hdf5plugin.readthedocs.io/en/stable/usage.html#blosc2
https://blosc.org/

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA feature request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions