NumFOCUS is pleased to announce the newest addition to our fiscally sponsored projects: Blosc, a very high performance meta-compressor specially designed for compressing binary data.
Blosc is a high performance compression library written in C that uses the blocking technique to make compression operations easier, faster, and more flexible. Blosc splits the datasets into blocks, then transparently packs them into compressed containers. Blosc extends standard compression by allowing the user to condition the data in every block with filters prior to the compression operation; these filters can be selected depending on the properties of the dataset to be compressed. In addition, Blosc provides a diversity of codecs to cover different needs (better compression, faster speed or a balance between the two). Finally, Blosc runs its operations in parallel to leverage the high number of cores in modern CPUs.
Blosc is designed to help in any application where (binary) data needs to be compressed as fast as possible so as to minimize the impact in handling compressed data transparently.
Blosc is currently used in many projects, among them PyTables (there is also a stand-alone hdf5-blosc plugin for general HDF5 applications), bcolz, and zarr (via its numcodecs subproject) and it is widely appreciated for its fast performance and stability. Via these projects Blosc is used in Earth Data Science, databases, the financial industry and many others. Hundreds of visitors access the different Blosc projects in GitHub on a daily basis.
The Blosc leadership body for NumFOCUS consists of Francesc Alted, Anthony Scopatz and Alberto Sabater.
With the addition of Blosc, the NumFOCUS fiscal sponsorship program now encompasses 26 open source scientific computing projects.