| Author: | Francesc Alted |
|---|---|
| Contact: | [email protected] |
| URL: | http://www.blosc.org |
- None yet.
- LZ4 codec updated to 1.9.1.
- Zstd codec updated to 1.4.1.
- BloscLZ codec updated to 2.0.0. Although this should be fully backward compatible, it contains important changes that affects mainly speed, but also compression ratios. Feedback on how it behaves on your own data is appreciated.
- Fix for building for clang with -march=haswell. See PR #262.
- Fix all the known warnings for GCC/Clang. Still some work to do for MSVC in this front.
- Due to some problems with several CI systems, the check for library symbols are deactivated now by default. If you want to enforce this check, use: cmake .. -DDEACTIVATE_SYMBOLS_CHECK=ON to re-activate it.
- Correct the check for the compressed size when the buffer is memcpyed. This was a regression introduced in 1.16.0. Fixes #261.
- Fixed a regression in 1.16.0 that prevented to compress empty buffers (see #260).
- Zstd updated to 1.3.8 (from 1.3.7).
Now the functions that execute Blosc decompressions are safe by default for untrusted/possibly corrupted inputs. The additional checks seem to not affect performance significantly (see some benchmarks in #258), so this is why they are the default now.
The previous functions (with less safety) checks are still available with a '_unsafe' suffix. The complete list is:
- blosc_decompress_unsafe()
- blosc_decompress_ctx_unsafe()
- blosc_getitem_unsafe()
Also, a new API function named blosc_cbuffer_validate(), for validating Blosc compressed data, has been added.
For details, see PR #258. Thanks to Jeremy Maitin-Shepard.
Fixed a bug in blosc_compress() that could lead to thread deadlock under some situations. See #251. Thanks to @wenjuno for the report and the fix.
Fix data race in shuffle.c host_implementation initialization. Fixes #253. Thanks to Jeremy Maitin-Shepard.
- Add workaround for Visual Studio 2008's lack of a stdint.h file to blosclz.c.
- The blosc_compress() and blosc_decompress() interfaces are now fork-safe, preventing child-process deadlocks in fork-based multiprocessing applications. These interfaces with BLOSC_NOLOCK were, and continue to be, fork-safe. _ctx interface context reuse continues to be unsafe in the child process post-fork. See #241. Thanks to Alex Ford.
- Replaced //-comments with /**/-comments and other improvements for compatibility with quite old gcc compilers. See PR #243. Thanks to Andreas Martin.
- Empty buffers can be compressed again (this was unadvertedly prevented while fixing #234). See #247. Thanks to Valentin Haenel.
- LZ4 internal codec upgraded to 1.8.3 (from 1.8.1.2).
- Zstd internal codec upgraded to 1.3.7 (from 1.3.4).
- Added a new DEACTIVATE_SSE2 option for cmake that is useful for disabling SSE2 when doing cross-compilation (see #236).
- New check for detecting output buffers smaller than BLOSC_MAX_OVERHEAD. Fixes #234.
- The complib and version parameters for blosc_get_complib_info() can be safely set to NULL now. This allows to call this function even if the user is not interested in these parameters (so no need to reserve memory for them). Fixes #228.
- In some situations that a supposedly blosc chunk is passed to blosc_decompress(), one might end with an Arithmetic exception. This is probably due to the chunk not being an actual blosc chunk, and divisions by zero might occur. A protection has been added for this. See #237.
- Use win32/pthread.c on all Windows builds, even those with GNU compilers. Rational: although MinGW provides a more full-featured pthreads replacement, it doesn't seem to accomplish anything here since the functionality in win32/pthread.c is sufficient for Blosc. Furthermore, using the MinGW pthreads adds an additional library dependency to libblosc that is annoying for binary distribution. For example, it got in the way of distributing cross-compiled Windows binaries for use with Julia, since they want the resulting libblosc.dll to be usable on any Windows machine even where MinGW is not installed. See PR #224. Thanks to Steven G. Johnson.
- Zstd internal sources have been updated to 1.3.4.
- Reverted the $Configuration var in CMake configuration for Windows so as to restore the compatibility with MS VisualStudio compilers.
- Fixed a bug that caused C-Blosc to crash on platforms requiring strict alignment (as in some kinds of ARM CPUs). Fixes #223. Thanks to Elvis Stansvik and Michael Hudson-Doyle for their help.
- Fixed a piece of code that was not C89 compliant. C89 compliance is needed mainly by MS VS2008 which is still used for creating Python 2 extensions.
- Remove the (spurious) $Configuration var in cmake config for Windows. Thanks to Francis Brissette for pointing this out.
New split mode that favors forward compatibility. That means that, from now on, all the buffers created starting with blosc 1.14.0 will be forward compatible with any previous versions of the library --at least until 1.3.0, when support for multi-codec was introduced.
So as to select the split mode, a new API function has been introduced: https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.h#L500 Also, the BLOSC_SPLITMODE environment variable is honored when using the blosc_compress() function. See https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.h#L209
There is a dedicated blog entry about this at: http://blosc.org/posts/new-forward-compat-policy/ More info in PR #216.
Caveat Emptor: Note that Blosc versions from 1.11.0 to 1.14.0 might generate buffers that cannot be read with versions < 1.11.0, so if forward compatibility is important to you, an upgrade to 1.14.0 is recommended.
All warnings during cmake build stage are enabled by default now. PR #218. Thanks to kalvdans.
Better checks on versions of formats inside Blosc. PR #219. Thanks to kalvdans.
The BLOSC_PRINT_SHUFFLE_ACCEL environment variable is honored now. This is useful for determining at runtime whether the different SIMD capabilities (only for x86 kind processors) are available to Blosc to get better performance during shuffle/bitshuffle operation. As an example, here it is the normal output for the simple.c example:
$ ./simple Blosc version info: 1.14.0.dev ($Date:: 2018-02-15 #$) Compression: 4000000 -> 41384 (96.7x) Decompression succesful! Succesful roundtrip!
and here with the BLOSC_PRINT_SHUFFLE_ACCEL environment variable set:
$ BLOSC_PRINT_SHUFFLE_ACCEL= ./simple Blosc version info: 1.14.0.dev ($Date:: 2018-02-15 #$) Shuffle CPU Information: SSE2 available: True SSE3 available: True SSSE3 available: True SSE4.1 available: True SSE4.2 available: True AVX2 available: True AVX512BW available: False XSAVE available: True XSAVE enabled: True XMM state enabled: True YMM state enabled: True ZMM state enabled: False Compression: 4000000 -> 41384 (96.7x) Decompression succesful! Succesful roundtrip!
Blosc only currently leverages the SSE2 and AVX2 instruction sets, but it can recognize all of the above. This is useful mainly for debugging.
- More tests for binaries in https://bintray.com/blosc/Conan.
- More tests for binaries in https://bintray.com/blosc/Conan.
- New conan binaries publicly accessible in https://bintray.com/blosc/Conan. Still experimental, but feedback is appreciated.
- Fixed a buffer overrun that happens when compressing small buffers and len(destination_buffer) < (len(source_buffer) + BLOSC_MAX_OVERHEAD). Reported by Ivan Smirnov.
- Tests work now when external compressors are located in non-system locations. Fixes #210. Thanks to Leif Walsh.
- C-Blosc can be compiled on CentOS 6 now.
- LZ4 internal codec upgraded to 1.8.1.
- Fixed a bug uncovered by the python-blosc test suite: when a buffer is to be copied, then we should reserve space for the header, not block pointers.
- Serious optimization of memory copy functions (see new blosc/fastcopy.c). This benefits the speed of all the codecs, but specially the BloscLZ one.
- As a result of the above, the BloscLZ codec received a new adjustment of knobs so that you should expect better compression ratios with it too.
- LZ4 internal sources have been updated to 1.8.0.
- Zstd internal sources have been updated to 1.3.3.
- Backported BloscLZ parameters that were fine-tuned for C-Blosc2. You should expect better compression ratios and faster operation, specially on modern CPUs. See: http://blosc.org/posts/blosclz-tuning/
- Snappy, Zlib and Zstd codecs are compiled internally now, even if they are installed in the machine. This has been done in order to avoid problems in machines having the shared libraries for the codecs accessible but not the includes (typical in Windows boxes). Also, the Zstd codec runs much faster when compiled internally. The previous behaviour can be restored by activating the cmake options PREFER_EXTERNAL_SNAPPY, PREFER_EXTERNAL_ZLIB and PREFER_EXTERNAL_ZSTD.
- Zstd internal sources have been updated to 1.3.0.
- Internal Zstd codec updated to 1.1.4.
- Fixed #181: bitshuffle filter for big endian machines.
- Internal Zstd codec updated to 1.1.3.
- New blocksize for complevel 8 in automatic mode. This should help specially the Zstd codec to achieve better compression ratios.
- Enabled use as a CMake subproject, exporting shared & static library targets for super-projects to use. See PRs #178, #179 and #180. Thanks to Kevin Murray.
- Internal LZ4 codec updated to 1.7.5.
- Internal Zstd codec updated to 1.1.2.
- Fixed a bug introduced in 1.11.0 and discovered by pandas test suite. This basically prevented to decompress buffers compressed with previous versions of C-Blosc. See: Blosc/python-blosc#115
- Internal Zstd codec upgraded to 1.0.0.
- New block size computation inherited from C-Blosc2. Benchmarks are saying that this benefits mainly to LZ4, LZ4HC, Zlib and Zstd codecs, both in speed and in compression ratios (although YMMV for your case).
- Added the @rpath flag in Mac OSX for shared libraries. Fixes #175.
- Added a fix for VS2008 discovered in: https://github.com/PyTables/PyTables/pull/569/files#diff-953cf824ebfea7208d2a2e312d9ccda2L126
- License changed from MIT to 3-clause BSD style.
- Force the use of --std=gnu99 when using gcc. Fixes #174.
- Removed an inconsistent check for C11 (__STDC_VERSION__ >= 201112L and _ISOC11_SOURCE) as this seem to pose problems on compilers doing different things in this check (e.g. clang). See Blosc/bloscpack#50.
Initial support for Zstandard (0.7.4). Zstandard (or Zstd for short) is a new compression library that allows better compression than Zlib, but that works typically faster (and some times much faster), making of it a good match for Blosc.
Although the Zstd format is considered stable (http://fastcompression.blogspot.com.es/2016_07_03_archive.html), its API is maturing very fast, and despite passing the extreme test suite for C-Blosc, this codec should be considered in beta for C-Blosc usage purposes. Please test it and report back any possible issues you may get.
- Reverted a mistake introduced in 1.7.1. At that time, bit-shuffling was enabled for typesize == 1 (i.e. strings), but the change also included byte-shuffling accidentally. This only affected performance, but in a quite bad way (a copy was needed). This has been fixed and byte-shuffling is not active when typesize == 1 anymore.
- Check whether Blosc is actually initialized before blosc_init(), blosc_destroy() and blosc_free_resources(). This makes the library more resistant to different initialization cycles (e.g. JuliaIO/Blosc.jl#19).
- The internal copies when clevel=0 are made now via memcpy(). At the beginning of C-Blosc development, benchmarks where saying that the internal, multi-threaded copies inside C-Blosc were faster than memcpy(), but 6 years later, memcpy() made greats strides in terms of efficiency. With this, you should expect an slight speed advantage (10% ~ 20%) when C-Blosc is used as a replacement of memcpy() (which should not be the most common scenario out there).
- Added a new DEACTIVATE_AVX2 cmake option to explicitly disable AVX2 at build-time. Thanks to James Bird.
- The
make -jNfor parallel compilation should work now. Thanks to James Bird.
- New blosc_get_nthreads() function to get the number of threads that will be used internally during compression/decompression (set by already existing blosc_set_nthreads()).
- New blosc_get_compressor() function to get the compressor that will be used internally during compression (set by already existing blosc_set_compressor()).
- New blosc_get_blocksize() function to get the internal blocksize to be used during compression (set by already existing blosc_set_blocksize()).
- Now, when the BLOSC_NOLOCK environment variable is set (to any value), the calls to blosc_compress() and blosc_decompress() will call blosc_compress_ctx() and blosc_decompress_ctx() under the hood so as to avoid the internal locks. See blosc.h for details. This allows multi-threaded apps calling the non _ctx() functions to avoid the internal locks in C-Blosc. For the not multi-threaded app though, it is in general slower to call the _ctx() functions so the use of BLOSC_NOLOCK is discouraged.
- In the same vein, from now on, when the BLOSC_NTHREADS environment variable is set to an integer, every call to blosc_compress() and blosc_decompress() will call blosc_set_nthreads(BLOSC_NTHREADS) before the actuall compression/decompression process. See blosc.h for details.
- Finally, if BLOSC_CLEVEL, BLOSC_SHUFFLE, BLOSC_TYPESIZE and/or BLOSC_COMPRESSOR variables are set in the environment, these will be also honored before calling blosc_compress().
- Calling blosc_init() before any other Blosc call, although recommended, is not necessary anymore. The idea is that you can use just the basic blosc_compress() and blosc_decompress() and control other parameters (nthreads, compressor, blocksize) by using environment variables (see above).
- Disable the use of __builtin_cpu_supports() for GCC 5.3.1 compatibility. Details in: https://lists.fedoraproject.org/archives/list/[email protected]/thread/ZM2L65WIZEEQHHLFERZYD5FAG7QY2OGB/
- The code is (again) compatible with VS2008 and VS2010. This is important for compatibility with Python 2.6/2.7/3.3/3.4.
- Introduced a new global lock during blosc_decompress() operation. As the blosc_compress() was already guarded by a global lock, this means that the compression/decompression is again thread safe. However, when using C-Blosc from multi-threaded environments, it is important to keep using the *_ctx() functions for performance reasons. NOTE: _ctx() functions will be replaced by more powerful ones in C-Blosc 2.0.
- Fixed a bug preventing bitshuffle to work correctly on getitem(). Now, everything with bitshuffle seems to work correctly.
- Fixed the thread initialization for blosc_decompress_ctx(). Issue #158. Thanks to Chris Webers.
- Fixed a bug in the blocksize computation introduced in 1.7.0. This could have been creating segfaults.
- Allow bitshuffle to run on 1-byte typesizes.
- New parametrization of the blocksize to be independent of the typesize. This allows a smoother speed throughout all typesizes.
- lz4 and lz4hc codecs upgraded to 1.7.2 (from 1.7.0).
- When calling set_nthreads() but not actually changing the number of threads in the internal pool does not teardown and setup it anymore. PR #153. Thanks to Santi Villalba.
Added a new 'bitshuffle' filter so that the shuffle takes place at a bit level and not just at a byte one, which is what it does the previous 'shuffle' filter.
For activating this new bit-level filter you only have to pass the symbol BLOSC_BITSHUFFLE to blosc_compress(). For the previous byte-level one, pass BLOSC_SHUFFLE. For disabling the shuffle, pass BLOSC_NOSHUFFLE.
This is a port of the existing filter in https://github.com/kiyo-masui/bitshuffle. Thanks to Kiyo Masui for changing the license and allowing its inclusion here.
New acceleration mode for LZ4 and BloscLZ codecs that enters in operation with complevel < 9. This allows for an important boost in speed with minimal compression ratio loss. Francesc Alted.
LZ4 codec updated to 1.7.0 (r130).
PREFER_EXTERNAL_COMPLIBS cmake option has been removed and replaced by the more fine grained PREFER_EXTERNAL_LZ4, PREFER_EXTERNAL_SNAPPY and PREFER_EXTERNAL_ZLIB. In order to allow the use of the new API introduced in LZ4 1.7.0, PREFER_EXTERNAL_LZ4 has been set to OFF by default, whereas PREFER_EXTERNAL_SNAPPY and PREFER_EXTERNAL_ZLIB continues to be ON.
Implemented SSE2 shuffle support for buffers containing a number of elements which is not a multiple of (typesize * vectorsize). Jack Pappas.
Added SSE2 shuffle/unshuffle routines for types larger than 16 bytes. Jack Pappas.
'test_basic' suite has been split in components for a much better granularity on what's a possibly failing test. Also, lots of new tests have been added. Jack Pappas.
Fixed compilation on non-Intel archs (tested on ARM). Zbyszek Szmek.
Modifyied cmake files in order to inform that AVX2 on Visual Studio is supported only in 2013 update 2 and higher.
Added a replacement for stdbool.h for Visual Studio < 2013.
blosclz codec adds Win64/Intel as a platform supporting unaligned addressing. That leads to a speed-up of 2.2x in decompression.
New blosc_get_version_string() function for retrieving the version of the c-blosc library. Useful when linking with dynamic libraries and one want to know its version.
New example (win-dynamic-linking.c) that shows how to link a Blosc DLL dynamically in run-time (Windows only).
The context.threads_started is initialized now when decompressing. This could cause crashes in case you decompressed before compressing (e.g. directly deserializing blosc buffers). @atchouprakov.
The HDF5 filter has been removed from c-blosc and moved into its own repo at: https://github.com/Blosc/hdf5
The MS Visual Studio 2008 has been tested with c-blosc for ensuring compatibility with extensions for Python 2.6 and up.
- Support for runtime detection of AVX2 and SSE2 SIMD instructions. These changes make it possible to compile one single binary that runs on a system that supports SSE2 or AVX2 (or neither), so the redistribution problem is fixed (see #101). Thanks to Julian Taylor and Jack Pappas.
- Added support for MinGW and TDM-GCC compilers for Windows. Thanks to yasushima-gd.
- Fixed a bug in blosclz that could potentially overwrite an area beyond the output buffer. See #113.
- New computation for blocksize so that larger typesizes (> 8 bytes) would benefit of much better compression ratios. Speed is not penalized too much.
- New parametrization of the hash table for blosclz codec. This allows better compression in many scenarios, while slightly increasing the speed.
- Support for AVX2 is here! The benchmarks with a 4-core Intel Haswell machine tell that both compression and decompression are accelerated around a 10%, reaching peaks of 9.6 GB/s during compression and 26 GB/s during decompression (memcpy() speed for this machine is 7.5 GB/s for writes and 11.7 GB/s for reads). Many thanks to @littlezhou for this nice work.
- Support for HPET (high precision timers) for the bench program. This is particularly important for microbenchmarks like bench is doing; since they take so little time to run, the granularity of a less-accurate timer may account for a significant portion of the runtime of the benchmark itself, skewing the results. Thanks to Jack Pappas.
- Updated to LZ4 1.6.0 (r128).
- Fix resource leak in t_blosc. Jack Pappas.
- Better checks during testing. Jack Pappas.
- Dynamically loadable HDF5 filter plugin. Kiyo Masui.
- Use llabs function (where available) instead of abs to avoid truncating the result. Jack Pappas.
- Use C11 aligned_alloc when it's available. Jack Pappas.
- Use the built-in stdint.h with MSVC when available. Jack Pappas.
- Only define the __SSE2__ symbol when compiling with MS Visual C++ and targeting x64 or x86 with the correct /arch flag set. This avoids re-defining the symbol which makes other compilers issue warnings. Jack Pappas.
- Reinitializing Blosc during a call to set_nthreads() so as to fix problems with contexts. Francesc Alted.
Using blosc_compress_ctx() / blosc_decompress_ctx() inside the HDF5 compressor for allowing operation in multiprocess scenarios. See: PyTables/PyTables#412
The drawback of this quick fix is that the Blosc filter will be only able to use a single thread until another solution can be devised.
- Updated to LZ4 1.5.0. Closes #74.
- Added the 'const' qualifier to non SSE2 shuffle functions. Closes #75.
- Explicitly call blosc_init() in HDF5 blosc_filter.c, fixing a segfault.
- Quite a few improvements in cmake files for HDF5 support. Thanks to Dana Robinson (The HDF Group).
- Variable 'class' caused problems compiling the HDF5 filter with g++. Thanks to Laurent Chapon.
- Small improvements on docstrings of c-blosc main functions.
Added new calls for allowing Blosc to be used simultaneously (i.e. lock free) from multi-threaded environments. The new functions are:
- blosc_compress_ctx(...)
- blosc_decompress_ctx(...)
See the new docstrings in blosc.h for how to use them. The previous API should be completely unaffected. Thanks to Christopher Speller.
Optimized copies during BloscLZ decompression. This can make BloscLZ to decompress up to 1.5x faster in some situations.
LZ4 and LZ4HC compressors updated to version 1.3.1.
Added an examples directory on how to link apps with Blosc.
stdlib.h moved from blosc.c to blosc.h as suggested by Rob Lathm.
Fix a warning for {snappy,lz4}-free compilation. Thanks to Andrew Schaaf.
Several improvements for CMakeLists.txt (cmake).
Fixing C99 compatibility warnings. Thanks to Christopher Speller.
- Fixed a bug in blosc_getitem() introduced in 1.4.0. Added a test for blosc_getitem() as well.
- Support for non-Intel and non-SSE2 architectures has been added. In particular, the Raspberry Pi platform (ARM) has been tested and all tests pass here.
- Architectures requiring strict access alignment are supported as well. Due to this, arquitectures with a high penalty in accessing unaligned data (e.g. Raspberry Pi, ARMv6) can compress up to 2.5x faster.
- LZ4 has been updated to r119 (1.2.0) so as to fix a possible security breach.
Updated to LZ4 r118 due to a (highly unlikely) security hole. For details see:
http://fastcompression.blogspot.fr/2014/06/debunking-lz4-20-years-old-bug-myth.html
- Removed a pointer from 'pointer from integer without a cast' compiler warning due to a bad macro definition.
- Fixed a false buffer overrun condition. This bug made c-blosc to fail, even if the failure was not real.
- Fixed the type of a buffer string.
- Updated to LZ4 1.1.3 (improved speed for 32-bit platforms).
- Added a new blosc_cbuffer_complib() for getting the compression library for a compressed buffer.
- Fix for compiling Snappy sources against MSVC 2008. Thanks to Mark Wiebe!
- Version for internal LZ4 and Snappy are now supported. When compiled against the external libraries, this info is not available because they do not support the symbols (yet).
Fixes for a series of issues with the filter for HDF5 and, in particular, a problem in the decompression buffer size that made it impossible to use the blosc_filter in combination with other ones (e.g. fletcher32). See PyTables/PyTables#21.
Thanks to Antonio Valentino for the fix!
A nice handful of compressors have been added to Blosc:
- LZ4 (http://code.google.com/p/lz4/): A very fast compressor/decompressor. Could be thought as a replacement of the original BloscLZ, but it can behave better is some scenarios.
- LZ4HC (http://code.google.com/p/lz4/): This is a variation of LZ4 that achieves much better compression ratio at the cost of being much slower for compressing. Decompression speed is unaffected (and sometimes better than when using LZ4 itself!), so this is very good for read-only datasets.
- Snappy (http://code.google.com/p/snappy/): A very fast compressor/decompressor. Could be thought as a replacement of the original BloscLZ, but it can behave better is some scenarios.
- Zlib (http://www.zlib.net/): This is a classic. It achieves very good compression ratios, at the cost of speed. However, decompression speed is still pretty good, so it is a good candidate for read-only datasets.
With this, you can select the compression library with the new function:
int blosc_set_complib(char* complib);
where you pass the library that you want to use (currently "blosclz", "lz4", "lz4hc", "snappy" and "zlib", but the list can grow in the future).
You can get more info about compressors support in you Blosc build by using these functions:
char* blosc_list_compressors(void); int blosc_get_complib_info(char *compressor, char **complib, char **version);
Added a blosc_init() and blosc_destroy() so that the global lock can be initialized safely. These new functions will also allow other kind of initializations/destructions in the future.
Existing applications using Blosc do not need to start using the new functions right away, as long as they calling blosc_set_nthreads() previous to anything else. However, using them is highly recommended.
Thanks to Oscar Villellas for the init/destroy suggestion, it is a nice idea!
- All important warnings removed for all tested platforms. This will allow less intrusiveness compilation experiences with applications including Blosc source code.
- The bench/bench.c has been updated so that it can be compiled on Windows again.
- The new web site has been set to: http://www.blosc.org
- Fixed a problem with global lock not being initialized. This affected mostly to Windows platforms. Thanks to Christoph Gohlke for finding the cure!
Now it is possible to call Blosc simultaneously from a parent threaded application without problems. This has been solved by setting a global lock so that the different calling threads do not execute Blosc routines at the same time. Of course, real threading work is still available inside Blosc itself. Thanks to Thibault North.
Support for cmake is now included. Linux, Mac OSX and Windows platforms are supported. Thanks to Thibault North, Antonio Valentino and Mark Wiebe.
Fixed many compilers warnings (specially about unused variables).
As a consequence of the above, as minimal change in the API has been introduced. That is, the previous API:
void blosc_free_resources(void)
has changed to:
int blosc_free_resources(void)
Now, a return value of 0 means that the resources have been released successfully. If the return value is negative, then it is not guaranteed that all the resources have been freed.
Many typos were fixed and docs have been improved. The script for generating nice plots for the included benchmarks has been improved too. Thanks to Valetin Haenel.
- Fix compile error with msvc compilers (Christoph Gohlke)
- Redefinition of the BLOSC_MAX_BUFFERSIZE constant as (INT_MAX - BLOSC_MAX_OVERHEAD) instead of just INT_MAX. This prevents to produce outputs larger than INT_MAX, which is not supported.
- exit() call has been replaced by a
return -1in blosc_compress() when checking for buffer sizes. Now programs will not just exit when the buffer is too large, but return a negative code. - Improvements in explicit casts. Blosc compiles without warnings (with GCC) now.
- Lots of improvements in docs, in particular a nice ascii-art diagram of the Blosc format (Valentin Haenel).
- Improvements to the plot-speeds.py (Valentin Haenel).
- [HDF5 filter] Adapted HDF5 filter to use HDF5 1.8 by default (Antonio Valentino).
- [HDF5 filter] New version of H5Z_class_t definition (Antonio Valentino).
- Much improved compression ratio when using large blocks (> 64 KB) and high compression levels (> 6) under some circumstances (special data distribution). Closes #7.
- Fixes for small typesizes (#6 and #1 of python-blosc).
- Added code to avoid calling blosc_set_nthreads more than necessary. That will improve performance up to 3x or more, specially for small chunksizes (< 1 MB).
- Added code for emulating pthreads API on Windows. No need to link explicitly with pthreads lib on Windows anymore. However, performance is a somewhat worse because the new emulation layer does not support the pthread_barrier_wait() call natively. But the big improvement in installation easiness is worth this penalty (most specially on 64-bit Windows, where pthreads-win32 support is flaky).
- New BLOSC_MAX_BUFFERSIZE, BLOSC_MAX_TYPESIZE and BLOSC_MAX_THREADS symbols are available in blosc.h. These can be useful for validating parameters in clients. Thanks to Robert Smallshire for suggesting that.
- A new BLOSC_MIN_HEADER_LENGTH symbol in blosc.h tells how many bytes long is the minimum length of a Blosc header. blosc_cbuffer_sizes() only needs these bytes to be passed to work correctly.
- Removed many warnings (related with potentially dangerous type-casting code) issued by MSVC 2008 in 64-bit mode.
- Fixed a problem with the computation of the blocksize in the Blosc filter for HDF5.
- Fixed a problem with large datatypes. See http://www.pytables.org/trac/ticket/288 for more info.
- Now Blosc is able to work well even if you fork an existing process with a pool of threads. Bug discovered when PyTables runs in multiprocess environments. See http://pytables.org/trac/ticket/295 for details.
- Added a new blosc_getitem() call to allow the retrieval of items in sizes smaller than the complete buffer. That is useful for the carray project, but certainly for others too.
- Added a filter for HDF5 so that people can use Blosc outside PyTables, if they want to.
- Many small improvements, specially in README files.
- Do not assume that size_t is uint_32 for every platform.
- Added more protection for large buffers or in allocation memory routines.
- The src/ directory has been renamed to blosc/.
- The maxbytes parameter in blosc_compress() has been renamed to destsize. This is for consistency with the blosc_decompress() parameters.
- Now, compression level 0 is allowed, meaning not compression at all. The overhead of this mode will be always BLOSC_MAX_OVERHEAD (16) bytes. This mode actually represents using Blosc as a basic memory container.
- Supported a new parameter maxbytes for
blosc_compress(). It represents a maximum of bytes for output. Tests unit added too. - Added 3 new functions for querying different metadata on compressed buffers. A test suite for testing the new API has been added too.
- Support for cross-platform big/little endian compatibility in Blosc headers has been added.
- Fixed several failures exposed by the extremesuite. The problem was a bad check for limits in the buffer size while compressing.
- Added a new suite in bench.c called
debugsuitethat is appropriate for debugging purposes. Now, theextremesuitecan be used for running the complete (and extremely long) suite.
- Fixed several nasty bugs uncovered by the new suites in bench.c. Thanks to Tony Theodore and Gabriel Beckers for their (very) responsive beta testing and feedback.
- Added several modes (suites), namely
suite,hardsuiteandextremehardsuitein bench.c so as to allow different levels of testing.
- Internal format version bumped to 2 in order to allow an easy way to indicate that a buffer is being saved uncompressed. This is not supported yet, but it might be in the future.
- Blosc can use threads now for leveraging the increasing number of multi-core processors out there. See README-threaded.txt for more info.
- Added a protection for MacOSX so that it has to not link against posix_memalign() funtion, which seems not available in old versions of MacOSX (for example, Tiger). At nay rate, posix_memalign() is not necessary on Mac because 16 bytes alignment is ensured by default. Thanks to Ivan Vilata. Fixes #3.