Saturday, October 29, 2011

Snappy and the compressed format

Google snappy: A fast compressor/decompressor

1. Optimized for:
64-bit platforms
x86 little-endian

2. Block compression
Snappy compressor works in 32KB blocks and does not do matching across blocks, it will never produce a bitstream with offsets larger than about 32768. However, the decompressor should not rely on this, as it may change in the future.

3. Snappy is a LZ77-type compressor with a fixed, byte-oriented encoding.
The format of snappy compressed data like the left figure.

Sunday, June 12, 2011

Cassandra Compression and the Performance Evaluation

Even though we had put the Cassandra away in all our products, we would like to share our works here.

Why we put away the Cassandra in our products? Because:
(1) It is a big wrong in Cassandra’s implementation, especially on it’s local storage engine layer, i.e. SSTable and Indexing.
(2) It is a big wrong to combine Bigtable and Dynamo. Dynamo’s hash ring architecture is a obsolete technolohy for scale, it’s consistency and replication policy is also unusable in big data storage.