• Igor Canadi's avatar
    New BlockBasedTable version -- better compressed block format · 9ab5adfc
    Igor Canadi authored
    This diff adds BlockBasedTable format_version = 2. New format version brings better compressed block format for these compressions:
    1) Zlib -- encode decompressed size in compressed block header
    2) BZip2 -- encode decompressed size in compressed block header
    3) LZ4 and LZ4HC -- instead of doing memcpy of size_t encode size as varint32. memcpy is very bad because the DB is not portable accross big/little endian machines or even platforms where size_t might be 8 or 4 bytes.
    It does not affect format for snappy.
    If you write a new database with format_version = 2, it will not be readable by RocksDB versions before 3.10. DB::Open() will return corruption in that case.
    Test Plan:
    Added a new test in db_test.
    I will also run db_bench and verify VSIZE when block_cache == 1GB
    Reviewers: yhchiang, rven, MarkCallaghan, dhruba, sdong
    Reviewed By: sdong
    Subscribers: dhruba, leveldb
    Differential Revision: https://reviews.facebook.net/D31461