Skip to content
  • Zhichao Cao's avatar
    Checksum for each SST file and stores in MANIFEST (#6216) · 4369f2c7
    Zhichao Cao authored
    Summary:
    In the current code base, RocksDB generate the checksum for each block and verify the checksum at usage. Current PR enable SST file checksum. After a SST file is generated by Flush or Compaction, RocksDB generate the SST file checksum and store the checksum value and checksum method name in the vs_info and MANIFEST as part for the FileMetadata.
    
    Added the enable_sst_file_checksum to Options to enable or disable file checksum. Added sst_file_checksum to Options such that user can plugin their own SST file checksum calculate method via overriding the SstFileChecksum class. The checksum information inlcuding uint32_t checksum value and a checksum name (string).  A new tool is added to LDB such that user can dump out a list of file checksum information from MANIFEST. If user enables the file checksum but does not provide the sst_file_checksum instance, RocksDB will use the default crc32checksum implemented in table/sst_file_checksum_crc32c.h
    Pull Request resolved: https://github.com/facebook/rocksdb/pull/6216
    
    Test Plan: Added the testing case in table_test and ldb_cmd_test to verify checksum is correct in different level. Pass make asan_check.
    
    Differential Revision: D19171461
    
    Pulled By: zhichao-cao
    
    fbshipit-source-id: b2e53479eefc5bb0437189eaa1941670e5ba8b87
    4369f2c7