Skip to content
  • Mike Kolupaev's avatar
    Add an option to prevent DB::Open() from querying sizes of all sst files (#6353) · 637e64b9
    Mike Kolupaev authored
    Summary:
    When paranoid_checks is on, DBImpl::CheckConsistency() iterates over all sst files and calls Env::GetFileSize() for each of them. As far as I could understand, this is pretty arbitrary and doesn't affect correctness - if filesystem doesn't corrupt fsynced files, the file sizes will always match; if it does, it may as well corrupt contents as well as sizes, and rocksdb doesn't check contents on open.
    
    If there are thousands of sst files, getting all their sizes takes a while. If, on top of that, Env is overridden to use some remote storage instead of local filesystem, it can be *really* slow and overload the remote storage service. This PR adds an option to not do GetFileSize(); instead it does GetChildren() for parent directory to check that all the expected sst files are at least present, but doesn't check their sizes.
    
    We can't just disable paranoid_checks instead because paranoid_checks do a few other important things: make the DB read-only on write errors, print error messages on read errors, etc.
    Pull Request resolved: https://github.com/facebook/rocksdb/pull/6353
    
    Test Plan: ran the added sanity check unit test. Will try it out in a LogDevice test cluster where the GetFileSize() calls are causing a lot of trouble.
    
    Differential Revision: D19656425
    
    Pulled By: al13n321
    
    fbshipit-source-id: c2c421b367633033760d1f56747bad206d1fbf82
    637e64b9