Skip to content
  • Peter Dillinger's avatar
    For ApproximateSizes, pro-rate table metadata size over data blocks (#6784) · 14eca6bf
    Peter Dillinger authored
    Summary:
    The implementation of GetApproximateSizes was inconsistent in
    its treatment of the size of non-data blocks of SST files, sometimes
    including and sometimes now. This was at its worst with large portion
    of table file used by filters and querying a small range that crossed
    a table boundary: the size estimate would include large filter size.
    
    It's conceivable that someone might want only to know the size in terms
    of data blocks, but I believe that's unlikely enough to ignore for now.
    Similarly, there's no evidence the internal function AppoximateOffsetOf
    is used for anything other than a one-sided ApproximateSize, so I intend
    to refactor to remove redundancy in a follow-up commit.
    
    So to fix this, GetApproximateSizes (and implementation details
    ApproximateSize and ApproximateOffsetOf) now consistently include in
    their returned sizes a portion of table file metadata (incl filters
    and indexes) based on the size portion of the data blocks in range. In
    other words, if a key range covers data blocks that are X% by size of all
    the table's data blocks, returned approximate size is X% of the total
    file size. It would technically be more accurate to attribute metadata
    based on number of keys, but that's not computationally efficient with
    data available and rarely a meaningful difference.
    
    Also includes miscellaneous comment improvements / clarifications.
    
    Also included is a new approximatesizerandom benchmark for db_bench.
    No significant performance difference seen with this change, whether ~700 ops/sec with cache_index_and_filter_blocks and small cache or ~150k ops/sec without cache_index_and_filter_blocks.
    Pull Request resolved: https://github.com/facebook/rocksdb/pull/6784
    
    Test Plan:
    Test added to DBTest.ApproximateSizesFilesWithErrorMargin.
    Old code running new test...
    
        [ RUN      ] DBTest.ApproximateSizesFilesWithErrorMargin
        db/db_test.cc:1562: Failure
        Expected: (size) <= (11 * 100), actual: 9478 vs 1100
    
    Other tests updated to reflect consistent accounting of metadata.
    
    Reviewed By: siying
    
    Differential Revision: D21334706
    
    Pulled By: pdillinger
    
    fbshipit-source-id: 6f86870e45213334fedbe9c73b4ebb1d8d611185
    14eca6bf
To find the state of this project's repository at the time of any of these versions, check out the tags.