1. 06 Jun, 2020 2 commits
    • anand76's avatar
      Update version · ff76f053
      anand76 authored
    • anand76's avatar
      Check iterator status BlockBasedTableReader::VerifyChecksumInBlocks() (#6909) · 402fe7d4
      anand76 authored
      The ```for``` loop in ```VerifyChecksumInBlocks``` only checks ```index_iter->Valid()``` which could be ```false``` either due to reaching the end of the index or, in case of partitioned index, it could be due to a checksum mismatch error when reading a 2nd level index block. Instead of throwing away the index iterator status, we need to return any errors back to the caller.
      Add a test in block_based_table_reader_test.cc.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6909
      Reviewed By: pdillinger
      Differential Revision: D21833922
      Pulled By: anand1976
      fbshipit-source-id: bc778ebf1121dbbdd768689de5183f07a9f0beae
  2. 04 Jun, 2020 1 commit
  3. 28 May, 2020 1 commit
  4. 19 May, 2020 1 commit
  5. 14 May, 2020 2 commits
    • Yanqin Jin's avatar
      Do not print u'string' in TARGETS file (#6841) · 98d19c30
      Yanqin Jin authored
      Before this PR, extra deps passed in from cmd line to buckifier will be parsed
      and used to populate a dict. Using this dict and printing to TARGETS file will
      lead to printing u'', disallowed by build tools. This PR removes the u''.
      Test Plan (local dev server):
      python buckifier/buckify_rocksdb.py  '{"fake": {"extra_deps": [":test_dep", "//fake/module:mock1"], "extra_compiler_flags": ["-Os", "-DROCKSDB_LITE"]}}'
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6841
      Reviewed By: siying
      Differential Revision: D21538155
      Pulled By: riversand963
      fbshipit-source-id: 09403668a4aa1a15bad7dac229c2bc8ce8ee1349
    • anand76's avatar
      Update HISTORY.md with a missing API change · 3f4008b1
      anand76 authored
  6. 12 May, 2020 1 commit
    • sdong's avatar
      Improve ldb consistency checks (#6802) · 8f9cc109
      sdong authored
      When using ldb, users cannot turn on force consistency check in most commands, while they cannot use checksonsistnecy with --try_load_options. The change fixes both by:
      1. checkconsistency now calls OpenDB() so that it gets all the options loading and sanitized options logic
      2. use options.check_consistency_checks = true by default, and add a --disable_consistency_checks to turn it off.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6802
      Test Plan: Add a new unit test. Some manual tests with corrupted DBs.
      Reviewed By: pdillinger
      Differential Revision: D21388051
      fbshipit-source-id: 8d122732d391b426e3982a1c3232a8e3763ffad0
  7. 09 May, 2020 7 commits
  8. 08 May, 2020 3 commits
  9. 02 May, 2020 1 commit
  10. 01 May, 2020 11 commits
    • Zhichao Cao's avatar
      Fix multiple CF replay failure in db_bench replay (#6787) · c8643edf
      Zhichao Cao authored
      The multiple CF hash map is not passed to the multi-thread worker. When using multi-thread replay for multiple CFs, it will cause segment fault. Pass the cf_map to the argument.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6787
      Test Plan: pass trace replay test.
      Reviewed By: yhchiang
      Differential Revision: D21339941
      Pulled By: zhichao-cao
      fbshipit-source-id: 434482b492287e6722c7cd5a706f057c5ec170ce
    • Yanqin Jin's avatar
      Add Github Action for some basic sanity test of PR (#6761) · 6acbbbf9
      Yanqin Jin authored
      Add Github Action to perform some basic sanity check for PR, inclding the
      1) Buck TARGETS file.
      On the one hand, The TARGETS file is used for internal buck, and we do not
      manually update it. On the other hand, we need to run the buckifier scripts to
      update TARGETS whenever new files are added, etc. With this Github Action, we
      make sure that every PR does not forget this step. The GH Action uses
      a Makefile target called check-buck-targets. Users can manually run `make
      check-buck-targets` on local machine.
      2) Code format
      We use clang-format-diff.py to format our code. The GH Action in this PR makes
      sure this step is not skipped. The checking script build_tools/format-diff.sh assumes that `clang-format-diff.py` is executable.
      On host running GH Action, it is difficult to download `clang-format-diff.py` and make it
      executable. Therefore, we modified build_tools/format-diff.sh to handle the case in which there is a non-executable clang-format-diff.py file in the top-level rocksdb repo directory.
      Test Plan (Github and devserver):
      Watch for Github Action result in the `Checks` tab.
      On dev server
      make check-format
      make check-buck-targets
      make check
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6761
      Test Plan: Watch for Github Action result in the `Checks` tab.
      Reviewed By: pdillinger
      Differential Revision: D21260209
      Pulled By: riversand963
      fbshipit-source-id: c646e2f37c6faf9f0614b68aa0efc818cff96787
    • sdong's avatar
      Remove the support of setting CompressionOptions.parallel_threads from string for now (#6782) · 6504ae0c
      sdong authored
      The current way of implementing CompressionOptions.parallel_threads introduces a format change. We plan to change CompressionOptions's serailization format to a new JSON-like format, which would be another format change. We would like to consolidate the two format changes into one, rather than making some users to change twice. Hold CompressionOptions.parallel_threads from being supported by option string for now. Will add it back after the general CompressionOptions's format change.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6782
      Test Plan: Run all existing tests.
      Reviewed By: zhichao-cao
      Differential Revision: D21338614
      fbshipit-source-id: bca2dac3cb37d4e6e64b52cbbe8ea749cd848685
    • Cheng Chang's avatar
      Make users explicitly be aware of prepare before commit (#6775) · ef0c3eda
      Cheng Chang authored
      In current commit protocol of pessimistic transaction, if the transaction is not prepared before commit, the commit protocol implicitly assumes that the user wants to commit without prepare.
      This PR adds TransactionOptions::skip_prepare, the default value is `true` because if set to `false`, all existing users who commit without prepare need to update their code to set skip_prepare to true. Although this does not force the user to explicitly express their intention of skip_prepare, it at least lets the user be aware of the assumption of being able to commit without prepare.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6775
      Test Plan: added a new unit test TransactionTest::CommitWithoutPrepare
      Reviewed By: lth
      Differential Revision: D21313270
      Pulled By: cheng-chang
      fbshipit-source-id: 3d95b7c9b2d6cdddc09bdd66c561bc4fae8c3251
    • sdong's avatar
      Disallow BlockBasedTableBuilder to set status from non-OK (#6776) · 079e50d2
      sdong authored
      There is no systematic mechanism to prevent BlockBasedTableBuilder's status to be set from non-OK to OK. Adding a mechanism to force this will help us prevent failures in the future.
      The solution is to only make it possible to set the status code if the status code to set is not OK.
      Since the status code passed to CompressAndVerifyBlock() is changed, a mini refactoring is done too so that the output arguments are changed from reference to pointers, based on Google C++ Style.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6776
      Test Plan: Run all existing test.
      Reviewed By: pdillinger
      Differential Revision: D21314382
      fbshipit-source-id: 27000c10f1e4c121661e026548d6882066409375
    • sdong's avatar
      Flag CompressionOptions::parallel_threads to be experimental (#6781) · 6277e280
      sdong authored
      The feature of CompressionOptions::parallel_threads is still not yet mature. Mention it to be experimental in the comments for now.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6781
      Reviewed By: pdillinger
      Differential Revision: D21330678
      fbshipit-source-id: d7dd7d099fb002a5c6a5d8da689ce5ee08a9eb13
    • anand76's avatar
      Pass a timeout to FileSystem for random reads (#6751) · ab13d43e
      anand76 authored
      Calculate ```IOOptions::timeout``` using ```ReadOptions::deadline``` and pass it to ```FileSystem::Read/FileSystem::MultiRead```. This allows us to impose a tighter bound on the time taken by Get/MultiGet on FileSystem/Envs that support IO timeouts. Even on those that don't support, check in ```RandomAccessFileReader::Read``` and ```MultiRead``` and return ```Status::TimedOut()``` if the deadline is exceeded.
      For now, TableReader creation, which might do file opens and reads, are not covered. It will be implemented in another PR.
      Update existing unit tests to verify the correct timeout value is being passed
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6751
      Reviewed By: riversand963
      Differential Revision: D21285631
      Pulled By: anand1976
      fbshipit-source-id: d89af843e5a91ece866e87aa29438b52a65a8567
    • Peter Dillinger's avatar
      Fix assertion that can fail on sst corruption (#6780) · eecd8fba
      Peter Dillinger authored
      An assertion that a char == a CompressionType (unsigned char)
      originally cast from a char can fail if the original value is negative,
      due to numeric promotion.  The assertion should pass even if the value
      is invalid CompressionType, because the callee
      UncompressBlockContentsForCompressionType checks for that and reports
      status appropriately.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6780
      Test Plan:
      Temporarily change kZSTD = 0x88 and see tests fail. Make this
      change (in addition), and tests pass.
      Reviewed By: siying
      Differential Revision: D21328498
      Pulled By: pdillinger
      fbshipit-source-id: 61caf8d815581ce49261ecb7ab0f396e9ac4bb92
    • Levi Tamasi's avatar
      Keep track of obsolete blob files in VersionSet (#6755) · fe238e54
      Levi Tamasi authored
      The patch adds logic to keep track of obsolete blob files. A blob file becomes
      obsolete when the last `shared_ptr` that points to the corresponding
      `SharedBlobFileMetaData` object goes away, which, in turn, happens when the
      last `Version` that contains the blob file is destroyed. No longer needed blob
      files are added to the obsolete list in `VersionSet` using a custom deleter to
      avoid unnecessary coupling between `SharedBlobFileMetaData` and `VersionSet`.
      Obsolete blob files are returned by `VersionSet::GetObsoleteFiles` and stored
      in `JobContext`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6755
      Test Plan: `make check`
      Reviewed By: riversand963
      Differential Revision: D21233155
      Pulled By: ltamasi
      fbshipit-source-id: 47757e06fdc0127f27ed57f51abd27893d9a7b7a
    • Adam Retter's avatar
      Add Slack forum to README (#6773) · cf342464
      Adam Retter authored
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6773
      Reviewed By: siying
      Differential Revision: D21310229
      Pulled By: pdillinger
      fbshipit-source-id: c0d52d0c51121d307d7d5c1374abc7bf78b0c4cf
    • Ziyue Yang's avatar
      Add an option for parallel compression in for db_stress (#6722) · e619a20e
      Ziyue Yang authored
      This commit adds an `compression_parallel_threads` option in
      db_stress. It also fixes the naming of parallel compression
      option in db_bench to keep it aligned with others.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6722
      Reviewed By: pdillinger
      Differential Revision: D21091385
      fbshipit-source-id: c9ba8c4e5cc327ff9e6094a6dc6a15fcff70f100
  11. 30 Apr, 2020 4 commits
  12. 29 Apr, 2020 6 commits
    • Peter Dillinger's avatar
      Fix LITE build (#6770) · 8086e5e2
      Peter Dillinger authored
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6770
      Test Plan: make LITE=1 check
      Reviewed By: ajkr
      Differential Revision: D21296261
      Pulled By: pdillinger
      fbshipit-source-id: b6075cc13a6d6db48617b7e0e9ebeea9364dfd9f
    • anand76's avatar
      Fix a valgrind failure due to DBBasicTestMultiGetDeadline (#6756) · 335ea73e
      anand76 authored
      Fix a valgrind failure.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6756
      Test Plan: valgrind_test
      Reviewed By: pdillinger
      Differential Revision: D21284660
      Pulled By: anand1976
      fbshipit-source-id: 39bf1bd130b6adb585ddbf2f9aa2f53dbf666f80
    • mrambacher's avatar
      Add Functions to OptionTypeInfo (#6422) · 618bf638
      mrambacher authored
      Added functions for parsing, serializing, and comparing elements to OptionTypeInfo.  These functions allow all of the special cases that could not be handled directly in the map of OptionTypeInfo to be moved into the map.  Using these functions, every type can be handled via the map rather than special cased.
      By adding these functions, the code for handling options can become more standardized (fewer special cases) and (eventually) handled completely by common classes.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6422
      Test Plan: pass make check
      Reviewed By: siying
      Differential Revision: D21269005
      Pulled By: zhichao-cao
      fbshipit-source-id: 9ba71c721a38ebf9ee88259d60bd81b3282b9077
    • Peter Dillinger's avatar
      Clarifying comments in db.h (#6768) · b810e62b
      Peter Dillinger authored
      And fix a confusingly worded log message
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6768
      Reviewed By: anand1976
      Differential Revision: D21284527
      Pulled By: pdillinger
      fbshipit-source-id: f03c1422c229a901c3a65e524740452349626164
    • Peter Dillinger's avatar
      Basic MultiGet support for partitioned filters (#6757) · bae6f586
      Peter Dillinger authored
      In MultiGet, access each applicable filter partition only once
      per batch, rather than for each applicable key. Also,
      * Fix Bloom stats for MultiGet
      * Fix/refactor MultiGetContext::Range::KeysLeft, including
      * Add efficient BitsSetToOne implementation
      * Assert that MultiGetContext::Range does not go beyond shift range
      Performance test: Generate db:
          $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 -partition_index_and_filters=true
      Before (middle performing run of three; note some missing Bloom stats):
          $ ./db_bench --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget'
          multireadrandom :      26.403 micros/op 597517 ops/sec; (548427 of 671968 found)
          rocksdb.block.cache.filter.hit COUNT : 83443275
          rocksdb.bloom.filter.useful COUNT : 0
          rocksdb.bloom.filter.full.positive COUNT : 0
          rocksdb.bloom.filter.full.true.positive COUNT : 7931450
          rocksdb.number.multiget.get COUNT : 385984
          rocksdb.number.multiget.keys.read COUNT : 12351488
          rocksdb.number.multiget.bytes.read COUNT : 793145000
          rocksdb.number.multiget.keys.found COUNT : 7931450
      After (middle performing run of three):
          $ ./db_bench_new --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget'
          multireadrandom :      21.024 micros/op 752963 ops/sec; (705188 of 863968 found)
          rocksdb.block.cache.filter.hit COUNT : 49856682
          rocksdb.bloom.filter.useful COUNT : 45684579
          rocksdb.bloom.filter.full.positive COUNT : 10395458
          rocksdb.bloom.filter.full.true.positive COUNT : 9908456
          rocksdb.number.multiget.get COUNT : 481984
          rocksdb.number.multiget.keys.read COUNT : 15423488
          rocksdb.number.multiget.bytes.read COUNT : 990845600
          rocksdb.number.multiget.keys.found COUNT : 9908456
      So that's about 25% higher throughput even for random keys
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6757
      Test Plan: unit test included
      Reviewed By: anand1976
      Differential Revision: D21243256
      Pulled By: pdillinger
      fbshipit-source-id: 5644a1468d9e8c8575be02f4e04bc5d62dbbb57f
    • Peter Dillinger's avatar
      HISTORY.md update for bzip upgrade (#6767) · a7f0b27b
      Peter Dillinger authored
      See https://github.com/facebook/rocksdb/issues/6714 and https://github.com/facebook/rocksdb/issues/6703
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6767
      Reviewed By: riversand963
      Differential Revision: D21283307
      Pulled By: pdillinger
      fbshipit-source-id: 8463bec725669d13846c728ad4b5bde43f9a84f8