1. 06 6月, 2020 3 次提交
    • Peter Dillinger's avatar
      Misc things for ASSERT_STATUS_CHECKED, also gcc 4.8.5 (#6871) · 1482c869
      Peter Dillinger 创作于
      Summary:
      * Print stack trace on status checked failure
      * Make folly_synchronization_distributed_mutex_test a parallel test
      * Disable ldb_test.py and rocksdb_dump_test.sh with
        ASSERT_STATUS_CHECKED (broken)
      * Fix shadow warning in random_access_file_reader.h reported by gcc
        4.8.5 (ROCKSDB_NO_FBCODE), also https://github.com/facebook/rocksdb/issues/6866
      * Work around compiler bug on max_align_t for gcc < 4.9
      * Remove an apparently wrong comment in status.h
      * Use check_some in Travis config (for proper diagnostic output)
      * Fix ignored Status in loop in options_helper.cc
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6871
      
      Test Plan: manual, CI
      
      Reviewed By: ajkr
      
      Differential Revision: D21706619
      
      Pulled By: pdillinger
      
      fbshipit-source-id: daf6364173d6689904eb394461a69a11f5bee2cb
      1482c869
    • anand76's avatar
      Update version · ff76f053
      anand76 创作于
      ff76f053
    • anand76's avatar
      Check iterator status BlockBasedTableReader::VerifyChecksumInBlocks() (#6909) · 402fe7d4
      anand76 创作于
      Summary:
      The ```for``` loop in ```VerifyChecksumInBlocks``` only checks ```index_iter->Valid()``` which could be ```false``` either due to reaching the end of the index or, in case of partitioned index, it could be due to a checksum mismatch error when reading a 2nd level index block. Instead of throwing away the index iterator status, we need to return any errors back to the caller.
      
      Tests:
      Add a test in block_based_table_reader_test.cc.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6909
      
      Reviewed By: pdillinger
      
      Differential Revision: D21833922
      
      Pulled By: anand1976
      
      fbshipit-source-id: bc778ebf1121dbbdd768689de5183f07a9f0beae
      402fe7d4
  2. 04 6月, 2020 1 次提交
  3. 28 5月, 2020 1 次提交
  4. 19 5月, 2020 1 次提交
  5. 14 5月, 2020 2 次提交
    • Yanqin Jin's avatar
      Do not print u'string' in TARGETS file (#6841) · 98d19c30
      Yanqin Jin 创作于
      Summary:
      Before this PR, extra deps passed in from cmd line to buckifier will be parsed
      and used to populate a dict. Using this dict and printing to TARGETS file will
      lead to printing u'', disallowed by build tools. This PR removes the u''.
      
      Test Plan (local dev server):
      ```
      python buckifier/buckify_rocksdb.py  '{"fake": {"extra_deps": [":test_dep", "//fake/module:mock1"], "extra_compiler_flags": ["-Os", "-DROCKSDB_LITE"]}}'
      ```
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6841
      
      Reviewed By: siying
      
      Differential Revision: D21538155
      
      Pulled By: riversand963
      
      fbshipit-source-id: 09403668a4aa1a15bad7dac229c2bc8ce8ee1349
      98d19c30
    • anand76's avatar
      Update HISTORY.md with a missing API change · 3f4008b1
      anand76 创作于
      3f4008b1
  6. 12 5月, 2020 1 次提交
    • sdong's avatar
      Improve ldb consistency checks (#6802) · 8f9cc109
      sdong 创作于
      Summary:
      When using ldb, users cannot turn on force consistency check in most commands, while they cannot use checksonsistnecy with --try_load_options. The change fixes both by:
      1. checkconsistency now calls OpenDB() so that it gets all the options loading and sanitized options logic
      2. use options.check_consistency_checks = true by default, and add a --disable_consistency_checks to turn it off.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6802
      
      Test Plan: Add a new unit test. Some manual tests with corrupted DBs.
      
      Reviewed By: pdillinger
      
      Differential Revision: D21388051
      
      fbshipit-source-id: 8d122732d391b426e3982a1c3232a8e3763ffad0
      8f9cc109
  7. 09 5月, 2020 7 次提交
  8. 08 5月, 2020 3 次提交
  9. 02 5月, 2020 1 次提交
  10. 01 5月, 2020 11 次提交
    • Zhichao Cao's avatar
      Fix multiple CF replay failure in db_bench replay (#6787) · c8643edf
      Zhichao Cao 创作于
      Summary:
      The multiple CF hash map is not passed to the multi-thread worker. When using multi-thread replay for multiple CFs, it will cause segment fault. Pass the cf_map to the argument.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6787
      
      Test Plan: pass trace replay test.
      
      Reviewed By: yhchiang
      
      Differential Revision: D21339941
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: 434482b492287e6722c7cd5a706f057c5ec170ce
      c8643edf
    • Yanqin Jin's avatar
      Add Github Action for some basic sanity test of PR (#6761) · 6acbbbf9
      Yanqin Jin 创作于
      Summary:
      Add Github Action to perform some basic sanity check for PR, inclding the
      following.
      1) Buck TARGETS file.
      On the one hand, The TARGETS file is used for internal buck, and we do not
      manually update it. On the other hand, we need to run the buckifier scripts to
      update TARGETS whenever new files are added, etc. With this Github Action, we
      make sure that every PR does not forget this step. The GH Action uses
      a Makefile target called check-buck-targets. Users can manually run `make
      check-buck-targets` on local machine.
      
      2) Code format
      We use clang-format-diff.py to format our code. The GH Action in this PR makes
      sure this step is not skipped. The checking script build_tools/format-diff.sh assumes that `clang-format-diff.py` is executable.
      On host running GH Action, it is difficult to download `clang-format-diff.py` and make it
      executable. Therefore, we modified build_tools/format-diff.sh to handle the case in which there is a non-executable clang-format-diff.py file in the top-level rocksdb repo directory.
      
      Test Plan (Github and devserver):
      Watch for Github Action result in the `Checks` tab.
      On dev server
      ```
      make check-format
      make check-buck-targets
      make check
      ```
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6761
      
      Test Plan: Watch for Github Action result in the `Checks` tab.
      
      Reviewed By: pdillinger
      
      Differential Revision: D21260209
      
      Pulled By: riversand963
      
      fbshipit-source-id: c646e2f37c6faf9f0614b68aa0efc818cff96787
      6acbbbf9
    • sdong's avatar
      Remove the support of setting CompressionOptions.parallel_threads from string for now (#6782) · 6504ae0c
      sdong 创作于
      Summary:
      The current way of implementing CompressionOptions.parallel_threads introduces a format change. We plan to change CompressionOptions's serailization format to a new JSON-like format, which would be another format change. We would like to consolidate the two format changes into one, rather than making some users to change twice. Hold CompressionOptions.parallel_threads from being supported by option string for now. Will add it back after the general CompressionOptions's format change.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6782
      
      Test Plan: Run all existing tests.
      
      Reviewed By: zhichao-cao
      
      Differential Revision: D21338614
      
      fbshipit-source-id: bca2dac3cb37d4e6e64b52cbbe8ea749cd848685
      6504ae0c
    • Cheng Chang's avatar
      Make users explicitly be aware of prepare before commit (#6775) · ef0c3eda
      Cheng Chang 创作于
      Summary:
      In current commit protocol of pessimistic transaction, if the transaction is not prepared before commit, the commit protocol implicitly assumes that the user wants to commit without prepare.
      
      This PR adds TransactionOptions::skip_prepare, the default value is `true` because if set to `false`, all existing users who commit without prepare need to update their code to set skip_prepare to true. Although this does not force the user to explicitly express their intention of skip_prepare, it at least lets the user be aware of the assumption of being able to commit without prepare.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6775
      
      Test Plan: added a new unit test TransactionTest::CommitWithoutPrepare
      
      Reviewed By: lth
      
      Differential Revision: D21313270
      
      Pulled By: cheng-chang
      
      fbshipit-source-id: 3d95b7c9b2d6cdddc09bdd66c561bc4fae8c3251
      ef0c3eda
    • sdong's avatar
      Disallow BlockBasedTableBuilder to set status from non-OK (#6776) · 079e50d2
      sdong 创作于
      Summary:
      There is no systematic mechanism to prevent BlockBasedTableBuilder's status to be set from non-OK to OK. Adding a mechanism to force this will help us prevent failures in the future.
      
      The solution is to only make it possible to set the status code if the status code to set is not OK.
      
      Since the status code passed to CompressAndVerifyBlock() is changed, a mini refactoring is done too so that the output arguments are changed from reference to pointers, based on Google C++ Style.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6776
      
      Test Plan: Run all existing test.
      
      Reviewed By: pdillinger
      
      Differential Revision: D21314382
      
      fbshipit-source-id: 27000c10f1e4c121661e026548d6882066409375
      079e50d2
    • sdong's avatar
      Flag CompressionOptions::parallel_threads to be experimental (#6781) · 6277e280
      sdong 创作于
      Summary:
      The feature of CompressionOptions::parallel_threads is still not yet mature. Mention it to be experimental in the comments for now.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6781
      
      Reviewed By: pdillinger
      
      Differential Revision: D21330678
      
      fbshipit-source-id: d7dd7d099fb002a5c6a5d8da689ce5ee08a9eb13
      6277e280
    • anand76's avatar
      Pass a timeout to FileSystem for random reads (#6751) · ab13d43e
      anand76 创作于
      Summary:
      Calculate ```IOOptions::timeout``` using ```ReadOptions::deadline``` and pass it to ```FileSystem::Read/FileSystem::MultiRead```. This allows us to impose a tighter bound on the time taken by Get/MultiGet on FileSystem/Envs that support IO timeouts. Even on those that don't support, check in ```RandomAccessFileReader::Read``` and ```MultiRead``` and return ```Status::TimedOut()``` if the deadline is exceeded.
      
      For now, TableReader creation, which might do file opens and reads, are not covered. It will be implemented in another PR.
      
      Tests:
      Update existing unit tests to verify the correct timeout value is being passed
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6751
      
      Reviewed By: riversand963
      
      Differential Revision: D21285631
      
      Pulled By: anand1976
      
      fbshipit-source-id: d89af843e5a91ece866e87aa29438b52a65a8567
      ab13d43e
    • Peter Dillinger's avatar
      Fix assertion that can fail on sst corruption (#6780) · eecd8fba
      Peter Dillinger 创作于
      Summary:
      An assertion that a char == a CompressionType (unsigned char)
      originally cast from a char can fail if the original value is negative,
      due to numeric promotion.  The assertion should pass even if the value
      is invalid CompressionType, because the callee
      UncompressBlockContentsForCompressionType checks for that and reports
      status appropriately.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6780
      
      Test Plan:
      Temporarily change kZSTD = 0x88 and see tests fail. Make this
      change (in addition), and tests pass.
      
      Reviewed By: siying
      
      Differential Revision: D21328498
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 61caf8d815581ce49261ecb7ab0f396e9ac4bb92
      eecd8fba
    • Levi Tamasi's avatar
      Keep track of obsolete blob files in VersionSet (#6755) · fe238e54
      Levi Tamasi 创作于
      Summary:
      The patch adds logic to keep track of obsolete blob files. A blob file becomes
      obsolete when the last `shared_ptr` that points to the corresponding
      `SharedBlobFileMetaData` object goes away, which, in turn, happens when the
      last `Version` that contains the blob file is destroyed. No longer needed blob
      files are added to the obsolete list in `VersionSet` using a custom deleter to
      avoid unnecessary coupling between `SharedBlobFileMetaData` and `VersionSet`.
      Obsolete blob files are returned by `VersionSet::GetObsoleteFiles` and stored
      in `JobContext`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6755
      
      Test Plan: `make check`
      
      Reviewed By: riversand963
      
      Differential Revision: D21233155
      
      Pulled By: ltamasi
      
      fbshipit-source-id: 47757e06fdc0127f27ed57f51abd27893d9a7b7a
      fe238e54
    • Adam Retter's avatar
      Add Slack forum to README (#6773) · cf342464
      Adam Retter 创作于
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6773
      
      Reviewed By: siying
      
      Differential Revision: D21310229
      
      Pulled By: pdillinger
      
      fbshipit-source-id: c0d52d0c51121d307d7d5c1374abc7bf78b0c4cf
      cf342464
    • Ziyue Yang's avatar
      Add an option for parallel compression in for db_stress (#6722) · e619a20e
      Ziyue Yang 创作于
      Summary:
      This commit adds an `compression_parallel_threads` option in
      db_stress. It also fixes the naming of parallel compression
      option in db_bench to keep it aligned with others.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6722
      
      Reviewed By: pdillinger
      
      Differential Revision: D21091385
      
      fbshipit-source-id: c9ba8c4e5cc327ff9e6094a6dc6a15fcff70f100
      e619a20e
  11. 30 4月, 2020 4 次提交
    • Zhichao Cao's avatar
      Fix potential size_t overflow in import_column_family (#6762) · 8c694025
      Zhichao Cao 创作于
      Summary:
      The issue is reported in https://github.com/facebook/rocksdb/issues/6753 . size_t is unsigned and if sorted_file.size() is 0, the end condition of i will be extremely large, cause segment fault in sorted_files[i] and sorted_files[i+1]. Added condition to fix it.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6762
      
      Test Plan: make asan_check
      
      Reviewed By: pdillinger
      
      Differential Revision: D21323063
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: 56ce59201949ed319448228553202b8642c2cc3a
      8c694025
    • anand76's avatar
      Fix a couple of bugs in FaultInjectionTestFS (#6777) · b938e604
      anand76 创作于
      Summary:
      Fix the following cases that can cause false alarms in db_stress when read fault injection is
       enabled -
      1. Turn off corruption/truncation when direct IO is enabled. Since the actual IO size is larger than block size due to alignment requirements, the corruption may not result in a detectable error.
      2. Handle the case when the randomly generated string to overwrite the original block is identical to the original.
      
      Tests:
      Run db_stress w/ and wo/ direct IO and fault injection turned on
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6777
      
      Reviewed By: zhichao-cao
      
      Differential Revision: D21316734
      
      Pulled By: anand1976
      
      fbshipit-source-id: bf0e6468043063ca81ff877d4bf71d3f296c77aa
      b938e604
    • Peter Dillinger's avatar
      Fix bug in format-diff.sh (#6772) · 28fe8e46
      Peter Dillinger 创作于
      Summary:
      Nasty bug in which more/different changes would be applied than
      those shown to user
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6772
      
      Test Plan: manual
      
      Reviewed By: siying
      
      Differential Revision: D21304604
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 7e20740e513c9c300d1522511290a025b35abedc
      28fe8e46
    • Derrick Pallas's avatar
      Fix FilterBench when RTTI=0 (#6732) · 52723054
      Derrick Pallas 创作于
      
      
      Summary:
      The dynamic_cast in the filter benchmark causes release mode to fail due to
      no-rtti.  Replace with static_cast_with_check.
      Signed-off-by: default avatarDerrick Pallas <derrick@pallas.us>
      
      Addition by peterd: Remove unnecessary 2nd template arg on all static_cast_with_check
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6732
      
      Reviewed By: ltamasi
      
      Differential Revision: D21304260
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 6e8eb437c4ca5a16dbbfa4053d67c4ad55f1608c
      52723054
  12. 29 4月, 2020 5 次提交
    • Peter Dillinger's avatar
      Fix LITE build (#6770) · 8086e5e2
      Peter Dillinger 创作于
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6770
      
      Test Plan: make LITE=1 check
      
      Reviewed By: ajkr
      
      Differential Revision: D21296261
      
      Pulled By: pdillinger
      
      fbshipit-source-id: b6075cc13a6d6db48617b7e0e9ebeea9364dfd9f
      8086e5e2
    • anand76's avatar
      Fix a valgrind failure due to DBBasicTestMultiGetDeadline (#6756) · 335ea73e
      anand76 创作于
      Summary:
      Fix a valgrind failure.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6756
      
      Test Plan: valgrind_test
      
      Reviewed By: pdillinger
      
      Differential Revision: D21284660
      
      Pulled By: anand1976
      
      fbshipit-source-id: 39bf1bd130b6adb585ddbf2f9aa2f53dbf666f80
      335ea73e
    • mrambacher's avatar
      Add Functions to OptionTypeInfo (#6422) · 618bf638
      mrambacher 创作于
      Summary:
      Added functions for parsing, serializing, and comparing elements to OptionTypeInfo.  These functions allow all of the special cases that could not be handled directly in the map of OptionTypeInfo to be moved into the map.  Using these functions, every type can be handled via the map rather than special cased.
      
      By adding these functions, the code for handling options can become more standardized (fewer special cases) and (eventually) handled completely by common classes.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6422
      
      Test Plan: pass make check
      
      Reviewed By: siying
      
      Differential Revision: D21269005
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: 9ba71c721a38ebf9ee88259d60bd81b3282b9077
      618bf638
    • Peter Dillinger's avatar
      Clarifying comments in db.h (#6768) · b810e62b
      Peter Dillinger 创作于
      Summary:
      And fix a confusingly worded log message
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6768
      
      Reviewed By: anand1976
      
      Differential Revision: D21284527
      
      Pulled By: pdillinger
      
      fbshipit-source-id: f03c1422c229a901c3a65e524740452349626164
      b810e62b
    • Peter Dillinger's avatar
      Basic MultiGet support for partitioned filters (#6757) · bae6f586
      Peter Dillinger 创作于
      Summary:
      In MultiGet, access each applicable filter partition only once
      per batch, rather than for each applicable key. Also,
      
      * Fix Bloom stats for MultiGet
      * Fix/refactor MultiGetContext::Range::KeysLeft, including
      * Add efficient BitsSetToOne implementation
      * Assert that MultiGetContext::Range does not go beyond shift range
      
      Performance test: Generate db:
      
          $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 -partition_index_and_filters=true
          ...
      
      Before (middle performing run of three; note some missing Bloom stats):
      
          $ ./db_bench --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget'
          multireadrandom :      26.403 micros/op 597517 ops/sec; (548427 of 671968 found)
          rocksdb.block.cache.filter.hit COUNT : 83443275
          rocksdb.bloom.filter.useful COUNT : 0
          rocksdb.bloom.filter.full.positive COUNT : 0
          rocksdb.bloom.filter.full.true.positive COUNT : 7931450
          rocksdb.number.multiget.get COUNT : 385984
          rocksdb.number.multiget.keys.read COUNT : 12351488
          rocksdb.number.multiget.bytes.read COUNT : 793145000
          rocksdb.number.multiget.keys.found COUNT : 7931450
      
      After (middle performing run of three):
      
          $ ./db_bench_new --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget'
          multireadrandom :      21.024 micros/op 752963 ops/sec; (705188 of 863968 found)
          rocksdb.block.cache.filter.hit COUNT : 49856682
          rocksdb.bloom.filter.useful COUNT : 45684579
          rocksdb.bloom.filter.full.positive COUNT : 10395458
          rocksdb.bloom.filter.full.true.positive COUNT : 9908456
          rocksdb.number.multiget.get COUNT : 481984
          rocksdb.number.multiget.keys.read COUNT : 15423488
          rocksdb.number.multiget.bytes.read COUNT : 990845600
          rocksdb.number.multiget.keys.found COUNT : 9908456
      
      So that's about 25% higher throughput even for random keys
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6757
      
      Test Plan: unit test included
      
      Reviewed By: anand1976
      
      Differential Revision: D21243256
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 5644a1468d9e8c8575be02f4e04bc5d62dbbb57f
      bae6f586