- 06 Jun, 2020 2 commits
-
-
anand76 authored
-
anand76 authored
Summary: The ```for``` loop in ```VerifyChecksumInBlocks``` only checks ```index_iter->Valid()``` which could be ```false``` either due to reaching the end of the index or, in case of partitioned index, it could be due to a checksum mismatch error when reading a 2nd level index block. Instead of throwing away the index iterator status, we need to return any errors back to the caller. Tests: Add a test in block_based_table_reader_test.cc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6909 Reviewed By: pdillinger Differential Revision: D21833922 Pulled By: anand1976 fbshipit-source-id: bc778ebf1121dbbdd768689de5183f07a9f0beae
-
- 04 Jun, 2020 1 commit
-
-
Peter Dillinger authored
Partial back-port of #6905
-
- 28 May, 2020 1 commit
-
-
Yanqin Jin authored
-
- 19 May, 2020 1 commit
-
-
Yanqin Jin authored
Summary: In buck build with opt mode, target should not include rocksdb_test_lib. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6847 Test Plan: Watch for internal cont build. Reviewed By: ajkr Differential Revision: D21586803 Pulled By: riversand963 fbshipit-source-id: 76d253c18d16fac6cab86a8c3f6b471ad5b6efb3
-
- 14 May, 2020 2 commits
-
-
Yanqin Jin authored
Summary: Before this PR, extra deps passed in from cmd line to buckifier will be parsed and used to populate a dict. Using this dict and printing to TARGETS file will lead to printing u'', disallowed by build tools. This PR removes the u''. Test Plan (local dev server): ``` python buckifier/buckify_rocksdb.py '{"fake": {"extra_deps": [":test_dep", "//fake/module:mock1"], "extra_compiler_flags": ["-Os", "-DROCKSDB_LITE"]}}' ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6841 Reviewed By: siying Differential Revision: D21538155 Pulled By: riversand963 fbshipit-source-id: 09403668a4aa1a15bad7dac229c2bc8ce8ee1349
-
anand76 authored
-
- 12 May, 2020 1 commit
-
-
sdong authored
Summary: When using ldb, users cannot turn on force consistency check in most commands, while they cannot use checksonsistnecy with --try_load_options. The change fixes both by: 1. checkconsistency now calls OpenDB() so that it gets all the options loading and sanitized options logic 2. use options.check_consistency_checks = true by default, and add a --disable_consistency_checks to turn it off. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6802 Test Plan: Add a new unit test. Some manual tests with corrupted DBs. Reviewed By: pdillinger Differential Revision: D21388051 fbshipit-source-id: 8d122732d391b426e3982a1c3232a8e3763ffad0
-
- 09 May, 2020 7 commits
-
-
Yanqin Jin authored
Summary: 1. Update column_family_memtables_ to point to latest column_family_set in version_set after recovery. 2. Normalize file paths passed by application so that directories end with '/' or '\\'. 3. In addition to missing files, corrupted files are also ignored in best-efforts recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6824 Test Plan: COMPILE_WITH_ASAN=1 make check Reviewed By: anand1976 Differential Revision: D21463905 Pulled By: riversand963 fbshipit-source-id: c48db8843cc93c8c1c7139c474b64e6f775307d2
-
Andrew Kryczka authored
-
anand76 authored
Summary: https://github.com/facebook/rocksdb/issues/6389 replaced the #include of options.h in table.h with forward declarations, which is causing some build failures in RocksDB users in 6.10. Remove the forward declarations and #include options.h as recommended by the style guide - https://google.github.io/styleguide/cppguide.html#Forward_Declarations Pull Request resolved: https://github.com/facebook/rocksdb/pull/6823 Reviewed By: riversand963 Differential Revision: D21464078 Pulled By: anand1976 fbshipit-source-id: 6033ee2544d279690f57bb0db91bc83816cee11d
-
anand76 authored
Summary: Update HISTORY.md and version.h to 6.10. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6797 Reviewed By: zhichao-cao Differential Revision: D21371390 Pulled By: anand1976 fbshipit-source-id: 6017bca24fc5d12076d1ddaec7783c9b85712d42
-
Peter Dillinger authored
Summary: This very old test code bug was causing a new valgrind failure in MultiGetDeadlineExceeded Also fix hang in MultiGetDeadlineExceeded by unifying with some logic from another test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6803 Test Plan: run that unit test under valgrind, make check Reviewed By: siying Differential Revision: D21388470 Pulled By: pdillinger fbshipit-source-id: 0ce99d6d5eb8cd3195b17406892c8c5cff5fa5dd
-
Yanqin Jin authored
Summary: With consistency check enabled, VersionBuilder::SaveTo() may return error once corruption is detected while building versions. We should handle these errors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6801 Test Plan: make check Reviewed By: siying Differential Revision: D21385045 Pulled By: riversand963 fbshipit-source-id: 98f6424e2a4699b62befa21e9fe00e70a771118e
-
anand76 authored
Summary: Delete triggered compaction in universal compaction mode was causing a corruption when scheduled in parallel with other compactions. 1. When num_levels = 1, a file marked for compaction may be picked along with all older files in L0, without checking if any of them are already being compaction. This can cause unpredictable results like resurrection of older versions of keys or deleted keys. 2. When num_levels > 1, a delete triggered compaction would not get scheduled if it overlaps with a running regular compaction. However, the reverse is not true. This is due to the fact that in ```UniversalCompactionBuilder::CalculateSortedRuns```, it assumes that entire sorted runs are picked for compaction and only checks the first file in a sorted run to determine conflicts. This is violated by a delete triggered compaction as it works on a subset of a sorted run. Fix the bug for num_levels > 1, and disable the feature for now when num_levels = 1. After disabling this feature, files would still get marked for compaction, but no compaction would get scheduled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6799 Reviewed By: pdillinger Differential Revision: D21431286 Pulled By: anand1976 fbshipit-source-id: ae9f0bdb1d6ae2f10284847db731c23f43af164a
-
- 08 May, 2020 3 commits
-
-
Peter Dillinger authored
Summary: The error is assigning KeyContext::s to NotFound status in a table reader for a "not found in this table" case, which skips searching in later tables, like only a delete should. (The hash search index iterator is the only one that can return status NotFound even if Valid() == false.) This was detected by intermittent failure in MultiThreadedDBTest.MultiThreaded/5, a kHashSearch configuration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6821 Test Plan: modified existing unit test to reproduce problem Reviewed By: anand1976 Differential Revision: D21450469 Pulled By: pdillinger fbshipit-source-id: 7478003684d637dbd491cdac81468041a791be2c
-
sdong authored
Summary: We are swallowing some file consistency checking failures. This is not expected. We are fixing two cases: DB reopen and manifest dump. More places are not fixed and need follow-up. Error from CheckConsistencyForDeletes() is also swallowed, which is not fixed in this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6793 Test Plan: Add a unit test to cover the reopen case. Reviewed By: riversand963 Differential Revision: D21366525 fbshipit-source-id: eb438a322237814e8d5125f916a3c6de97f39ded
-
Andrew Kryczka authored
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6809 Reviewed By: pdillinger Differential Revision: D21411971 Pulled By: ajkr fbshipit-source-id: 900b6b0370b76e9a3e5e03f968e2ac1bbaab73b8
-
- 02 May, 2020 1 commit
-
-
Cheng Chang authored
Summary: When expiration is set in a pessimistic transaction, `txn_state_` is already updated to `AWAITING_PREPARE` in the `if (expiration_time_ > 0)` block, there is no need to update the state in `if (can_prepare)` block again. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6778 Test Plan: make check Reviewed By: lth Differential Revision: D21335319 Pulled By: cheng-chang fbshipit-source-id: 251d634cc7d1a0e86e673a59f0bda8584da5a35f
-
- 01 May, 2020 11 commits
-
-
Zhichao Cao authored
Summary: The multiple CF hash map is not passed to the multi-thread worker. When using multi-thread replay for multiple CFs, it will cause segment fault. Pass the cf_map to the argument. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6787 Test Plan: pass trace replay test. Reviewed By: yhchiang Differential Revision: D21339941 Pulled By: zhichao-cao fbshipit-source-id: 434482b492287e6722c7cd5a706f057c5ec170ce
-
Yanqin Jin authored
Summary: Add Github Action to perform some basic sanity check for PR, inclding the following. 1) Buck TARGETS file. On the one hand, The TARGETS file is used for internal buck, and we do not manually update it. On the other hand, we need to run the buckifier scripts to update TARGETS whenever new files are added, etc. With this Github Action, we make sure that every PR does not forget this step. The GH Action uses a Makefile target called check-buck-targets. Users can manually run `make check-buck-targets` on local machine. 2) Code format We use clang-format-diff.py to format our code. The GH Action in this PR makes sure this step is not skipped. The checking script build_tools/format-diff.sh assumes that `clang-format-diff.py` is executable. On host running GH Action, it is difficult to download `clang-format-diff.py` and make it executable. Therefore, we modified build_tools/format-diff.sh to handle the case in which there is a non-executable clang-format-diff.py file in the top-level rocksdb repo directory. Test Plan (Github and devserver): Watch for Github Action result in the `Checks` tab. On dev server ``` make check-format make check-buck-targets make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6761 Test Plan: Watch for Github Action result in the `Checks` tab. Reviewed By: pdillinger Differential Revision: D21260209 Pulled By: riversand963 fbshipit-source-id: c646e2f37c6faf9f0614b68aa0efc818cff96787
-
sdong authored
Summary: The current way of implementing CompressionOptions.parallel_threads introduces a format change. We plan to change CompressionOptions's serailization format to a new JSON-like format, which would be another format change. We would like to consolidate the two format changes into one, rather than making some users to change twice. Hold CompressionOptions.parallel_threads from being supported by option string for now. Will add it back after the general CompressionOptions's format change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6782 Test Plan: Run all existing tests. Reviewed By: zhichao-cao Differential Revision: D21338614 fbshipit-source-id: bca2dac3cb37d4e6e64b52cbbe8ea749cd848685
-
Cheng Chang authored
Summary: In current commit protocol of pessimistic transaction, if the transaction is not prepared before commit, the commit protocol implicitly assumes that the user wants to commit without prepare. This PR adds TransactionOptions::skip_prepare, the default value is `true` because if set to `false`, all existing users who commit without prepare need to update their code to set skip_prepare to true. Although this does not force the user to explicitly express their intention of skip_prepare, it at least lets the user be aware of the assumption of being able to commit without prepare. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6775 Test Plan: added a new unit test TransactionTest::CommitWithoutPrepare Reviewed By: lth Differential Revision: D21313270 Pulled By: cheng-chang fbshipit-source-id: 3d95b7c9b2d6cdddc09bdd66c561bc4fae8c3251
-
sdong authored
Summary: There is no systematic mechanism to prevent BlockBasedTableBuilder's status to be set from non-OK to OK. Adding a mechanism to force this will help us prevent failures in the future. The solution is to only make it possible to set the status code if the status code to set is not OK. Since the status code passed to CompressAndVerifyBlock() is changed, a mini refactoring is done too so that the output arguments are changed from reference to pointers, based on Google C++ Style. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6776 Test Plan: Run all existing test. Reviewed By: pdillinger Differential Revision: D21314382 fbshipit-source-id: 27000c10f1e4c121661e026548d6882066409375
-
sdong authored
Summary: The feature of CompressionOptions::parallel_threads is still not yet mature. Mention it to be experimental in the comments for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6781 Reviewed By: pdillinger Differential Revision: D21330678 fbshipit-source-id: d7dd7d099fb002a5c6a5d8da689ce5ee08a9eb13
-
anand76 authored
Summary: Calculate ```IOOptions::timeout``` using ```ReadOptions::deadline``` and pass it to ```FileSystem::Read/FileSystem::MultiRead```. This allows us to impose a tighter bound on the time taken by Get/MultiGet on FileSystem/Envs that support IO timeouts. Even on those that don't support, check in ```RandomAccessFileReader::Read``` and ```MultiRead``` and return ```Status::TimedOut()``` if the deadline is exceeded. For now, TableReader creation, which might do file opens and reads, are not covered. It will be implemented in another PR. Tests: Update existing unit tests to verify the correct timeout value is being passed Pull Request resolved: https://github.com/facebook/rocksdb/pull/6751 Reviewed By: riversand963 Differential Revision: D21285631 Pulled By: anand1976 fbshipit-source-id: d89af843e5a91ece866e87aa29438b52a65a8567
-
Peter Dillinger authored
Summary: An assertion that a char == a CompressionType (unsigned char) originally cast from a char can fail if the original value is negative, due to numeric promotion. The assertion should pass even if the value is invalid CompressionType, because the callee UncompressBlockContentsForCompressionType checks for that and reports status appropriately. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6780 Test Plan: Temporarily change kZSTD = 0x88 and see tests fail. Make this change (in addition), and tests pass. Reviewed By: siying Differential Revision: D21328498 Pulled By: pdillinger fbshipit-source-id: 61caf8d815581ce49261ecb7ab0f396e9ac4bb92
-
Levi Tamasi authored
Summary: The patch adds logic to keep track of obsolete blob files. A blob file becomes obsolete when the last `shared_ptr` that points to the corresponding `SharedBlobFileMetaData` object goes away, which, in turn, happens when the last `Version` that contains the blob file is destroyed. No longer needed blob files are added to the obsolete list in `VersionSet` using a custom deleter to avoid unnecessary coupling between `SharedBlobFileMetaData` and `VersionSet`. Obsolete blob files are returned by `VersionSet::GetObsoleteFiles` and stored in `JobContext`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6755 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21233155 Pulled By: ltamasi fbshipit-source-id: 47757e06fdc0127f27ed57f51abd27893d9a7b7a
-
Adam Retter authored
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6773 Reviewed By: siying Differential Revision: D21310229 Pulled By: pdillinger fbshipit-source-id: c0d52d0c51121d307d7d5c1374abc7bf78b0c4cf
-
Ziyue Yang authored
Summary: This commit adds an `compression_parallel_threads` option in db_stress. It also fixes the naming of parallel compression option in db_bench to keep it aligned with others. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6722 Reviewed By: pdillinger Differential Revision: D21091385 fbshipit-source-id: c9ba8c4e5cc327ff9e6094a6dc6a15fcff70f100
-
- 30 Apr, 2020 4 commits
-
-
Zhichao Cao authored
Summary: The issue is reported in https://github.com/facebook/rocksdb/issues/6753 . size_t is unsigned and if sorted_file.size() is 0, the end condition of i will be extremely large, cause segment fault in sorted_files[i] and sorted_files[i+1]. Added condition to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6762 Test Plan: make asan_check Reviewed By: pdillinger Differential Revision: D21323063 Pulled By: zhichao-cao fbshipit-source-id: 56ce59201949ed319448228553202b8642c2cc3a
-
anand76 authored
Summary: Fix the following cases that can cause false alarms in db_stress when read fault injection is enabled - 1. Turn off corruption/truncation when direct IO is enabled. Since the actual IO size is larger than block size due to alignment requirements, the corruption may not result in a detectable error. 2. Handle the case when the randomly generated string to overwrite the original block is identical to the original. Tests: Run db_stress w/ and wo/ direct IO and fault injection turned on Pull Request resolved: https://github.com/facebook/rocksdb/pull/6777 Reviewed By: zhichao-cao Differential Revision: D21316734 Pulled By: anand1976 fbshipit-source-id: bf0e6468043063ca81ff877d4bf71d3f296c77aa
-
Peter Dillinger authored
Summary: Nasty bug in which more/different changes would be applied than those shown to user Pull Request resolved: https://github.com/facebook/rocksdb/pull/6772 Test Plan: manual Reviewed By: siying Differential Revision: D21304604 Pulled By: pdillinger fbshipit-source-id: 7e20740e513c9c300d1522511290a025b35abedc
-
Derrick Pallas authored
Summary: The dynamic_cast in the filter benchmark causes release mode to fail due to no-rtti. Replace with static_cast_with_check. Signed-off-by:
Derrick Pallas <derrick@pallas.us> Addition by peterd: Remove unnecessary 2nd template arg on all static_cast_with_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6732 Reviewed By: ltamasi Differential Revision: D21304260 Pulled By: pdillinger fbshipit-source-id: 6e8eb437c4ca5a16dbbfa4053d67c4ad55f1608c
-
- 29 Apr, 2020 6 commits
-
-
Peter Dillinger authored
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6770 Test Plan: make LITE=1 check Reviewed By: ajkr Differential Revision: D21296261 Pulled By: pdillinger fbshipit-source-id: b6075cc13a6d6db48617b7e0e9ebeea9364dfd9f
-
anand76 authored
Summary: Fix a valgrind failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6756 Test Plan: valgrind_test Reviewed By: pdillinger Differential Revision: D21284660 Pulled By: anand1976 fbshipit-source-id: 39bf1bd130b6adb585ddbf2f9aa2f53dbf666f80
-
mrambacher authored
Summary: Added functions for parsing, serializing, and comparing elements to OptionTypeInfo. These functions allow all of the special cases that could not be handled directly in the map of OptionTypeInfo to be moved into the map. Using these functions, every type can be handled via the map rather than special cased. By adding these functions, the code for handling options can become more standardized (fewer special cases) and (eventually) handled completely by common classes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6422 Test Plan: pass make check Reviewed By: siying Differential Revision: D21269005 Pulled By: zhichao-cao fbshipit-source-id: 9ba71c721a38ebf9ee88259d60bd81b3282b9077
-
Peter Dillinger authored
Summary: And fix a confusingly worded log message Pull Request resolved: https://github.com/facebook/rocksdb/pull/6768 Reviewed By: anand1976 Differential Revision: D21284527 Pulled By: pdillinger fbshipit-source-id: f03c1422c229a901c3a65e524740452349626164
-
Peter Dillinger authored
Summary: In MultiGet, access each applicable filter partition only once per batch, rather than for each applicable key. Also, * Fix Bloom stats for MultiGet * Fix/refactor MultiGetContext::Range::KeysLeft, including * Add efficient BitsSetToOne implementation * Assert that MultiGetContext::Range does not go beyond shift range Performance test: Generate db: $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 -partition_index_and_filters=true ... Before (middle performing run of three; note some missing Bloom stats): $ ./db_bench --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget' multireadrandom : 26.403 micros/op 597517 ops/sec; (548427 of 671968 found) rocksdb.block.cache.filter.hit COUNT : 83443275 rocksdb.bloom.filter.useful COUNT : 0 rocksdb.bloom.filter.full.positive COUNT : 0 rocksdb.bloom.filter.full.true.positive COUNT : 7931450 rocksdb.number.multiget.get COUNT : 385984 rocksdb.number.multiget.keys.read COUNT : 12351488 rocksdb.number.multiget.bytes.read COUNT : 793145000 rocksdb.number.multiget.keys.found COUNT : 7931450 After (middle performing run of three): $ ./db_bench_new --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget' multireadrandom : 21.024 micros/op 752963 ops/sec; (705188 of 863968 found) rocksdb.block.cache.filter.hit COUNT : 49856682 rocksdb.bloom.filter.useful COUNT : 45684579 rocksdb.bloom.filter.full.positive COUNT : 10395458 rocksdb.bloom.filter.full.true.positive COUNT : 9908456 rocksdb.number.multiget.get COUNT : 481984 rocksdb.number.multiget.keys.read COUNT : 15423488 rocksdb.number.multiget.bytes.read COUNT : 990845600 rocksdb.number.multiget.keys.found COUNT : 9908456 So that's about 25% higher throughput even for random keys Pull Request resolved: https://github.com/facebook/rocksdb/pull/6757 Test Plan: unit test included Reviewed By: anand1976 Differential Revision: D21243256 Pulled By: pdillinger fbshipit-source-id: 5644a1468d9e8c8575be02f4e04bc5d62dbbb57f
-
Peter Dillinger authored
Summary: See https://github.com/facebook/rocksdb/issues/6714 and https://github.com/facebook/rocksdb/issues/6703 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6767 Reviewed By: riversand963 Differential Revision: D21283307 Pulled By: pdillinger fbshipit-source-id: 8463bec725669d13846c728ad4b5bde43f9a84f8
-