1. 13 Dec, 2019 9 commits
  2. 12 Dec, 2019 7 commits
  3. 11 Dec, 2019 9 commits
    • Yanqin Jin's avatar
      Add SyncWAL to db_stress (#6149) · 383f5071
      Yanqin Jin authored
      Summary:
      Add SyncWAL to db_stress. Specify with `-sync_wal_one_in=N` so that it will be
      called once every N operations on average.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6149
      
      Test Plan:
      ```
      $make db_stress
      $./db_stress -sync_wal_one_in=100 -ops_per_thread=100000
      ```
      
      Differential Revision: D18922529
      
      Pulled By: riversand963
      
      fbshipit-source-id: 4c0b8cb8fa21852722cffd957deddf688f12ea56
      383f5071
    • sdong's avatar
      db_stress: sometimes call CancelAllBackgroundWork() and Close() before closing DB (#6141) · 7a99162a
      sdong authored
      Summary:
      CancelAllBackgroundWork() and Close() are frequently used features but we don't cover it in stress test. Simply execute them before closing the DB with 1/2 chance.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6141
      
      Test Plan: Run "db_stress".
      
      Differential Revision: D18900861
      
      fbshipit-source-id: 49b46ccfae120d0f9de3e0543b82fb6d715949d0
      7a99162a
    • Adam Retter's avatar
      Add Visual Studio 2015 to AppVeyor (#5446) · 984b6e71
      Adam Retter authored
      Summary:
      This is required to compile on Windows with Visual Studio 2015, which is used for creating the RocksJava releases.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5446
      
      Differential Revision: D18924811
      
      fbshipit-source-id: a183a62e79a2af5aaf59cd08235458a172fe7dcb
      984b6e71
    • Peter Dillinger's avatar
      Add PauseBackgroundWork() to db_stress (#6148) · a6538571
      Peter Dillinger authored
      Summary:
      Worker thread will occasionally call PauseBackgroundWork(),
      briefly sleep (to avoid stalling itself) and then call
      ContinueBackgroundWork().
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6148
      
      Test Plan:
      some running of 'make blackbox_crash_test' with temporary
      printf output to confirm code occasionally reached.
      
      Differential Revision: D18913886
      
      Pulled By: pdillinger
      
      fbshipit-source-id: ae9356a803390929f3165dfb6a00194692ba92be
      a6538571
    • Adam Simpkins's avatar
      Add an option to the CMake build to disable building shared libraries (#6122) · 2bb5fc12
      Adam Simpkins authored
      Summary:
      Add an option to explicitly disable building shared versions of the
      RocksDB libraries.  The shared libraries cannot be built in cases where
      some dependencies are only available as static libraries.  This allows
      still building RocksDB in these situations.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6122
      
      Differential Revision: D18920740
      
      fbshipit-source-id: d24f66d93c68a1e65635e6e0b663bae62c903bca
      2bb5fc12
    • Yanqin Jin's avatar
      Use Env::GetChildren() instead of readdir (#6139) · 2b060c14
      Yanqin Jin authored
      Summary:
      For more portability, switch from readdir to Env::GetChildren() in ldb's
      manifest_dump subcommand.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6139
      
      Test Plan:
      ```
      $make check
      ```
      Manually check ldb command.
      
      Differential Revision: D18898197
      
      Pulled By: riversand963
      
      fbshipit-source-id: 92afca379e9fbe78ab70b2eb40d127daad8df5e2
      2b060c14
    • sdong's avatar
      db_stress: sometimes validate compact range data (#6140) · 14c38bac
      sdong authored
      Summary:
      Right now, in db_stress, compact range is simply executed without any immediate data validation. Add a simply validation which compares hash for all keys within the compact range to stay the same against the same snapshot before and after the compaction.
      
      Also, randomly tune most knobs of CompactRangeOptions.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6140
      
      Test Plan: Run db_stress with "--compact_range_one_in=2000 --compact_range_width=100000000" for a while. Manually ingest some hacky code and observe the error path.
      
      Differential Revision: D18900230
      
      fbshipit-source-id: d96e75bc8c38dd5ec702571ffe7cf5f4ea93ee10
      14c38bac
    • Jermy Li's avatar
      Fix compile error "folly/xx.h file not found" on Mac OS (#6145) · 1dd3194f
      Jermy Li authored
      Summary:
      Error message when running `make` on Mac OS with master branch (v6.6.0):
      ```
      $ make
      $DEBUG_LEVEL is 1
      Makefile:168: Warning: Compiling in debug mode. Don't use the resulting binary in production
      third-party/folly/folly/synchronization/WaitOptions.cpp:6:10: fatal error: 'folly/synchronization/WaitOptions.h' file not found
      #include <folly/synchronization/WaitOptions.h>
               ^
      1 error generated.
      third-party/folly/folly/synchronization/ParkingLot.cpp:6:10: fatal error: 'folly/synchronization/ParkingLot.h' file not found
      #include <folly/synchronization/ParkingLot.h>
               ^
      1 error generated.
      third-party/folly/folly/synchronization/DistributedMutex.cpp:6:10: fatal error: 'folly/synchronization/DistributedMutex.h' file not found
      #include <folly/synchronization/DistributedMutex.h>
               ^
      1 error generated.
      third-party/folly/folly/synchronization/AtomicNotification.cpp:6:10: fatal error: 'folly/synchronization/AtomicNotification.h' file not found
      #include <folly/synchronization/AtomicNotification.h>
               ^
      1 error generated.
      third-party/folly/folly/detail/Futex.cpp:6:10: fatal error: 'folly/detail/Futex.h' file not found
      #include <folly/detail/Futex.h>
               ^
      1 error generated.
        GEN      util/build_version.cc
      $DEBUG_LEVEL is 1
      Makefile:168: Warning: Compiling in debug mode. Don't use the resulting binary in production
      third-party/folly/folly/synchronization/WaitOptions.cpp:6:10: fatal error: 'folly/synchronization/WaitOptions.h' file not found
      #include <folly/synchronization/WaitOptions.h>
               ^
      1 error generated.
      third-party/folly/folly/synchronization/ParkingLot.cpp:6:10: fatal error: 'folly/synchronization/ParkingLot.h' file not found
      #include <folly/synchronization/ParkingLot.h>
               ^
      1 error generated.
      third-party/folly/folly/synchronization/DistributedMutex.cpp:6:10: fatal error: 'folly/synchronization/DistributedMutex.h' file not found
      #include <folly/synchronization/DistributedMutex.h>
               ^
      1 error generated.
      third-party/folly/folly/synchronization/AtomicNotification.cpp:6:10: fatal error: 'folly/synchronization/AtomicNotification.h' file not found
      #include <folly/synchronization/AtomicNotification.h>
               ^
      1 error generated.
      third-party/folly/folly/detail/Futex.cpp:6:10: fatal error: 'folly/detail/Futex.h' file not found
      #include <folly/detail/Futex.h>
      ```
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6145
      
      Differential Revision: D18910812
      
      fbshipit-source-id: 5a4475466c2d0601657831a0b48d34316b2f0816
      1dd3194f
    • Peter Dillinger's avatar
      Vary bloom_bits in db_crashtest (#6103) · 6380df5e
      Peter Dillinger authored
      Summary:
      Especially with non-integral bits/key now supported,
      db_crashtest should vary the bloom_bits configuration. The probabilities
      look like this:
      
      1/2 chance of a uniform int from 0 to 19. This includes overall 1/40
      chance of 0 which disables the bloom filter.
      
      1/2 chance of a float from a lognormal distribution with a median of 10.
      This always produces positive values but with a decent chance of < 1
      (overall ~1/40) or > 100 (overall ~1/40), the enforced/coerced
      implementation limits.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6103
      
      Test Plan:
      start 'make blackbox_crash_test' several times and look at
      configuration output
      
      Differential Revision: D18734877
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 4a38cb057d3b3fc1327f93199f65b9a9ffbd7316
      6380df5e
  4. 10 Dec, 2019 7 commits
    • sdong's avatar
      Apply formatter to some recent commits (#6138) · a68dff5c
      sdong authored
      Summary:
      Formatter somehow complains some recent lines changed. Apply them to make the formatter happy.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6138
      
      Test Plan: See CI passes.
      
      Differential Revision: D18895950
      
      fbshipit-source-id: 7d1696cf3e3a682bc10a30cdca748a23c6565255
      a68dff5c
    • sdong's avatar
      db_stress: Some code style improvements (#6137) · a960287d
      sdong authored
      Summary:
      Two changes:
      1. Prevent static variables in a header file
      2. Add "override" keyword when virtual functions are overridden.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6137
      
      Test Plan: Build db_stress with or without LITE.
      
      Differential Revision: D18892007
      
      fbshipit-source-id: 295356427a34473b23ed36d6ed4ef3ae35a32db0
      a960287d
    • Peter Dillinger's avatar
      Fix & test rocksdb_filterpolicy_create_bloom_full (#6132) · e43d2c44
      Peter Dillinger authored
      Summary:
      Add overrides needed in FilterPolicy wrapper to fix
      rocksdb_filterpolicy_create_bloom_full (see issue https://github.com/facebook/rocksdb/issues/6129). Re-enabled
      assertion in BloomFilterPolicy::CreateFilter that was being violated.
      Expanded c_test to identify Bloom filter implementations by FP counts.
      (Without the fix, updated test will trigger assertion and fail otherwise
      without the assertion.)
      
      Fixes https://github.com/facebook/rocksdb/issues/6129
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6132
      
      Test Plan: updated c_test, also run under valgrind.
      
      Differential Revision: D18864911
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 08e81d7b5368b08e501cd402ef5583f2650c19fa
      e43d2c44
    • sdong's avatar
      Fix thread_local_test failure caused by recent io_uring change (#6136) · 3c347821
      sdong authored
      Summary:
      thread_local_test now fails because it asserts no thread local instance is created when the test started. However, right now a thread local instance might be created when creating PosixEnv as a static variable. Fix the test by relaxing the assumption of starting from 0.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6136
      
      Test Plan: Find an environment where the test fails, and see it passes with the fix applied.
      
      Differential Revision: D18889224
      
      fbshipit-source-id: 7946f3bfea81d236f7bb1554076696705b211b92
      3c347821
    • Ziyue Yang's avatar
      Fix wrong ExtractUserKey usage in BlockBasedTableBuilder::EnterUnbuff… (#6100) · 7e2f8319
      Ziyue Yang authored
      Summary:
      BlockBasedTableBuilder uses ExtractUserKey in EnterUnbuffered. This would
      cause index filter building error, since user-provided timestamp is supported
      by ExtractUserKeyAndStripTimestamp, and it's used in Add. This commit changes
      ExtractUserKey to ExtractUserKeyAndStripTimestamp.
      
      A test case is also added by modifying DBBasicTestWithTimestampWithParam_
      PutAndGet test in db_basic_test to cover ExtractUserKeyAndStripTimestamp usage
      in both kBuffered and kUnbuffered state of BlockBasedTableBuilder.
      
      Before the ExtractUserKeyAndStripTimstamp fix:
      
      ```
      $ ./db_basic_test --gtest_filter="*PutAndGet*"
      Note: Google Test filter = *PutAndGet*
      [==========] Running 2 tests from 1 test case.
      [----------] Global test environment set-up.
      [----------] 2 tests from Timestamp/DBBasicTestWithTimestampWithParam
      [ RUN      ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/0
      db/db_basic_test.cc:2109: Failure
      db_->Get(ropts, cfh, "key" + std::to_string(j), &value)
      NotFound:
      db/db_basic_test.cc:2109: Failure
      db_->Get(ropts, cfh, "key" + std::to_string(j), &value)
      NotFound:
      db/db_basic_test.cc:2109: Failure
      db_->Get(ropts, cfh, "key" + std::to_string(j), &value)
      NotFound:
      db/db_basic_test.cc:2109: Failure
      db_->Get(ropts, cfh, "key" + std::to_string(j), &value)
      NotFound:
      db/db_basic_test.cc:2109: Failure
      db_->Get(ropts, cfh, "key" + std::to_string(j), &value)
      NotFound:
      [  FAILED  ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/0, where GetParam() = false (1177 ms)
      [ RUN      ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/1
      [       OK ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/1 (1056 ms)
      [----------] 2 tests from Timestamp/DBBasicTestWithTimestampWithParam (2233 ms total)
      
      [----------] Global test environment tear-down
      [==========] 2 tests from 1 test case ran. (2233 ms total)
      [  PASSED  ] 1 test.
      [  FAILED  ] 1 test, listed below:
      [  FAILED  ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/0, where GetParam() = false
      
       1 FAILED TEST
      ```
      
      After the ExtractUserKeyAndStripTimstamp fix:
      
      ```
      $ ./db_basic_test --gtest_filter="*PutAndGet*"
      Note: Google Test filter = *PutAndGet*
      [==========] Running 2 tests from 1 test case.
      [----------] Global test environment set-up.
      [----------] 2 tests from Timestamp/DBBasicTestWithTimestampWithParam
      [ RUN      ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/0
      [       OK ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/0 (1417 ms)
      [ RUN      ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/1
      [       OK ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/1 (1041 ms)
      [----------] 2 tests from Timestamp/DBBasicTestWithTimestampWithParam (2458 ms total)
      
      [----------] Global test environment tear-down
      [==========] 2 tests from 1 test case ran. (2458 ms total)
      [  PASSED  ] 2 tests.
      ```
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6100
      
      Differential Revision: D18769654
      
      Pulled By: riversand963
      
      fbshipit-source-id: 76c2cf2c9a5e0d85db95d98e812e6af0c2a15c6b
      7e2f8319
    • sdong's avatar
      Fix an asan warning caused by the recent io_uring change (#6135) · d1ae2c3f
      sdong authored
      Summary:
      ASAN reports:
      
      internal_repo_rocksdb/repo:db_test - MultiThreaded/MultiThreadedDBTest.MultiThreaded/43: fatal
      ==2692739==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6130000500ca at pc 0x0000006be780 bp 0x7efef85ccd20 sp 0x7efef85cc4d0
      [CONTEXT] === How to use this, how to get the raw stack trace, and more: fburl.com/ASAN ===
      [CONTEXT] READ of size 331 at 0x6130000500ca thread T195
      [CONTEXT]      #0 db_test_bin+0x6be77f                     __interceptor_strlen.part.35
      [CONTEXT]      https://github.com/facebook/rocksdb/issues/1 internal_repo_rocksdb/repo/include/rocksdb/slice.h:55 rocksdb::Slice::Slice(char const*)
      [CONTEXT]      https://github.com/facebook/rocksdb/issues/2 internal_repo_rocksdb/repo/env/io_posix.cc:522 rocksdb::PosixRandomAccessFile::MultiRead(rocksdb::ReadRequest*, unsigned long)
      
      I looked at env/io_posix.cc:522 but don't see a reason why the line needs to be there at all, because it is not used before overwritten. So it must be a line that is put there as a bug. Remove it.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6135
      
      Test Plan: Rerun the same test which passes after the fix. Run all the tests and make sure they all pass.
      
      Differential Revision: D18880251
      
      fbshipit-source-id: 3b84ac6a05b67b529c4202e0ceb4c047460f44f2
      d1ae2c3f
    • Peter Dillinger's avatar
      Use SpecialSkipListFactory in RecalculateScoreAfterPicking (#6125) · 3a6d9436
      Peter Dillinger authored
      Summary:
      Test DBTestUniversalCompaction.RecalculateScoreAfterPicking was
      flaky on ARM, so it now uses SpecialSkipListFactory (like other tests)
      for predictable memtable flushes.
      
      Fixes https://github.com/facebook/rocksdb/issues/5736
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6125
      
      Test Plan:
      while ./db_universal_compaction_test; do :; done # for a
      while on ARM and on Intel (both Linux)
      
      Differential Revision: D18864821
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 2f3ca0ea66ce420dcd6d41b0ec12377112a5a79f
      3a6d9436
  5. 09 Dec, 2019 2 commits
    • sdong's avatar
      Break db_stress_tool.cc to a list of source files (#6134) · 7d79b326
      sdong authored
      Summary:
      db_stress_tool.cc now is a giant file. In order to main it easier to improve and maintain, break it down to multiple source files.
      Most classes are turned into their own files. Separate .h and .cc files are created for gflag definiations. Another .h and .cc files are created for some common functions. Some test execution logic that is only loosely related to class StressTest is moved to db_stress_driver.h and db_stress_driver.cc. All the files are located under db_stress_tool/. The directory name is created as such because if we end it with either stress or test, .gitignore will ignore any file under it and makes it prone to issues in developements.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6134
      
      Test Plan: Build under GCC7 with and without LITE on using GNU Make. Build with GCC 4.8. Build with cmake with -DWITH_TOOL=1
      
      Differential Revision: D18876064
      
      fbshipit-source-id: b25d0a7451840f31ac0f5ebb0068785f783fdf7d
      7d79b326
    • suzanwen's avatar
      Isolate building db_bench from tests with `WITH_BENCHMARK_TOOLS` option. (#6098) · bac38c99
      suzanwen authored
      Summary:
      Isolate `db_bench` from building tests, out of respect for the related comments.
      Let building tests yields to `WITH_TEST=ON` AND `CMAKE_BUILD_TYPE=Debug` both,
      and building `db_bench` yields to `WITH_BENCHMARK_TOOLS=ON`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6098
      
      Test Plan: cmake -DCMAKE_BUILD_TYPE=Debug/Release -DWITH_TESTS=ON/OFF -DWITH_BENCHMARK_TOOLS=ON/OFF -DWITH_TOOLS=ON/OFF && make
      
      Differential Revision: D18856891
      
      Pulled By: riversand963
      
      fbshipit-source-id: addbee8ad6abefb877843a313b4630cfab3ce4f0
      bac38c99
  6. 08 Dec, 2019 1 commit
    • sdong's avatar
      PosixRandomAccessFile::MultiRead() to use I/O uring if supported (#5881) · e3a82bb9
      sdong authored
      Summary:
      Right now, PosixRandomAccessFile::MultiRead() executes read requests in parallel. In this PR, it leverages I/O Uring library to run it in parallel, even when page cache is enabled. This function will fall back if the kernel version doesn't support it.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5881
      
      Test Plan: Run the unit test on a kernel version supporting it and make sure all tests pass, and run a unit test on kernel version supporting it and see it pass. Before merging, will also run stress test and see it passes.
      
      Differential Revision: D17742266
      
      fbshipit-source-id: e05699c925ac04fdb42379456a4e23e4ebcb803a
      e3a82bb9
  7. 07 Dec, 2019 1 commit
  8. 06 Dec, 2019 3 commits
  9. 04 Dec, 2019 1 commit
    • Yanqin Jin's avatar
      Make folly-related targets comply with verbosity (#6120) · 4edb4284
      Yanqin Jin authored
      Summary:
      Before this fix, `make all` will emit full compilation command when building
      object files in the third-party/folly directory even if default verbosity is
      0 (AM_DEFAULT_VERBOSITY).
      
      Test Plan (devserver):
      ```
      $make all | tee build.log
      $make check
      ```
      Check build.log to verify.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6120
      
      Differential Revision: D18795621
      
      Pulled By: riversand963
      
      fbshipit-source-id: 04641a8359cd4fd55034e6e797ed85de29ee2fe2
      4edb4284