1. 21 Feb, 2020 2 commits
  2. 20 Feb, 2020 1 commit
  3. 19 Feb, 2020 3 commits
  4. 15 Feb, 2020 4 commits
  5. 14 Feb, 2020 9 commits
    • wolfkdy's avatar
      refine code (#6420) · 29e24434
      wolfkdy authored
      Summary:
      I create a new branch from the branch new upsteram/master and "git merge --squash".
      Maybe it will fix everything.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6420
      
      Differential Revision: D19897152
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: 6575d9e3b23e360f42ee1480b43028b5fcc20136
      29e24434
    • Manuel Ung's avatar
      WriteUnPrepared: Fix assertion during recovery (#6419) · 908b1ee6
      Manuel Ung authored
      Summary:
      During recovery, multiple (un)prepared batches could exist in the same WAL record due to group commit. This breaks an assertion in `MemTableInserter::MarkBeginPrepare`.
      
      To fix, reset unprepared_batch_ to false after `MarkEndPrepare`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6419
      
      Differential Revision: D19896148
      
      Pulled By: lth
      
      fbshipit-source-id: b1a32ef88f775a0881264a18bd1a4a5b8c85eee3
      908b1ee6
    • Manuel Ung's avatar
      WriteUnPrepared: Enable WAL during crash recovery (#6418) · fb571509
      Manuel Ung authored
      Summary:
      Unfortunately, it seems like mysqld reuses xids across machine restarts. When that happens, we could have something like the following happening:
      
      ```
      BEGIN_PREPARE(unprepared) Put(a) END_PREPARE(xid = 1)
      -- crash and recover with Put(a) rolled back as it was not prepared
      BEGIN_PREPARE(prepared) Put(b) END_PREPARE(xid = 1)
      COMMIT(xid = 1)
      -- crash and recover with both a, b
      ```
      
      To solve this, we will have to log the rollback batch into the WAL during recovery.
      
      WritePrepared already logs the rollback batch into the WAL, if a rollback happens after prepare, so there is no problem there.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6418
      
      Differential Revision: D19896151
      
      Pulled By: lth
      
      fbshipit-source-id: 2ff65ddc5fe75efd57736fed4b7cd7a109d26609
      fb571509
    • sdong's avatar
      Should flush and sync WAL when writing it in DB::Open() (#6417) · ac8e89a4
      sdong authored
      Summary:
      A recent fix related to 2pc https://github.com/facebook/rocksdb/pull/6313/ writes something to WAL, but does not flush or sync. This causes assertion failure "impl->TEST_WALBufferIsEmpty()" if manual_wal_flush = true. We should fsync the entry to make sure a second power reset can recover.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6417
      
      Test Plan: Add manual_wal_flush=true case in TransactionTest.DoubleCrashInRecovery and fix a bug in the test so that the bug can be reproduced. It passes with the fix.
      
      Differential Revision: D19894537
      
      fbshipit-source-id: f1e84e49e2269f583c6019743118292cd8b6598e
      ac8e89a4
    • Cheng Chang's avatar
      Fix flaky test DecreaseNumBgThreads (#6393) · 46516778
      Cheng Chang authored
      Summary:
      The DecreaseNumBgThreads test keeps failing on Windows in AppVeyor.
      It fails because it depends on a timed wait for the tasks to be dequeued from the threadpool's internal queue, but within the specified time, the task might have not been scheduled onto the newly created threads.
      https://github.com/facebook/rocksdb/pull/6232 tries to fix this by waiting for longer time to let the threads scheduled.
      This PR tries to fix this by replacing the timed wait with a synchronization on the task's internal conditional variable.
      When the number of threads increases, instead of guessing the time needed for the task to be scheduled, it directly blocks on the conditional variable until the task starts running.
      But when thread number is reduced, it still does a timed wait, but this does not lead to the flakiness now, will try to remove these timed waits in a future PR.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6393
      
      Test Plan: Wait to see whether AppVeyor tests pass.
      
      Differential Revision: D19890928
      
      Pulled By: cheng-chang
      
      fbshipit-source-id: 4e56e4addf625c98c0876e62d9d57a6f0a156f76
      46516778
    • sdong's avatar
      Remove IO URING compiler flags (#6415) · acfee40a
      sdong authored
      Summary:
      Since IO Uring feature is not stable. Remove it from buck configuration.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6415
      
      Test Plan: See internal build pass
      
      Differential Revision: D19892988
      
      fbshipit-source-id: 7fc01efc2af5ed707fb8e4e4674223aeb83cd5ea
      acfee40a
    • Huisheng Liu's avatar
      Fix destroydb (#6308) · 5138764e
      Huisheng Liu authored
      Summary:
      It's observed on Windows DestroyDB failed to remove the log file because the logger is still alive in sst file manager and holding a handle to the log file. This fix makes sure the logger is released before attempt to clear the database directory.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6308
      
      Differential Revision: D19818829
      
      Pulled By: riversand963
      
      fbshipit-source-id: 54c3e6859aadaaba4a49b3e851b73dc35ec7dc6a
      5138764e
    • sdong's avatar
      Fix db_bench LITE build recently broken (#6411) · df3f33dd
      sdong authored
      Summary:
      A recent change https://github.com/facebook/rocksdb/pull/6386 broke LITE build in a trivial way. Fix it.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6411
      
      Test Plan: Run "LITE=1 make all"
      
      Differential Revision: D19871765
      
      fbshipit-source-id: 74f0ad3f8a9d666fbde0da7fd29ba1547a811f77
      df3f33dd
    • Cheng Chang's avatar
      Revert usage of Defer. (#6410) · a676001f
      Cheng Chang authored
      Summary:
      Seems like this caused the following test failure on AppVeyor:
      DBTest2.CrashInRecoveryMultipleCF
      c:\projects\rocksdb\db\db_test_util.cc(107): error: DestroyDB(dbname_, options)
      IO error: Failed to delete: C:\projects\rocksdb\db_tests\\testrocksdb-3112//db_test2_10791409581227174103/000013.sst: Access is denied.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6410
      
      Test Plan: Wait to see whether the AppVeyor test passes.
      
      Differential Revision: D19879872
      
      Pulled By: cheng-chang
      
      fbshipit-source-id: 59a9c55ca88566e9210c0b715ecc45a4fd9afe26
      a676001f
  6. 13 Feb, 2020 2 commits
  7. 12 Feb, 2020 2 commits
  8. 11 Feb, 2020 10 commits
    • Yanqin Jin's avatar
      Pass info_log_level to the inner logger of AutoRollLogger (#6388) · 28aa09dc
      Yanqin Jin authored
      Summary:
      Before this fix, the info_log_level passed from CreateLoggerFromOptions() will
      be ignored by AutoRollLogger::logger_. This PR fixes it by setting the info log
      level of logger_ during ResetLogger().
      
      Test plan (dev server):
      ```
      COMPILE_WITH_TSAN=1 make all && make check
      make all && make check
      ```
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6388
      
      Differential Revision: D19828045
      
      Pulled By: riversand963
      
      fbshipit-source-id: e1ac7de3a2090bee53b6c667f71a11f1774163e6
      28aa09dc
    • anand76's avatar
      Revert "Check KeyContext status in MultiGet (#6387)" (#6401) · 35ed530d
      anand76 authored
      Summary:
      This reverts commit d70011bc. The commit is causing some stress test failure due to unexpected Status::MergeInProgress() return for some keys.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6401
      
      Differential Revision: D19826623
      
      Pulled By: anand1976
      
      fbshipit-source-id: edd634cede9cb7bdd2cb8f46e662ea709b16d2f1
      35ed530d
    • Cheng Chang's avatar
      Update unit tests for PinnableSlice (#6399) · d3ba398b
      Cheng Chang authored
      Summary:
      1. remove AssertEmpty because calling methods on moved objects is discouraged.
      2. add a test to assert that the internal buffer is moved instead of being copied.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6399
      
      Test Plan:
      make slice_test && ./slice_test
      USE_CLANG=1 make analyze
      
      Differential Revision: D19825372
      
      Pulled By: cheng-chang
      
      fbshipit-source-id: 2e26f8ce5ec3edbfce067db045e80bd433e704f4
      d3ba398b
    • Cheng Chang's avatar
      Add utility class Defer (#6382) · dafb5680
      Cheng Chang authored
      Summary:
      Add a utility class `Defer` to defer the execution of a function until the Defer object goes out of scope.
      Used in VersionSet:: ProcessManifestWrites as an example.
      The inline comments for class `Defer` have more details.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6382
      
      Test Plan: `make defer_test version_set_test && ./defer_test && ./version_set_test`
      
      Differential Revision: D19797538
      
      Pulled By: cheng-chang
      
      fbshipit-source-id: b1a9b7306e4fd4f48ec2ab55783caa561a315f0f
      dafb5680
    • Levi Tamasi's avatar
      Do not move VersionEdit into AtomicGroupReadBuffer (#6400) · cbf5f3be
      Levi Tamasi authored
      Summary:
      https://github.com/facebook/rocksdb/pull/6383 surfaced an issue with
      `VersionSet`/`ReactiveVersionSet` and `AtomicGroupReadBuffer::AddEdit`
      (which was added in https://github.com/facebook/rocksdb/pull/5411):
      `AddEdit` moves the `VersionEdit` passed to it into `replay_buffer_`,
      however, the client `VersionSet` classes keep using it afterwards. This
      *seemed to* work before the refactoring but it really did not: since
      `VersionEdit` used to have a user-declared destructor, no move
      constructor/move assignment operator was generated, and the `move` in
      `AddEdit` was really a copy. The patch makes the copy explicit. Note: it
      should be possible to rework this logic so that we can get away
      with the move but for now, this should fix the issue.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6400
      
      Test Plan:
      `make check`
      `make analyze`
      
      Differential Revision: D19824466
      
      Pulled By: ltamasi
      
      fbshipit-source-id: f38033967daf2a39c78dcd6e12978bafe37632b4
      cbf5f3be
    • Zhichao Cao's avatar
      Checksum for each SST file and stores in MANIFEST (#6216) · 4369f2c7
      Zhichao Cao authored
      Summary:
      In the current code base, RocksDB generate the checksum for each block and verify the checksum at usage. Current PR enable SST file checksum. After a SST file is generated by Flush or Compaction, RocksDB generate the SST file checksum and store the checksum value and checksum method name in the vs_info and MANIFEST as part for the FileMetadata.
      
      Added the enable_sst_file_checksum to Options to enable or disable file checksum. Added sst_file_checksum to Options such that user can plugin their own SST file checksum calculate method via overriding the SstFileChecksum class. The checksum information inlcuding uint32_t checksum value and a checksum name (string).  A new tool is added to LDB such that user can dump out a list of file checksum information from MANIFEST. If user enables the file checksum but does not provide the sst_file_checksum instance, RocksDB will use the default crc32checksum implemented in table/sst_file_checksum_crc32c.h
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6216
      
      Test Plan: Added the testing case in table_test and ldb_cmd_test to verify checksum is correct in different level. Pass make asan_check.
      
      Differential Revision: D19171461
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: b2e53479eefc5bb0437189eaa1941670e5ba8b87
      4369f2c7
    • sdong's avatar
      Make clang analyze happy with options_test (#6398) · 594e815e
      sdong authored
      Summary:
      clang analysis shows following warning:
      
      options/options_test.cc:1554:24: warning: The left operand of '-' is a garbage value
                  (file_size - 1) / readahead_size + 1);
                   ~~~~~~~~~ ^
      
      Explicitly initialize file_size and add an assertion to make clang analysis happy.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6398
      
      Test Plan: Run "make analysis" and see the warning goes away.
      
      Differential Revision: D19819662
      
      fbshipit-source-id: 1589ea91c0c8f78242538f01448e4ad0e5fbc219
      594e815e
    • sdong's avatar
      Try to fix some analysis failures · b2bc1da5
      sdong authored
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6384
      
      Test Plan: Wait and see the analysis result.
      
      Differential Revision: D19781072
      
      fbshipit-source-id: 75e7cb6ee619ebd289841eaabea03dd075c09d3b
      b2bc1da5
    • Yutian Li's avatar
      Add error status for no_slowdown & low priority write (#6396) · 2e0159ec
      Yutian Li authored
      Summary:
      When `no_slowdown` is enabled, it returns `Status::Incomplete("Write stall")` if a stall would occur. This patch adds descriptive text for when `no_slowdown` and `low_pri` are enabled.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6396
      
      Differential Revision: D19808978
      
      Pulled By: cheng-chang
      
      fbshipit-source-id: a53b0d25ed414c821a086531e0222027f925e627
      2e0159ec
    • Kefu Chai's avatar
      utilities/env_librados: copy use bufferlist::iterator (#6395) · debc4ef1
      Kefu Chai authored
      Summary:
      to adapt the change in ceph upstream where the bufferlist::copy() method
      was removed in
      https://github.com/ceph/ceph/commit/c724369010a753bd44e11a534d1f42156c4fc12d
      
      
      
      Signed-off-by: default avatarKefu Chai <tchaikov@gmail.com>
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6395
      
      Differential Revision: D19816815
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: 9210767b91af0ecdcf5dfaa3e70edcaeea55135f
      debc4ef1
  9. 08 Feb, 2020 7 commits
    • blackyblack's avatar
      Only add gtest target when building with tests (#6377) · 84b41a69
      blackyblack authored
      Summary:
      External CMAKE projects cannot add own gtest target due to CMP0002 policy. We should not add gtest target unless we build with tests.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6377
      
      Differential Revision: D19801708
      
      fbshipit-source-id: 662484683e8737e10397ddb0d17705da5243c4f9
      84b41a69
    • anand76's avatar
      Check KeyContext status in MultiGet (#6387) · d70011bc
      anand76 authored
      Summary:
      Currently, any IO errors and checksum mismatches while reading data
      blocks, are being ignored by the batched MultiGet. Its only looking at
      the GetContext state. Fix that.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6387
      
      Test Plan: Add unit tests
      
      Differential Revision: D19799819
      
      Pulled By: anand1976
      
      fbshipit-source-id: 46133dccbb04e64067b9fe6cda73e282203db969
      d70011bc
    • Robert Yang's avatar
      db/write_thread.cc: Initialize state (#6275) · 4e457278
      Robert Yang authored
      
      
      Summary:
      Fixed an error when compiled with -Og:
      db/write_thread.cc:183:14: error: 'state' may be used uninitialized in this function [-Werror=maybe-uninitialized]
      
      Signed-off-by: default avatarRobert Yang <liezhi.yang@windriver.com>
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6275
      
      Differential Revision: D19381755
      
      fbshipit-source-id: a90bf3cd4a7248d9d71219e918fc6253deb97e3c
      4e457278
    • sdong's avatar
      Allow readahead when reading option files. (#6372) · 876c2dbf
      sdong authored
      Summary:
      Right, when reading from option files, no readahead is used and 8KB buffer is used. It might introduce high latency if the file system provide high latency and doesn't do readahead. Instead, introduce a readahead to the file. When calling inside DB, infer the value from options.log_readahead. Otherwise, a default 512KB readahead size is used.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6372
      
      Test Plan: Add --log_readahead_size in db_bench. Run it with several options and observe read size from option files using strace.
      
      Differential Revision: D19727739
      
      fbshipit-source-id: e6d8053b0a64259abc087f1f388b9cd66fa8a583
      876c2dbf
    • Cheng Chang's avatar
      Support move semantics for PinnableSlice (#6374) · b42fa149
      Cheng Chang authored
      Summary:
      It's logically correct for PinnableSlice to support move semantics to transfer ownership of the pinned memory region. This PR adds both move constructor and move assignment to PinnableSlice.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6374
      
      Test Plan:
      A set of unit tests for the move semantics are added in slice_test.
      So `make slice_test && ./slice_test`.
      
      Differential Revision: D19739254
      
      Pulled By: cheng-chang
      
      fbshipit-source-id: f898bd811bb05b2d87384ec58b645e9915e8e0b1
      b42fa149
    • Chad Austin's avatar
      Fix Buck build on macOS (#6378) · 25fbdc5a
      Chad Austin authored
      Summary:
      liburing is a Linux-specific dependency, so make sure it's configured in the Linux-only Buck rules.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6378
      
      Test Plan:
      ```
      ~/fbcode $ cp internal_repo_rocksdb/repo/TARGETS rocksdb/src
      ~/fbcode $ buck build mode/mac eden
      ```
      
      Reviewed By: chadaustin
      
      Differential Revision: D19760039
      
      Pulled By: riversand963
      
      fbshipit-source-id: 2abfce81c8b17965ef76012262cd117708e0294f
      25fbdc5a
    • Levi Tamasi's avatar
      Clean up VersionEdit a bit (#6383) · 752c87af
      Levi Tamasi authored
      Summary:
      This is a bunch of small improvements to `VersionEdit`. Namely, the patch
      
      * Makes the names and order of variables, methods, and code chunks related
        to the various information elements more consistent, and adds missing
        getters for the sake of completeness.
      * Initializes previously uninitialized stack variables.
      * Marks all getters const to improve const correctness.
      * Adds in-class initializers and removes the default ctor that would
        create an object with uninitialized built-in fields and call `Clear`
        afterwards.
      * Adds a new type alias for new files and changes the existing `typedef`
        for deleted files into a type alias as well.
      * Makes the helper method `DecodeNewFile4From` private.
      * Switches from long-winded iterator syntax to range based loops in a
        couple of places.
      * Fixes a couple of assignments where an integer 0 was assigned to
        boolean members.
      * Fixes a getter which used to return a `const std::string` instead of
      the intended `const std::string&`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6383
      
      Test Plan: make check
      
      Differential Revision: D19780537
      
      Pulled By: ltamasi
      
      fbshipit-source-id: b0b4f09fee0ec0e7c7b7a6d76bfe5346e91824d0
      752c87af