1. 21 Feb, 2020 2 commits
  2. 20 Feb, 2020 1 commit
  3. 19 Feb, 2020 3 commits
  4. 15 Feb, 2020 4 commits
  5. 14 Feb, 2020 9 commits
    • wolfkdy's avatar
      refine code (#6420) · 29e24434
      wolfkdy authored
      I create a new branch from the branch new upsteram/master and "git merge --squash".
      Maybe it will fix everything.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6420
      Differential Revision: D19897152
      Pulled By: zhichao-cao
      fbshipit-source-id: 6575d9e3b23e360f42ee1480b43028b5fcc20136
    • Manuel Ung's avatar
      WriteUnPrepared: Fix assertion during recovery (#6419) · 908b1ee6
      Manuel Ung authored
      During recovery, multiple (un)prepared batches could exist in the same WAL record due to group commit. This breaks an assertion in `MemTableInserter::MarkBeginPrepare`.
      To fix, reset unprepared_batch_ to false after `MarkEndPrepare`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6419
      Differential Revision: D19896148
      Pulled By: lth
      fbshipit-source-id: b1a32ef88f775a0881264a18bd1a4a5b8c85eee3
    • Manuel Ung's avatar
      WriteUnPrepared: Enable WAL during crash recovery (#6418) · fb571509
      Manuel Ung authored
      Unfortunately, it seems like mysqld reuses xids across machine restarts. When that happens, we could have something like the following happening:
      BEGIN_PREPARE(unprepared) Put(a) END_PREPARE(xid = 1)
      -- crash and recover with Put(a) rolled back as it was not prepared
      BEGIN_PREPARE(prepared) Put(b) END_PREPARE(xid = 1)
      COMMIT(xid = 1)
      -- crash and recover with both a, b
      To solve this, we will have to log the rollback batch into the WAL during recovery.
      WritePrepared already logs the rollback batch into the WAL, if a rollback happens after prepare, so there is no problem there.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6418
      Differential Revision: D19896151
      Pulled By: lth
      fbshipit-source-id: 2ff65ddc5fe75efd57736fed4b7cd7a109d26609
    • sdong's avatar
      Should flush and sync WAL when writing it in DB::Open() (#6417) · ac8e89a4
      sdong authored
      A recent fix related to 2pc https://github.com/facebook/rocksdb/pull/6313/ writes something to WAL, but does not flush or sync. This causes assertion failure "impl->TEST_WALBufferIsEmpty()" if manual_wal_flush = true. We should fsync the entry to make sure a second power reset can recover.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6417
      Test Plan: Add manual_wal_flush=true case in TransactionTest.DoubleCrashInRecovery and fix a bug in the test so that the bug can be reproduced. It passes with the fix.
      Differential Revision: D19894537
      fbshipit-source-id: f1e84e49e2269f583c6019743118292cd8b6598e
    • Cheng Chang's avatar
      Fix flaky test DecreaseNumBgThreads (#6393) · 46516778
      Cheng Chang authored
      The DecreaseNumBgThreads test keeps failing on Windows in AppVeyor.
      It fails because it depends on a timed wait for the tasks to be dequeued from the threadpool's internal queue, but within the specified time, the task might have not been scheduled onto the newly created threads.
      https://github.com/facebook/rocksdb/pull/6232 tries to fix this by waiting for longer time to let the threads scheduled.
      This PR tries to fix this by replacing the timed wait with a synchronization on the task's internal conditional variable.
      When the number of threads increases, instead of guessing the time needed for the task to be scheduled, it directly blocks on the conditional variable until the task starts running.
      But when thread number is reduced, it still does a timed wait, but this does not lead to the flakiness now, will try to remove these timed waits in a future PR.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6393
      Test Plan: Wait to see whether AppVeyor tests pass.
      Differential Revision: D19890928
      Pulled By: cheng-chang
      fbshipit-source-id: 4e56e4addf625c98c0876e62d9d57a6f0a156f76
    • sdong's avatar
      Remove IO URING compiler flags (#6415) · acfee40a
      sdong authored
      Since IO Uring feature is not stable. Remove it from buck configuration.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6415
      Test Plan: See internal build pass
      Differential Revision: D19892988
      fbshipit-source-id: 7fc01efc2af5ed707fb8e4e4674223aeb83cd5ea
    • Huisheng Liu's avatar
      Fix destroydb (#6308) · 5138764e
      Huisheng Liu authored
      It's observed on Windows DestroyDB failed to remove the log file because the logger is still alive in sst file manager and holding a handle to the log file. This fix makes sure the logger is released before attempt to clear the database directory.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6308
      Differential Revision: D19818829
      Pulled By: riversand963
      fbshipit-source-id: 54c3e6859aadaaba4a49b3e851b73dc35ec7dc6a
    • sdong's avatar
      Fix db_bench LITE build recently broken (#6411) · df3f33dd
      sdong authored
      A recent change https://github.com/facebook/rocksdb/pull/6386 broke LITE build in a trivial way. Fix it.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6411
      Test Plan: Run "LITE=1 make all"
      Differential Revision: D19871765
      fbshipit-source-id: 74f0ad3f8a9d666fbde0da7fd29ba1547a811f77
    • Cheng Chang's avatar
      Revert usage of Defer. (#6410) · a676001f
      Cheng Chang authored
      Seems like this caused the following test failure on AppVeyor:
      c:\projects\rocksdb\db\db_test_util.cc(107): error: DestroyDB(dbname_, options)
      IO error: Failed to delete: C:\projects\rocksdb\db_tests\\testrocksdb-3112//db_test2_10791409581227174103/000013.sst: Access is denied.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6410
      Test Plan: Wait to see whether the AppVeyor test passes.
      Differential Revision: D19879872
      Pulled By: cheng-chang
      fbshipit-source-id: 59a9c55ca88566e9210c0b715ecc45a4fd9afe26
  6. 13 Feb, 2020 2 commits
  7. 12 Feb, 2020 2 commits
  8. 11 Feb, 2020 10 commits
    • Yanqin Jin's avatar
      Pass info_log_level to the inner logger of AutoRollLogger (#6388) · 28aa09dc
      Yanqin Jin authored
      Before this fix, the info_log_level passed from CreateLoggerFromOptions() will
      be ignored by AutoRollLogger::logger_. This PR fixes it by setting the info log
      level of logger_ during ResetLogger().
      Test plan (dev server):
      COMPILE_WITH_TSAN=1 make all && make check
      make all && make check
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6388
      Differential Revision: D19828045
      Pulled By: riversand963
      fbshipit-source-id: e1ac7de3a2090bee53b6c667f71a11f1774163e6
    • anand76's avatar
      Revert "Check KeyContext status in MultiGet (#6387)" (#6401) · 35ed530d
      anand76 authored
      This reverts commit d70011bc. The commit is causing some stress test failure due to unexpected Status::MergeInProgress() return for some keys.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6401
      Differential Revision: D19826623
      Pulled By: anand1976
      fbshipit-source-id: edd634cede9cb7bdd2cb8f46e662ea709b16d2f1
    • Cheng Chang's avatar
      Update unit tests for PinnableSlice (#6399) · d3ba398b
      Cheng Chang authored
      1. remove AssertEmpty because calling methods on moved objects is discouraged.
      2. add a test to assert that the internal buffer is moved instead of being copied.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6399
      Test Plan:
      make slice_test && ./slice_test
      USE_CLANG=1 make analyze
      Differential Revision: D19825372
      Pulled By: cheng-chang
      fbshipit-source-id: 2e26f8ce5ec3edbfce067db045e80bd433e704f4
    • Cheng Chang's avatar
      Add utility class Defer (#6382) · dafb5680
      Cheng Chang authored
      Add a utility class `Defer` to defer the execution of a function until the Defer object goes out of scope.
      Used in VersionSet:: ProcessManifestWrites as an example.
      The inline comments for class `Defer` have more details.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6382
      Test Plan: `make defer_test version_set_test && ./defer_test && ./version_set_test`
      Differential Revision: D19797538
      Pulled By: cheng-chang
      fbshipit-source-id: b1a9b7306e4fd4f48ec2ab55783caa561a315f0f
    • Levi Tamasi's avatar
      Do not move VersionEdit into AtomicGroupReadBuffer (#6400) · cbf5f3be
      Levi Tamasi authored
      https://github.com/facebook/rocksdb/pull/6383 surfaced an issue with
      `VersionSet`/`ReactiveVersionSet` and `AtomicGroupReadBuffer::AddEdit`
      (which was added in https://github.com/facebook/rocksdb/pull/5411):
      `AddEdit` moves the `VersionEdit` passed to it into `replay_buffer_`,
      however, the client `VersionSet` classes keep using it afterwards. This
      *seemed to* work before the refactoring but it really did not: since
      `VersionEdit` used to have a user-declared destructor, no move
      constructor/move assignment operator was generated, and the `move` in
      `AddEdit` was really a copy. The patch makes the copy explicit. Note: it
      should be possible to rework this logic so that we can get away
      with the move but for now, this should fix the issue.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6400
      Test Plan:
      `make check`
      `make analyze`
      Differential Revision: D19824466
      Pulled By: ltamasi
      fbshipit-source-id: f38033967daf2a39c78dcd6e12978bafe37632b4
    • Zhichao Cao's avatar
      Checksum for each SST file and stores in MANIFEST (#6216) · 4369f2c7
      Zhichao Cao authored
      In the current code base, RocksDB generate the checksum for each block and verify the checksum at usage. Current PR enable SST file checksum. After a SST file is generated by Flush or Compaction, RocksDB generate the SST file checksum and store the checksum value and checksum method name in the vs_info and MANIFEST as part for the FileMetadata.
      Added the enable_sst_file_checksum to Options to enable or disable file checksum. Added sst_file_checksum to Options such that user can plugin their own SST file checksum calculate method via overriding the SstFileChecksum class. The checksum information inlcuding uint32_t checksum value and a checksum name (string).  A new tool is added to LDB such that user can dump out a list of file checksum information from MANIFEST. If user enables the file checksum but does not provide the sst_file_checksum instance, RocksDB will use the default crc32checksum implemented in table/sst_file_checksum_crc32c.h
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6216
      Test Plan: Added the testing case in table_test and ldb_cmd_test to verify checksum is correct in different level. Pass make asan_check.
      Differential Revision: D19171461
      Pulled By: zhichao-cao
      fbshipit-source-id: b2e53479eefc5bb0437189eaa1941670e5ba8b87
    • sdong's avatar
      Make clang analyze happy with options_test (#6398) · 594e815e
      sdong authored
      clang analysis shows following warning:
      options/options_test.cc:1554:24: warning: The left operand of '-' is a garbage value
                  (file_size - 1) / readahead_size + 1);
                   ~~~~~~~~~ ^
      Explicitly initialize file_size and add an assertion to make clang analysis happy.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6398
      Test Plan: Run "make analysis" and see the warning goes away.
      Differential Revision: D19819662
      fbshipit-source-id: 1589ea91c0c8f78242538f01448e4ad0e5fbc219
    • sdong's avatar
      Try to fix some analysis failures · b2bc1da5
      sdong authored
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6384
      Test Plan: Wait and see the analysis result.
      Differential Revision: D19781072
      fbshipit-source-id: 75e7cb6ee619ebd289841eaabea03dd075c09d3b
    • Yutian Li's avatar
      Add error status for no_slowdown & low priority write (#6396) · 2e0159ec
      Yutian Li authored
      When `no_slowdown` is enabled, it returns `Status::Incomplete("Write stall")` if a stall would occur. This patch adds descriptive text for when `no_slowdown` and `low_pri` are enabled.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6396
      Differential Revision: D19808978
      Pulled By: cheng-chang
      fbshipit-source-id: a53b0d25ed414c821a086531e0222027f925e627
    • Kefu Chai's avatar
      utilities/env_librados: copy use bufferlist::iterator (#6395) · debc4ef1
      Kefu Chai authored
      to adapt the change in ceph upstream where the bufferlist::copy() method
      was removed in
      Signed-off-by: default avatarKefu Chai <tchaikov@gmail.com>
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6395
      Differential Revision: D19816815
      Pulled By: zhichao-cao
      fbshipit-source-id: 9210767b91af0ecdcf5dfaa3e70edcaeea55135f
  9. 08 Feb, 2020 7 commits
    • blackyblack's avatar
      Only add gtest target when building with tests (#6377) · 84b41a69
      blackyblack authored
      External CMAKE projects cannot add own gtest target due to CMP0002 policy. We should not add gtest target unless we build with tests.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6377
      Differential Revision: D19801708
      fbshipit-source-id: 662484683e8737e10397ddb0d17705da5243c4f9
    • anand76's avatar
      Check KeyContext status in MultiGet (#6387) · d70011bc
      anand76 authored
      Currently, any IO errors and checksum mismatches while reading data
      blocks, are being ignored by the batched MultiGet. Its only looking at
      the GetContext state. Fix that.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6387
      Test Plan: Add unit tests
      Differential Revision: D19799819
      Pulled By: anand1976
      fbshipit-source-id: 46133dccbb04e64067b9fe6cda73e282203db969
    • Robert Yang's avatar
      db/write_thread.cc: Initialize state (#6275) · 4e457278
      Robert Yang authored
      Fixed an error when compiled with -Og:
      db/write_thread.cc:183:14: error: 'state' may be used uninitialized in this function [-Werror=maybe-uninitialized]
      Signed-off-by: default avatarRobert Yang <liezhi.yang@windriver.com>
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6275
      Differential Revision: D19381755
      fbshipit-source-id: a90bf3cd4a7248d9d71219e918fc6253deb97e3c
    • sdong's avatar
      Allow readahead when reading option files. (#6372) · 876c2dbf
      sdong authored
      Right, when reading from option files, no readahead is used and 8KB buffer is used. It might introduce high latency if the file system provide high latency and doesn't do readahead. Instead, introduce a readahead to the file. When calling inside DB, infer the value from options.log_readahead. Otherwise, a default 512KB readahead size is used.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6372
      Test Plan: Add --log_readahead_size in db_bench. Run it with several options and observe read size from option files using strace.
      Differential Revision: D19727739
      fbshipit-source-id: e6d8053b0a64259abc087f1f388b9cd66fa8a583
    • Cheng Chang's avatar
      Support move semantics for PinnableSlice (#6374) · b42fa149
      Cheng Chang authored
      It's logically correct for PinnableSlice to support move semantics to transfer ownership of the pinned memory region. This PR adds both move constructor and move assignment to PinnableSlice.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6374
      Test Plan:
      A set of unit tests for the move semantics are added in slice_test.
      So `make slice_test && ./slice_test`.
      Differential Revision: D19739254
      Pulled By: cheng-chang
      fbshipit-source-id: f898bd811bb05b2d87384ec58b645e9915e8e0b1
    • Chad Austin's avatar
      Fix Buck build on macOS (#6378) · 25fbdc5a
      Chad Austin authored
      liburing is a Linux-specific dependency, so make sure it's configured in the Linux-only Buck rules.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6378
      Test Plan:
      ~/fbcode $ cp internal_repo_rocksdb/repo/TARGETS rocksdb/src
      ~/fbcode $ buck build mode/mac eden
      Reviewed By: chadaustin
      Differential Revision: D19760039
      Pulled By: riversand963
      fbshipit-source-id: 2abfce81c8b17965ef76012262cd117708e0294f
    • Levi Tamasi's avatar
      Clean up VersionEdit a bit (#6383) · 752c87af
      Levi Tamasi authored
      This is a bunch of small improvements to `VersionEdit`. Namely, the patch
      * Makes the names and order of variables, methods, and code chunks related
        to the various information elements more consistent, and adds missing
        getters for the sake of completeness.
      * Initializes previously uninitialized stack variables.
      * Marks all getters const to improve const correctness.
      * Adds in-class initializers and removes the default ctor that would
        create an object with uninitialized built-in fields and call `Clear`
      * Adds a new type alias for new files and changes the existing `typedef`
        for deleted files into a type alias as well.
      * Makes the helper method `DecodeNewFile4From` private.
      * Switches from long-winded iterator syntax to range based loops in a
        couple of places.
      * Fixes a couple of assignments where an integer 0 was assigned to
        boolean members.
      * Fixes a getter which used to return a `const std::string` instead of
      the intended `const std::string&`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6383
      Test Plan: make check
      Differential Revision: D19780537
      Pulled By: ltamasi
      fbshipit-source-id: b0b4f09fee0ec0e7c7b7a6d76bfe5346e91824d0