1. 11 6月, 2020 2 次提交
    • Levi Tamasi's avatar
      Use a per-thread path for the export directory in import_column_family_test (#6962) · fbe2d259
      Levi Tamasi 创作于
      Summary:
      This is required so that the test cases can safely be run in parallel.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6962
      
      Test Plan: `make check`
      
      Reviewed By: zhichao-cao
      
      Differential Revision: D21980060
      
      Pulled By: ltamasi
      
      fbshipit-source-id: 616b7a0b686155d3874848b9098c67ad3f47efcc
      fbe2d259
    • Andrew Kryczka's avatar
      save a key comparison in block seeks (#6646) · e6be168a
      Andrew Kryczka 创作于
      Summary:
      This saves up to two key comparisons in block seeks. The first key
      comparison saved is a redundant key comparison against the restart key
      where the linear scan starts. This comparison is saved in all cases
      except when the found key is in the first restart interval. The
      second key comparison saved is a redundant key comparison against the
      restart key where the linear scan ends. This is only saved in cases
      where all keys in the restart interval are less than the target
      (probability roughly `1/restart_interval`).
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6646
      
      Test Plan:
      ran a benchmark with mostly default settings and counted key comparisons
      
      before: `user_key_comparison_count = 19399529`
      after: `user_key_comparison_count = 18431498`
      
      setup command:
      
      ```
      $ TEST_TMPDIR=/dev/shm/dbbench ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -max_background_jobs=12 -level_compaction_dynamic_level_bytes=true -num=10000000
      ```
      
      benchmark command:
      
      ```
      $ TEST_TMPDIR=/dev/shm/dbbench/ ./db_bench -use_existing_db=true -benchmarks=readrandom -disable_auto_compactions=true -num=10000000 -compression_type=none -reads=1000000 -perf_level=3
      ```
      
      Reviewed By: pdillinger
      
      Differential Revision: D20849707
      
      Pulled By: ajkr
      
      fbshipit-source-id: 1f01c5cd99ea771fd27974046e37b194f1cdcfac
      e6be168a
  2. 10 6月, 2020 3 次提交
  3. 09 6月, 2020 5 次提交
    • sdong's avatar
      Introduce some Linux build to CircleCI (#6937) · 6a8ddd37
      sdong 创作于
      Summary:
      Moving towards the long term goal of moving most CI build to CircleCI when possible, add some Linux tests in CircleCI. This is not all what we can include to CircleCI. For example, Java builds are not includ
      ed.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6937
      
      Test Plan: Watch CI build results.
      
      Reviewed By: pdillinger
      
      Differential Revision: D21941605
      
      fbshipit-source-id: db6aead3c45f523386d4fb30d224cfde573cccad
      6a8ddd37
    • anand76's avatar
      Fix a bug in looking up duplicate keys with MultiGet (#6953) · 1fb3593f
      anand76 创作于
      Summary:
      When MultiGet is called with duplicate keys, and the key matches the
      largest key in an SST file and the value type is merge, only the first
      instance of the duplicate key is returned with correct results. This is
      due to the incorrect assumption that if a key in a batch is equal to the
      largest key in the file, the next key cannot be present in that file.
      
      Tests:
      Add a new unit test
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6953
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21935898
      
      Pulled By: anand1976
      
      fbshipit-source-id: a2cc327a15150e23fd997546ca64d1c33021cb4c
      1fb3593f
    • Levi Tamasi's avatar
      Add convenience method GetFileMetaDataByNumber (#6940) · f5e64945
      Levi Tamasi 创作于
      Summary:
      The patch adds a convenience method `GetFileMetaDataByNumber` that
      builds on the `FileLocation` functionality introduced recently (see
      https://github.com/facebook/rocksdb/pull/6862). This method makes it possible to
      retrieve the `FileMetaData` directly as opposed to having to go through
      `LevelFiles` and friends.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6940
      
      Test Plan: `make check`
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21905946
      
      Pulled By: ltamasi
      
      fbshipit-source-id: af99e19de21242b2b4a87594a535c6028d16ee72
      f5e64945
    • Zitan Chen's avatar
      Implement a new subcommand "identify" for sst_dump (#6943) · 119b26fa
      Zitan Chen 创作于
      Summary:
      Implemented a subcommand of sst_dump called identify, which determines whether a file is an SST file or identifies and lists all the SST files in a directory;
      
      This update also fixes the problem that sst_dump exits with a success state even if target file/directory does not exist/is not an SST file/is empty/is corrupted.
      
      One test is added to sst_dump_test.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6943
      
      Test Plan: Passed make check and a few manual tests
      
      Reviewed By: pdillinger
      
      Differential Revision: D21928985
      
      Pulled By: gg814
      
      fbshipit-source-id: 9a8b48e0cf1a0e96b13f42b690aba8ad981afad3
      119b26fa
    • Zhichao Cao's avatar
      decouple the dependency of trace_analyzer_test unit test (#6941) · fb08330f
      Zhichao Cao 创作于
      Summary:
      Since gflags use the global variable to store the flags passed in. In the unit test, if we git one flag per unit test, the result is that all the flags are combined together in the following tests. Therefore, it has the dependency. In this PR, we pass the full arguments each time to ensure that the old arguments will be overwritten by the new one such that the dependency is removed.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6941
      
      Test Plan: make asan_check. run each unit test in trace_analyzer_test independently and in arbitrary orders.
      
      Reviewed By: pdillinger
      
      Differential Revision: D21909176
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: dca550a0a4a205c30faa620e258a020a3b5b4e13
      fb08330f
  4. 08 6月, 2020 1 次提交
    • Yanqin Jin's avatar
      Remove unnecessary inclusion of version_edit.h in env (#6952) · 3020df9d
      Yanqin Jin 创作于
      Summary:
      In db_options.c, we should avoid including header files in the `db` directory to avoid introducing unnecessary dependency. The reason why `version_edit.h` has been included in `db_options.cc` is because we need two constants, `kUnknownChecksum` and `kUnknownChecksumFuncName`. We can put these two constants as `constexpr` in the public header `file_checksum.h`.
      
      Test plan (devserver):
      make check
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6952
      
      Reviewed By: zhichao-cao
      
      Differential Revision: D21925341
      
      Pulled By: riversand963
      
      fbshipit-source-id: 2902f3b74c97f0cf16c58ad24c095c787c3a40e2
      3020df9d
  5. 06 6月, 2020 8 次提交
  6. 05 6月, 2020 7 次提交
    • Peter Dillinger's avatar
      Fix some defects reported by Coverity Scan (#6933) · aaece2a9
      Peter Dillinger 创作于
      Summary:
      Confusing checks for null that are never null
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6933
      
      Test Plan: make check
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21885466
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 4b48e03c2a33727f2702b0d12292f9fda5a3c475
      aaece2a9
    • Peter Dillinger's avatar
      Fix more defects reported by Coverity Scan (#6935) · c7432cc3
      Peter Dillinger 创作于
      Summary:
      Mostly uninitialized values: some probably written before use, but some seem like bugs. Also, destructor needs to be virtual, and possible use-after-free in test
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6935
      
      Test Plan: make check
      
      Reviewed By: siying
      
      Differential Revision: D21885484
      
      Pulled By: pdillinger
      
      fbshipit-source-id: e2e7cb0a0cf196f2b55edd16f0634e81f6cc8e08
      c7432cc3
    • Yanqin Jin's avatar
      Close file to avoid file-descriptor leakage (#6936) · a8170d77
      Yanqin Jin 创作于
      Summary:
      When operation on an open file descriptor fails, we should close the file descriptor.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6936
      
      Test Plan: make check
      
      Reviewed By: pdillinger
      
      Differential Revision: D21885458
      
      Pulled By: riversand963
      
      fbshipit-source-id: ba077a76b256a8537f21e22e4ec198f45390bf50
      a8170d77
    • sdong's avatar
      Make StringAppendOperatorTest a parameterized test (#6930) · 6cbe9d97
      sdong 创作于
      Summary:
      StringAppendOperatorTest right now runs in a mode where RUN_ALL_TESTS() is executed twice for the same test but different settings. This creates a problem with a tool that expects every test to run once. Fix it by using a parameterized test instead.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6930
      
      Test Plan: Run the test and see it passed.
      
      Reviewed By: ltamasi
      
      Differential Revision: D21874145
      
      fbshipit-source-id: 55520b2d7f1ba9f3cba1e2d087fe86f43fb06145
      6cbe9d97
    • sdong's avatar
      Fix ThreadLocalTest.SequentialReadWriteTest failure when running individually (#6929) · 31bd2d79
      sdong 创作于
      Summary:
      When running ThreadLocalTest.SequentialReadWriteTest individually, the test fails with:
      
      ] ./thread_local_test --gtest_filter="*SequentialReadWriteTest*"
      Note: Google Test filter = *SequentialReadWriteTest*
      [==========] Running 1 test from 1 test case.
      [----------] Global test environment set-up.
      [----------] 1 test from ThreadLocalTest
      [ RUN      ] ThreadLocalTest.SequentialReadWriteTest
      internal_repo_rocksdb/repo/util/thread_local_test.cc:144: Failure
            Expected: IDChecker::PeekId()
            Which is: 3
      To be equal to: base_id + 1u
            Which is: 2
      [  FAILED  ] ThreadLocalTest.SequentialReadWriteTest (1 ms)
      [----------] 1 test from ThreadLocalTest (1 ms total)
      
      [----------] Global test environment tear-down
      [==========] 1 test from 1 test case ran. (1 ms total)
      [  PASSED  ] 0 tests.
      [  FAILED  ] 1 test, listed below:
      [  FAILED  ] ThreadLocalTest.SequentialReadWriteTest
      
       1 FAILED TEST
      
      It appears that when running as the first test, PeakId() was updated twice. I didn't dig into it why but it doesn't seem to break the contract. Relax the assertion to make it pass.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6929
      
      Test Plan: Run the test individually and as the whole thread_local_test
      
      Reviewed By: riversand963
      
      Differential Revision: D21873999
      
      fbshipit-source-id: 1dcb6a2e9c38b6afd848027308bfe633342b7548
      31bd2d79
    • mrambacher's avatar
      Fix two core dumps when files are missing (#6922) · e85cbdb4
      mrambacher 创作于
      Summary:
      The LDB create and drop column family commands failed to check if theere was a valid database prior to dereferencing it, leading to a core dump.
      
      The SstFileDumper prefetch code would dereference a file when the file did not exist as part of the Prefetch code.  This dereference was moved inside an st.ok() check.
      
      Tests were added for both failure conditions.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6922
      
      Reviewed By: gg814
      
      Differential Revision: D21884024
      
      Pulled By: pdillinger
      
      fbshipit-source-id: bddd45c299aa9dc7e928c17a37a96521f8c9149e
      e85cbdb4
    • sdong's avatar
      env_test */RunMany/* tests to run individually (#6931) · 0b45a68c
      sdong 创作于
      Summary:
      When run */RunMany/* tests individually, e.g. ChrootEnvWithDirectIO/EnvPosixTestWithParam.RunMany/0, they hang. It's because they insert to background thread pool without initializing them. Fix it.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6931
      
      Test Plan: Run ChrootEnvWithDirectIO/EnvPosixTestWithParam.RunMany/0 by itself and see it passes.
      
      Reviewed By: riversand963
      
      Differential Revision: D21875603
      
      fbshipit-source-id: 7f848174c1a660254a2b1f7e11cca5370793ba30
      0b45a68c
  7. 04 6月, 2020 14 次提交
    • Yanqin Jin's avatar
      Fix a typo (bug) when setting error during Flush (#6928) · 2f326183
      Yanqin Jin 创作于
      Summary:
      As title. The prior change to the line is a typo. Fixing it.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6928
      
      Test Plan: make check
      
      Reviewed By: zhichao-cao
      
      Differential Revision: D21873587
      
      Pulled By: riversand963
      
      fbshipit-source-id: f4837fc8792d7106bc230b7b499dfbb7a2847430
      2f326183
    • Zitan Chen's avatar
      API change: DB::OpenForReadOnly will not write to the file system unless... · 02df00d9
      Zitan Chen 创作于
      API change: DB::OpenForReadOnly will not write to the file system unless create_if_missing is true (#6900)
      
      Summary:
      DB::OpenForReadOnly will not write anything to the file system (i.e., create directories or files for the DB) unless create_if_missing is true.
      
      This change also fixes some subcommands of ldb, which write to the file system even if the purpose is for readonly.
      
      Two tests for this updated behavior of DB::OpenForReadOnly are also added.
      
      Other minor changes:
      1. Updated HISTORY.md to include this API change of DB::OpenForReadOnly;
      2. Updated the help information for the put and batchput subcommands of ldb with the option [--create_if_missing];
      3. Updated the comment of Env::DeleteDir to emphasize that it returns OK only if the directory to be deleted is empty.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6900
      
      Test Plan: passed make check; also manually tested a few ldb subcommands
      
      Reviewed By: pdillinger
      
      Differential Revision: D21822188
      
      Pulled By: gg814
      
      fbshipit-source-id: 604cc0f0d0326a937ee25a32cdc2b512f9a3be6e
      02df00d9
    • sdong's avatar
      Make sure core components not depend on gtest (#6921) · 055b4d25
      sdong 创作于
      Summary:
      We recently removed the dependencies of core components on gtest. Add a Travis test to make sure it doesn't regress. Change cmake setting so that the gtest related components are only included when tests, benchmarks or stress tools are included in the build. Add this build setting in Travis to confirm it.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6921
      
      Test Plan: See Travis passes
      
      Reviewed By: ajkr
      
      Differential Revision: D21863564
      
      fbshipit-source-id: df26f50a8305a04ff19ffa8069a1857ecee10289
      055b4d25
    • Stanislav Tkach's avatar
      Add (some) getters for options to the C API (#6925) · b7c825d5
      Stanislav Tkach 创作于
      Summary:
      Additionally I have extended the incomplete test added in the https://github.com/facebook/rocksdb/issues/6880.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6925
      
      Reviewed By: ajkr
      
      Differential Revision: D21869788
      
      Pulled By: pdillinger
      
      fbshipit-source-id: e9db80f259c57ca1bdcbc2c66cb938cb1ac26e48
      b7c825d5
    • sdong's avatar
      Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) · afa35188
      sdong 创作于
      Summary:
      This reverts commit 8d87e9ce.
      
      Based on offline discussions, it's too early to upgrade to gtest 1.10, as it prevents some developers from using an older version of gtest to integrate to some other systems. Revert it for now.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6923
      
      Reviewed By: pdillinger
      
      Differential Revision: D21864799
      
      fbshipit-source-id: d0726b1ff649fc911b9378f1763316200bd363fc
      afa35188
    • Hans Holmberg's avatar
      Route GetTestDirectory to FileSystem in CompositeEnvWrappers (#6896) · 0f85d163
      Hans Holmberg 创作于
      Summary:
      GetTestDirectory implies a file system operation (it creates the
      default test directory if missing), so it should be routed to
      the FileSystem rather than the Env.
      
      Also remove the GetTestDirectory implementation in the PosixEnv,
      since it overrides GetTestDirectory in CompositeEnv making it
      impossible to override with a custom FileSystem.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6896
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21868984
      
      Pulled By: ajkr
      
      fbshipit-source-id: e79bfef758d06dacef727c54b96abe62e78726fd
      0f85d163
    • hfrt456's avatar
      fix IsDirectory function in env_hdfs.cc (#6917) · f005dac2
      hfrt456 创作于
      Summary:
      fix IsDirectory function for hdfsEnv
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6917
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21865020
      
      Pulled By: riversand963
      
      fbshipit-source-id: ad69ed564d027b7bbdf4c693dd57cd02622fb3f8
      f005dac2
    • Levi Tamasi's avatar
      Mention the consistency check improvement in HISTORY.md (#6924) · 0b8c549b
      Levi Tamasi 创作于
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6924
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21865662
      
      Pulled By: ltamasi
      
      fbshipit-source-id: 83a01bcbb779cfba941154a36a9e735293a93211
      0b8c549b
    • Hao Chen's avatar
      correct level information in version_set.cc (#6920) · ffe08ffc
      Hao Chen 创作于
      Summary:
      fix these two issues https://github.com/facebook/rocksdb/issues/6912  and https://github.com/facebook/rocksdb/issues/6667
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6920
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21864885
      
      Pulled By: ajkr
      
      fbshipit-source-id: 10e21fc1851b67a59d44358f59c64fa5523bd263
      ffe08ffc
    • Anatoly Zhmur's avatar
      Add zstd_max_train_bytes to c interop (#6796) · 22e5c513
      Anatoly Zhmur 创作于
      Summary:
      Added setting of zstd_max_train_bytes compression option parameter to c interop.
      
      rocksdb_options_set_bottommost_compression_options was using bool parameter and thus not exported, updated it to unsigned char and added to c.h as well.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6796
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21611471
      
      Pulled By: ajkr
      
      fbshipit-source-id: caaaf153de934837ad9af283c7f8c025ff0b0cf5
      22e5c513
    • mrambacher's avatar
      Add OptionTypeInfo::Vector to parse/serialize vectors (#6424) · 0a17d953
      mrambacher 创作于
      Summary:
      The OptionTypeInfo::Vector method allows a vector<T> to be converted to/from strings via the options.
      
      The kVectorInt and kVectorCompressionType vectors were replaced with this methodology.
      
      As part of this change, the NextToken method was added to the OptionTypeInfo.  This method was refactored from code within the StringToMap function.
      
      Future types that could use this functionality include the EventListener vectors.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6424
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21832368
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: e1ca766faff139d54e6e8407a9ec09ece6517439
      0a17d953
    • Lucian Petrut's avatar
      Posix threads (#6865) · 172adce7
      Lucian Petrut 创作于
      
      
      Summary:
      Rocksdb is using the c++11 std::threads feature. The issue is that
      MINGW only supports it when using Posix threads.
      
      This change will allow rocksdb::port::WindowsThread to be replaced
      with std::thread, which in turn will allow Rocksdb to be cross
      compiled using MINGW.
      
      At the same time, we'll have to use GetCurrentProcessId instead of _getpid.
      
      Signed-off-by: default avatarLucian Petrut <lpetrut@cloudbasesolutions.com>
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6865
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21864285
      
      Pulled By: ajkr
      
      fbshipit-source-id: 0982eed313e7d34d351b1364c1ccc722da473205
      172adce7
    • Peter Dillinger's avatar
      Some fixes for gcc 4.8 and add to Travis (#6915) · 43f8a9dc
      Peter Dillinger 创作于
      Summary:
      People keep breaking the gcc 4.8 compilation due to different
      warnings for shadowing member functions with locals. Adding to Travis
      to keep compatibility. (gcc 4.8 is default on CentOS 7.)
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6915
      
      Test Plan: local and Travis
      
      Reviewed By: siying
      
      Differential Revision: D21842894
      
      Pulled By: pdillinger
      
      fbshipit-source-id: bdcd4385127ee5d1cc222d87e53fb3695c32a9d4
      43f8a9dc
    • Levi Tamasi's avatar
      Improve consistency checks in VersionBuilder (#6901) · 78e291b1
      Levi Tamasi 创作于
      Summary:
      The patch cleans up the code and improves the consistency checks around
      adding/deleting table files in `VersionBuilder`. Namely, it makes the checks
      stricter and improves them in the following ways:
      1) A table file can now only be deleted from the LSM tree using the level it
      resides on. Earlier, there was some unnecessary wiggle room for
      trivially moved files (they could be deleted using a lower level number than
      the actual one).
      2) A table file cannot be added to the tree if it is already present in the tree
      on any level (not just the target level). The earlier code only had an assertion
      (which is a no-op in release builds) that the newly added file is not already
      present on the target level.
      3) The above consistency checks around state transitions are now mandatory,
      as opposed to the earlier `CheckConsistencyForDeletes`, which was a no-op
      in release mode unless `force_consistency_checks` was set to `true`. The rationale
      here is that assuming that the initial state is consistent, a valid transition leads to a
      next state that is also consistent; however, an *invalid* transition offers no such
      guarantee. Hence it makes sense to validate the transitions unconditionally,
      and save `force_consistency_checks` for the paranoid checks that re-validate
      the entire state.
      4) The new checks build on the mechanism introduced in https://github.com/facebook/rocksdb/pull/6862,
      which enables us to efficiently look up the location (level and position within level)
      of files in a `Version` by file number. This makes the consistency checks much more
      efficient than the earlier `CheckConsistencyForDeletes`, which essentially
      performed a linear search.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6901
      
      Test Plan:
      Extended the unit tests and ran:
      
      `make check`
      `make whitebox_crash_test`
      
      Reviewed By: ajkr
      
      Differential Revision: D21822714
      
      Pulled By: ltamasi
      
      fbshipit-source-id: e2b29c8b6da1bf0f59004acc889e4870b2d18215
      78e291b1