1. 12 6月, 2020 3 次提交
    • Levi Tamasi's avatar
      Revisit the handling of the case when a file is re-added to the same level (#6939) · d854abad
      Levi Tamasi 创作于
      Summary:
      https://github.com/facebook/rocksdb/pull/6901 subtly changed the handling of the corner case
      when a table file is deleted from a level, then re-added to the same level. (Note: this
      should be extremely rare; one scenario that comes to mind is a trivial move followed by
      a call to `ReFitLevel` that moves the file back to the original level.) Before that change,
      a new `FileMetaData` object was created as a result of this sequence; after the change,
      the original `FileMetaData` was essentially resurrected (since the deletion and the addition
      simply cancel each other out with the change). This patch restores the original behavior,
      which is more intuitive considering the interface, and in sync with how trivial moves are handled.
      (Also note that `FileMetaData` contains some mutable data members, the values of which
      might be different in the resurrected object and the freshly created one.)
      The PR also fixes a bug in this area: with the original pre-6901 code, `VersionBuilder`
      would add the same file twice to the same level in the scenario described above.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6939
      
      Test Plan: `make check`
      
      Reviewed By: ajkr
      
      Differential Revision: D21905580
      
      Pulled By: ltamasi
      
      fbshipit-source-id: da07ae45384ecf3c6c53506d106432d88a7ec9df
      d854abad
    • Levi Tamasi's avatar
      Turn DBTest2.CompressionFailures into a parameterized test (#6968) · 722ebba8
      Levi Tamasi 创作于
      Summary:
      `DBTest2.CompressionFailures` currently tests many configurations
      sequentially using nested loops, which often leads to timeouts
      in our test system. The patch turns it into a parameterized test
      instead.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6968
      
      Test Plan: `make check`
      
      Reviewed By: siying
      
      Differential Revision: D22006954
      
      Pulled By: ltamasi
      
      fbshipit-source-id: f71f2f7108086b7651ecfce3d79a7fab24620b2c
      722ebba8
    • Zhichao Cao's avatar
      Ingest SST files with checksum information (#6891) · b3585a11
      Zhichao Cao 创作于
      Summary:
      Application can ingest SST files with file checksum information, such that during ingestion, DB is able to check data integrity and identify of the SST file. The PR introduces generate_and_verify_file_checksum to IngestExternalFileOption to control if the ingested checksum information should be verified with the generated checksum.
      
          1. If generate_and_verify_file_checksum options is *FALSE*: *1)* if DB does not enable SST file checksum, the checksum information ingested will be ignored; *2)* if DB enables the SST file checksum and the checksum function name matches the checksum function name in DB, we trust the ingested checksum, store it in Manifest. If the checksum function name does not match, we treat that as an error and fail the IngestExternalFile() call.
          2. If generate_and_verify_file_checksum options is *TRUE*: *1)* if DB does not enable SST file checksum, the checksum information ingested will be ignored; *2)* if DB enable the SST file checksum, we will use the checksum generator from DB to calculate the checksum for each ingested SST files after they are copied or moved. Then, compare the checksum results with the ingested checksum information: _A)_ if the checksum function name does not match, _verification always report true_ and we store the DB generated checksum information in Manifest. _B)_ if the checksum function name mach, and checksum match, ingestion continues and stores the checksum information in the Manifest. Otherwise, terminate file ingestion and report file corruption.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6891
      
      Test Plan: added unit test, pass make asan_check
      
      Reviewed By: pdillinger
      
      Differential Revision: D21935988
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: 7b55f486632db467e76d72602218d0658aa7f6ed
      b3585a11
  2. 11 6月, 2020 2 次提交
    • Levi Tamasi's avatar
      Use a per-thread path for the export directory in import_column_family_test (#6962) · fbe2d259
      Levi Tamasi 创作于
      Summary:
      This is required so that the test cases can safely be run in parallel.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6962
      
      Test Plan: `make check`
      
      Reviewed By: zhichao-cao
      
      Differential Revision: D21980060
      
      Pulled By: ltamasi
      
      fbshipit-source-id: 616b7a0b686155d3874848b9098c67ad3f47efcc
      fbe2d259
    • Andrew Kryczka's avatar
      save a key comparison in block seeks (#6646) · e6be168a
      Andrew Kryczka 创作于
      Summary:
      This saves up to two key comparisons in block seeks. The first key
      comparison saved is a redundant key comparison against the restart key
      where the linear scan starts. This comparison is saved in all cases
      except when the found key is in the first restart interval. The
      second key comparison saved is a redundant key comparison against the
      restart key where the linear scan ends. This is only saved in cases
      where all keys in the restart interval are less than the target
      (probability roughly `1/restart_interval`).
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6646
      
      Test Plan:
      ran a benchmark with mostly default settings and counted key comparisons
      
      before: `user_key_comparison_count = 19399529`
      after: `user_key_comparison_count = 18431498`
      
      setup command:
      
      ```
      $ TEST_TMPDIR=/dev/shm/dbbench ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -max_background_jobs=12 -level_compaction_dynamic_level_bytes=true -num=10000000
      ```
      
      benchmark command:
      
      ```
      $ TEST_TMPDIR=/dev/shm/dbbench/ ./db_bench -use_existing_db=true -benchmarks=readrandom -disable_auto_compactions=true -num=10000000 -compression_type=none -reads=1000000 -perf_level=3
      ```
      
      Reviewed By: pdillinger
      
      Differential Revision: D20849707
      
      Pulled By: ajkr
      
      fbshipit-source-id: 1f01c5cd99ea771fd27974046e37b194f1cdcfac
      e6be168a
  3. 10 6月, 2020 3 次提交
  4. 09 6月, 2020 5 次提交
    • sdong's avatar
      Introduce some Linux build to CircleCI (#6937) · 6a8ddd37
      sdong 创作于
      Summary:
      Moving towards the long term goal of moving most CI build to CircleCI when possible, add some Linux tests in CircleCI. This is not all what we can include to CircleCI. For example, Java builds are not includ
      ed.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6937
      
      Test Plan: Watch CI build results.
      
      Reviewed By: pdillinger
      
      Differential Revision: D21941605
      
      fbshipit-source-id: db6aead3c45f523386d4fb30d224cfde573cccad
      6a8ddd37
    • anand76's avatar
      Fix a bug in looking up duplicate keys with MultiGet (#6953) · 1fb3593f
      anand76 创作于
      Summary:
      When MultiGet is called with duplicate keys, and the key matches the
      largest key in an SST file and the value type is merge, only the first
      instance of the duplicate key is returned with correct results. This is
      due to the incorrect assumption that if a key in a batch is equal to the
      largest key in the file, the next key cannot be present in that file.
      
      Tests:
      Add a new unit test
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6953
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21935898
      
      Pulled By: anand1976
      
      fbshipit-source-id: a2cc327a15150e23fd997546ca64d1c33021cb4c
      1fb3593f
    • Levi Tamasi's avatar
      Add convenience method GetFileMetaDataByNumber (#6940) · f5e64945
      Levi Tamasi 创作于
      Summary:
      The patch adds a convenience method `GetFileMetaDataByNumber` that
      builds on the `FileLocation` functionality introduced recently (see
      https://github.com/facebook/rocksdb/pull/6862). This method makes it possible to
      retrieve the `FileMetaData` directly as opposed to having to go through
      `LevelFiles` and friends.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6940
      
      Test Plan: `make check`
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21905946
      
      Pulled By: ltamasi
      
      fbshipit-source-id: af99e19de21242b2b4a87594a535c6028d16ee72
      f5e64945
    • Zitan Chen's avatar
      Implement a new subcommand "identify" for sst_dump (#6943) · 119b26fa
      Zitan Chen 创作于
      Summary:
      Implemented a subcommand of sst_dump called identify, which determines whether a file is an SST file or identifies and lists all the SST files in a directory;
      
      This update also fixes the problem that sst_dump exits with a success state even if target file/directory does not exist/is not an SST file/is empty/is corrupted.
      
      One test is added to sst_dump_test.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6943
      
      Test Plan: Passed make check and a few manual tests
      
      Reviewed By: pdillinger
      
      Differential Revision: D21928985
      
      Pulled By: gg814
      
      fbshipit-source-id: 9a8b48e0cf1a0e96b13f42b690aba8ad981afad3
      119b26fa
    • Zhichao Cao's avatar
      decouple the dependency of trace_analyzer_test unit test (#6941) · fb08330f
      Zhichao Cao 创作于
      Summary:
      Since gflags use the global variable to store the flags passed in. In the unit test, if we git one flag per unit test, the result is that all the flags are combined together in the following tests. Therefore, it has the dependency. In this PR, we pass the full arguments each time to ensure that the old arguments will be overwritten by the new one such that the dependency is removed.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6941
      
      Test Plan: make asan_check. run each unit test in trace_analyzer_test independently and in arbitrary orders.
      
      Reviewed By: pdillinger
      
      Differential Revision: D21909176
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: dca550a0a4a205c30faa620e258a020a3b5b4e13
      fb08330f
  5. 08 6月, 2020 1 次提交
    • Yanqin Jin's avatar
      Remove unnecessary inclusion of version_edit.h in env (#6952) · 3020df9d
      Yanqin Jin 创作于
      Summary:
      In db_options.c, we should avoid including header files in the `db` directory to avoid introducing unnecessary dependency. The reason why `version_edit.h` has been included in `db_options.cc` is because we need two constants, `kUnknownChecksum` and `kUnknownChecksumFuncName`. We can put these two constants as `constexpr` in the public header `file_checksum.h`.
      
      Test plan (devserver):
      make check
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6952
      
      Reviewed By: zhichao-cao
      
      Differential Revision: D21925341
      
      Pulled By: riversand963
      
      fbshipit-source-id: 2902f3b74c97f0cf16c58ad24c095c787c3a40e2
      3020df9d
  6. 06 6月, 2020 8 次提交
  7. 05 6月, 2020 7 次提交
    • Peter Dillinger's avatar
      Fix some defects reported by Coverity Scan (#6933) · aaece2a9
      Peter Dillinger 创作于
      Summary:
      Confusing checks for null that are never null
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6933
      
      Test Plan: make check
      
      Reviewed By: cheng-chang
      
      Differential Revision: D21885466
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 4b48e03c2a33727f2702b0d12292f9fda5a3c475
      aaece2a9
    • Peter Dillinger's avatar
      Fix more defects reported by Coverity Scan (#6935) · c7432cc3
      Peter Dillinger 创作于
      Summary:
      Mostly uninitialized values: some probably written before use, but some seem like bugs. Also, destructor needs to be virtual, and possible use-after-free in test
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6935
      
      Test Plan: make check
      
      Reviewed By: siying
      
      Differential Revision: D21885484
      
      Pulled By: pdillinger
      
      fbshipit-source-id: e2e7cb0a0cf196f2b55edd16f0634e81f6cc8e08
      c7432cc3
    • Yanqin Jin's avatar
      Close file to avoid file-descriptor leakage (#6936) · a8170d77
      Yanqin Jin 创作于
      Summary:
      When operation on an open file descriptor fails, we should close the file descriptor.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6936
      
      Test Plan: make check
      
      Reviewed By: pdillinger
      
      Differential Revision: D21885458
      
      Pulled By: riversand963
      
      fbshipit-source-id: ba077a76b256a8537f21e22e4ec198f45390bf50
      a8170d77
    • sdong's avatar
      Make StringAppendOperatorTest a parameterized test (#6930) · 6cbe9d97
      sdong 创作于
      Summary:
      StringAppendOperatorTest right now runs in a mode where RUN_ALL_TESTS() is executed twice for the same test but different settings. This creates a problem with a tool that expects every test to run once. Fix it by using a parameterized test instead.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6930
      
      Test Plan: Run the test and see it passed.
      
      Reviewed By: ltamasi
      
      Differential Revision: D21874145
      
      fbshipit-source-id: 55520b2d7f1ba9f3cba1e2d087fe86f43fb06145
      6cbe9d97
    • sdong's avatar
      Fix ThreadLocalTest.SequentialReadWriteTest failure when running individually (#6929) · 31bd2d79
      sdong 创作于
      Summary:
      When running ThreadLocalTest.SequentialReadWriteTest individually, the test fails with:
      
      ] ./thread_local_test --gtest_filter="*SequentialReadWriteTest*"
      Note: Google Test filter = *SequentialReadWriteTest*
      [==========] Running 1 test from 1 test case.
      [----------] Global test environment set-up.
      [----------] 1 test from ThreadLocalTest
      [ RUN      ] ThreadLocalTest.SequentialReadWriteTest
      internal_repo_rocksdb/repo/util/thread_local_test.cc:144: Failure
            Expected: IDChecker::PeekId()
            Which is: 3
      To be equal to: base_id + 1u
            Which is: 2
      [  FAILED  ] ThreadLocalTest.SequentialReadWriteTest (1 ms)
      [----------] 1 test from ThreadLocalTest (1 ms total)
      
      [----------] Global test environment tear-down
      [==========] 1 test from 1 test case ran. (1 ms total)
      [  PASSED  ] 0 tests.
      [  FAILED  ] 1 test, listed below:
      [  FAILED  ] ThreadLocalTest.SequentialReadWriteTest
      
       1 FAILED TEST
      
      It appears that when running as the first test, PeakId() was updated twice. I didn't dig into it why but it doesn't seem to break the contract. Relax the assertion to make it pass.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6929
      
      Test Plan: Run the test individually and as the whole thread_local_test
      
      Reviewed By: riversand963
      
      Differential Revision: D21873999
      
      fbshipit-source-id: 1dcb6a2e9c38b6afd848027308bfe633342b7548
      31bd2d79
    • mrambacher's avatar
      Fix two core dumps when files are missing (#6922) · e85cbdb4
      mrambacher 创作于
      Summary:
      The LDB create and drop column family commands failed to check if theere was a valid database prior to dereferencing it, leading to a core dump.
      
      The SstFileDumper prefetch code would dereference a file when the file did not exist as part of the Prefetch code.  This dereference was moved inside an st.ok() check.
      
      Tests were added for both failure conditions.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6922
      
      Reviewed By: gg814
      
      Differential Revision: D21884024
      
      Pulled By: pdillinger
      
      fbshipit-source-id: bddd45c299aa9dc7e928c17a37a96521f8c9149e
      e85cbdb4
    • sdong's avatar
      env_test */RunMany/* tests to run individually (#6931) · 0b45a68c
      sdong 创作于
      Summary:
      When run */RunMany/* tests individually, e.g. ChrootEnvWithDirectIO/EnvPosixTestWithParam.RunMany/0, they hang. It's because they insert to background thread pool without initializing them. Fix it.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6931
      
      Test Plan: Run ChrootEnvWithDirectIO/EnvPosixTestWithParam.RunMany/0 by itself and see it passes.
      
      Reviewed By: riversand963
      
      Differential Revision: D21875603
      
      fbshipit-source-id: 7f848174c1a660254a2b1f7e11cca5370793ba30
      0b45a68c
  8. 04 6月, 2020 11 次提交