1. 12 Jun, 2020 3 commits
    • Levi Tamasi's avatar
      Revisit the handling of the case when a file is re-added to the same level (#6939) · d854abad
      Levi Tamasi authored
      https://github.com/facebook/rocksdb/pull/6901 subtly changed the handling of the corner case
      when a table file is deleted from a level, then re-added to the same level. (Note: this
      should be extremely rare; one scenario that comes to mind is a trivial move followed by
      a call to `ReFitLevel` that moves the file back to the original level.) Before that change,
      a new `FileMetaData` object was created as a result of this sequence; after the change,
      the original `FileMetaData` was essentially resurrected (since the deletion and the addition
      simply cancel each other out with the change). This patch restores the original behavior,
      which is more intuitive considering the interface, and in sync with how trivial moves are handled.
      (Also note that `FileMetaData` contains some mutable data members, the values of which
      might be different in the resurrected object and the freshly created one.)
      The PR also fixes a bug in this area: with the original pre-6901 code, `VersionBuilder`
      would add the same file twice to the same level in the scenario described above.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6939
      Test Plan: `make check`
      Reviewed By: ajkr
      Differential Revision: D21905580
      Pulled By: ltamasi
      fbshipit-source-id: da07ae45384ecf3c6c53506d106432d88a7ec9df
    • Levi Tamasi's avatar
      Turn DBTest2.CompressionFailures into a parameterized test (#6968) · 722ebba8
      Levi Tamasi authored
      `DBTest2.CompressionFailures` currently tests many configurations
      sequentially using nested loops, which often leads to timeouts
      in our test system. The patch turns it into a parameterized test
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6968
      Test Plan: `make check`
      Reviewed By: siying
      Differential Revision: D22006954
      Pulled By: ltamasi
      fbshipit-source-id: f71f2f7108086b7651ecfce3d79a7fab24620b2c
    • Zhichao Cao's avatar
      Ingest SST files with checksum information (#6891) · b3585a11
      Zhichao Cao authored
      Application can ingest SST files with file checksum information, such that during ingestion, DB is able to check data integrity and identify of the SST file. The PR introduces generate_and_verify_file_checksum to IngestExternalFileOption to control if the ingested checksum information should be verified with the generated checksum.
          1. If generate_and_verify_file_checksum options is *FALSE*: *1)* if DB does not enable SST file checksum, the checksum information ingested will be ignored; *2)* if DB enables the SST file checksum and the checksum function name matches the checksum function name in DB, we trust the ingested checksum, store it in Manifest. If the checksum function name does not match, we treat that as an error and fail the IngestExternalFile() call.
          2. If generate_and_verify_file_checksum options is *TRUE*: *1)* if DB does not enable SST file checksum, the checksum information ingested will be ignored; *2)* if DB enable the SST file checksum, we will use the checksum generator from DB to calculate the checksum for each ingested SST files after they are copied or moved. Then, compare the checksum results with the ingested checksum information: _A)_ if the checksum function name does not match, _verification always report true_ and we store the DB generated checksum information in Manifest. _B)_ if the checksum function name mach, and checksum match, ingestion continues and stores the checksum information in the Manifest. Otherwise, terminate file ingestion and report file corruption.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6891
      Test Plan: added unit test, pass make asan_check
      Reviewed By: pdillinger
      Differential Revision: D21935988
      Pulled By: zhichao-cao
      fbshipit-source-id: 7b55f486632db467e76d72602218d0658aa7f6ed
  2. 11 Jun, 2020 2 commits
    • Levi Tamasi's avatar
      Use a per-thread path for the export directory in import_column_family_test (#6962) · fbe2d259
      Levi Tamasi authored
      This is required so that the test cases can safely be run in parallel.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6962
      Test Plan: `make check`
      Reviewed By: zhichao-cao
      Differential Revision: D21980060
      Pulled By: ltamasi
      fbshipit-source-id: 616b7a0b686155d3874848b9098c67ad3f47efcc
    • Andrew Kryczka's avatar
      save a key comparison in block seeks (#6646) · e6be168a
      Andrew Kryczka authored
      This saves up to two key comparisons in block seeks. The first key
      comparison saved is a redundant key comparison against the restart key
      where the linear scan starts. This comparison is saved in all cases
      except when the found key is in the first restart interval. The
      second key comparison saved is a redundant key comparison against the
      restart key where the linear scan ends. This is only saved in cases
      where all keys in the restart interval are less than the target
      (probability roughly `1/restart_interval`).
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6646
      Test Plan:
      ran a benchmark with mostly default settings and counted key comparisons
      before: `user_key_comparison_count = 19399529`
      after: `user_key_comparison_count = 18431498`
      setup command:
      $ TEST_TMPDIR=/dev/shm/dbbench ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -max_background_jobs=12 -level_compaction_dynamic_level_bytes=true -num=10000000
      benchmark command:
      $ TEST_TMPDIR=/dev/shm/dbbench/ ./db_bench -use_existing_db=true -benchmarks=readrandom -disable_auto_compactions=true -num=10000000 -compression_type=none -reads=1000000 -perf_level=3
      Reviewed By: pdillinger
      Differential Revision: D20849707
      Pulled By: ajkr
      fbshipit-source-id: 1f01c5cd99ea771fd27974046e37b194f1cdcfac
  3. 10 Jun, 2020 3 commits
  4. 09 Jun, 2020 5 commits
    • sdong's avatar
      Introduce some Linux build to CircleCI (#6937) · 6a8ddd37
      sdong authored
      Moving towards the long term goal of moving most CI build to CircleCI when possible, add some Linux tests in CircleCI. This is not all what we can include to CircleCI. For example, Java builds are not includ
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6937
      Test Plan: Watch CI build results.
      Reviewed By: pdillinger
      Differential Revision: D21941605
      fbshipit-source-id: db6aead3c45f523386d4fb30d224cfde573cccad
    • anand76's avatar
      Fix a bug in looking up duplicate keys with MultiGet (#6953) · 1fb3593f
      anand76 authored
      When MultiGet is called with duplicate keys, and the key matches the
      largest key in an SST file and the value type is merge, only the first
      instance of the duplicate key is returned with correct results. This is
      due to the incorrect assumption that if a key in a batch is equal to the
      largest key in the file, the next key cannot be present in that file.
      Add a new unit test
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6953
      Reviewed By: cheng-chang
      Differential Revision: D21935898
      Pulled By: anand1976
      fbshipit-source-id: a2cc327a15150e23fd997546ca64d1c33021cb4c
    • Levi Tamasi's avatar
      Add convenience method GetFileMetaDataByNumber (#6940) · f5e64945
      Levi Tamasi authored
      The patch adds a convenience method `GetFileMetaDataByNumber` that
      builds on the `FileLocation` functionality introduced recently (see
      https://github.com/facebook/rocksdb/pull/6862). This method makes it possible to
      retrieve the `FileMetaData` directly as opposed to having to go through
      `LevelFiles` and friends.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6940
      Test Plan: `make check`
      Reviewed By: cheng-chang
      Differential Revision: D21905946
      Pulled By: ltamasi
      fbshipit-source-id: af99e19de21242b2b4a87594a535c6028d16ee72
    • Zitan Chen's avatar
      Implement a new subcommand "identify" for sst_dump (#6943) · 119b26fa
      Zitan Chen authored
      Implemented a subcommand of sst_dump called identify, which determines whether a file is an SST file or identifies and lists all the SST files in a directory;
      This update also fixes the problem that sst_dump exits with a success state even if target file/directory does not exist/is not an SST file/is empty/is corrupted.
      One test is added to sst_dump_test.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6943
      Test Plan: Passed make check and a few manual tests
      Reviewed By: pdillinger
      Differential Revision: D21928985
      Pulled By: gg814
      fbshipit-source-id: 9a8b48e0cf1a0e96b13f42b690aba8ad981afad3
    • Zhichao Cao's avatar
      decouple the dependency of trace_analyzer_test unit test (#6941) · fb08330f
      Zhichao Cao authored
      Since gflags use the global variable to store the flags passed in. In the unit test, if we git one flag per unit test, the result is that all the flags are combined together in the following tests. Therefore, it has the dependency. In this PR, we pass the full arguments each time to ensure that the old arguments will be overwritten by the new one such that the dependency is removed.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6941
      Test Plan: make asan_check. run each unit test in trace_analyzer_test independently and in arbitrary orders.
      Reviewed By: pdillinger
      Differential Revision: D21909176
      Pulled By: zhichao-cao
      fbshipit-source-id: dca550a0a4a205c30faa620e258a020a3b5b4e13
  5. 08 Jun, 2020 1 commit
    • Yanqin Jin's avatar
      Remove unnecessary inclusion of version_edit.h in env (#6952) · 3020df9d
      Yanqin Jin authored
      In db_options.c, we should avoid including header files in the `db` directory to avoid introducing unnecessary dependency. The reason why `version_edit.h` has been included in `db_options.cc` is because we need two constants, `kUnknownChecksum` and `kUnknownChecksumFuncName`. We can put these two constants as `constexpr` in the public header `file_checksum.h`.
      Test plan (devserver):
      make check
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6952
      Reviewed By: zhichao-cao
      Differential Revision: D21925341
      Pulled By: riversand963
      fbshipit-source-id: 2902f3b74c97f0cf16c58ad24c095c787c3a40e2
  6. 06 Jun, 2020 8 commits
  7. 05 Jun, 2020 7 commits
    • Peter Dillinger's avatar
      Fix some defects reported by Coverity Scan (#6933) · aaece2a9
      Peter Dillinger authored
      Confusing checks for null that are never null
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6933
      Test Plan: make check
      Reviewed By: cheng-chang
      Differential Revision: D21885466
      Pulled By: pdillinger
      fbshipit-source-id: 4b48e03c2a33727f2702b0d12292f9fda5a3c475
    • Peter Dillinger's avatar
      Fix more defects reported by Coverity Scan (#6935) · c7432cc3
      Peter Dillinger authored
      Mostly uninitialized values: some probably written before use, but some seem like bugs. Also, destructor needs to be virtual, and possible use-after-free in test
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6935
      Test Plan: make check
      Reviewed By: siying
      Differential Revision: D21885484
      Pulled By: pdillinger
      fbshipit-source-id: e2e7cb0a0cf196f2b55edd16f0634e81f6cc8e08
    • Yanqin Jin's avatar
      Close file to avoid file-descriptor leakage (#6936) · a8170d77
      Yanqin Jin authored
      When operation on an open file descriptor fails, we should close the file descriptor.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6936
      Test Plan: make check
      Reviewed By: pdillinger
      Differential Revision: D21885458
      Pulled By: riversand963
      fbshipit-source-id: ba077a76b256a8537f21e22e4ec198f45390bf50
    • sdong's avatar
      Make StringAppendOperatorTest a parameterized test (#6930) · 6cbe9d97
      sdong authored
      StringAppendOperatorTest right now runs in a mode where RUN_ALL_TESTS() is executed twice for the same test but different settings. This creates a problem with a tool that expects every test to run once. Fix it by using a parameterized test instead.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6930
      Test Plan: Run the test and see it passed.
      Reviewed By: ltamasi
      Differential Revision: D21874145
      fbshipit-source-id: 55520b2d7f1ba9f3cba1e2d087fe86f43fb06145
    • sdong's avatar
      Fix ThreadLocalTest.SequentialReadWriteTest failure when running individually (#6929) · 31bd2d79
      sdong authored
      When running ThreadLocalTest.SequentialReadWriteTest individually, the test fails with:
      ] ./thread_local_test --gtest_filter="*SequentialReadWriteTest*"
      Note: Google Test filter = *SequentialReadWriteTest*
      [==========] Running 1 test from 1 test case.
      [----------] Global test environment set-up.
      [----------] 1 test from ThreadLocalTest
      [ RUN      ] ThreadLocalTest.SequentialReadWriteTest
      internal_repo_rocksdb/repo/util/thread_local_test.cc:144: Failure
            Expected: IDChecker::PeekId()
            Which is: 3
      To be equal to: base_id + 1u
            Which is: 2
      [  FAILED  ] ThreadLocalTest.SequentialReadWriteTest (1 ms)
      [----------] 1 test from ThreadLocalTest (1 ms total)
      [----------] Global test environment tear-down
      [==========] 1 test from 1 test case ran. (1 ms total)
      [  PASSED  ] 0 tests.
      [  FAILED  ] 1 test, listed below:
      [  FAILED  ] ThreadLocalTest.SequentialReadWriteTest
       1 FAILED TEST
      It appears that when running as the first test, PeakId() was updated twice. I didn't dig into it why but it doesn't seem to break the contract. Relax the assertion to make it pass.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6929
      Test Plan: Run the test individually and as the whole thread_local_test
      Reviewed By: riversand963
      Differential Revision: D21873999
      fbshipit-source-id: 1dcb6a2e9c38b6afd848027308bfe633342b7548
    • mrambacher's avatar
      Fix two core dumps when files are missing (#6922) · e85cbdb4
      mrambacher authored
      The LDB create and drop column family commands failed to check if theere was a valid database prior to dereferencing it, leading to a core dump.
      The SstFileDumper prefetch code would dereference a file when the file did not exist as part of the Prefetch code.  This dereference was moved inside an st.ok() check.
      Tests were added for both failure conditions.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6922
      Reviewed By: gg814
      Differential Revision: D21884024
      Pulled By: pdillinger
      fbshipit-source-id: bddd45c299aa9dc7e928c17a37a96521f8c9149e
    • sdong's avatar
      env_test */RunMany/* tests to run individually (#6931) · 0b45a68c
      sdong authored
      When run */RunMany/* tests individually, e.g. ChrootEnvWithDirectIO/EnvPosixTestWithParam.RunMany/0, they hang. It's because they insert to background thread pool without initializing them. Fix it.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6931
      Test Plan: Run ChrootEnvWithDirectIO/EnvPosixTestWithParam.RunMany/0 by itself and see it passes.
      Reviewed By: riversand963
      Differential Revision: D21875603
      fbshipit-source-id: 7f848174c1a660254a2b1f7e11cca5370793ba30
  8. 04 Jun, 2020 11 commits