1. 21 Apr, 2018 1 commit
  2. 23 Feb, 2018 2 commits
  3. 22 Jul, 2017 2 commits
  4. 16 Jul, 2017 1 commit
  5. 28 Apr, 2017 1 commit
  6. 04 Apr, 2017 1 commit
  7. 06 Jul, 2016 1 commit
  8. 22 Jun, 2016 1 commit
    • omegaga's avatar
      Add a read option to enable background purge when cleaning up iterators · c4e19b77
      omegaga authored
      Summary:
      Add a read option `background_purge_on_iterator_cleanup` to avoid deleting files in foreground when destroying iterators.
      Instead, a job is scheduled in high priority queue and would be executed in a separate background thread.
      
      Test Plan: Add a variant of PurgeObsoleteFileTest. Turn on background purge option in the new test, and use sleeping task to ensure files are deleted in background.
      
      Reviewers: IslamAbdelRahman, sdong
      
      Reviewed By: IslamAbdelRahman
      
      Subscribers: andrewkr, dhruba
      
      Differential Revision: https://reviews.facebook.net/D59499
      c4e19b77
  9. 01 Mar, 2016 1 commit
    • Andrew Kryczka's avatar
      Handle concurrent manifest update and backup creation · 69c471bd
      Andrew Kryczka authored
      Summary:
      Fixed two related race conditions in backup creation.
      
      (1) CreateNewBackup() uses DB::DisableFileDeletions() to prevent table files
      from being deleted while it is copying; however, the MANIFEST file could still
      rotate during this time. The fix is to stop deleting the old manifest in the
      rotation logic. It will be deleted safely later when PurgeObsoleteFiles() runs
      (can only happen when file deletions are enabled).
      
      (2) CreateNewBackup() did not account for the CURRENT file being mutable.
      This is significant because the files returned by GetLiveFiles() contain a
      particular manifest filename, but the manifest to which CURRENT refers can
      change at any time. This causes problems when CURRENT changes between the call
      to GetLiveFiles() and when it's copied to the backup directory. To workaround this, I
      manually forge a CURRENT file referring to the manifest filename returned in
      GetLiveFiles().
      
      (2) also applies to the checkpointing code, so let me know if this approach is
      good and I'll make the same change there.
      
      Test Plan:
      new test for roll manifest during backup creation.
      
      running the test before this change:
      
        $ ./backupable_db_test --gtest_filter=BackupableDBTest.ChangeManifestDuringBackupCreation
        ...
        IO error: /tmp/rocksdbtest-9383/backupable_db/MANIFEST-000001: No such file or directory
      
      running the test after this change:
      
        $ ./backupable_db_test --gtest_filter=BackupableDBTest.ChangeManifestDuringBackupCreation
        ...
        [ RUN      ] BackupableDBTest.ChangeManifestDuringBackupCreation
        [       OK ] BackupableDBTest.ChangeManifestDuringBackupCreation (2836 ms)
      
      Reviewers: IslamAbdelRahman, anthony, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D54711
      69c471bd
  10. 10 Feb, 2016 1 commit
  11. 16 Dec, 2015 1 commit
    • Gunnar Kudrjavets's avatar
      Fix minor bugs in delete operator, snprintf, and size_t usage · 97265f5f
      Gunnar Kudrjavets authored
      Summary:
      List of changes:
      
      1) Fix the snprintf() usage in cases where wrong variable was used to determine the output buffer size.
      
      2) Remove unnecessary checks before calling delete operator.
      
      3) Increase code correctness by using size_t type when getting vector's size.
      
      4) Unify the coding style by removing namespace::std usage at the top of the file to confirm to the majority usage.
      
      5) Fix various lint errors pointed out by 'arc lint'.
      
      Test Plan:
      Code review and build:
      
      git diff
      make clean
      make -j 32 commit-prereq
      arc lint
      
      Reviewers: kradhakrishnan, sdong, rven, anthony, yhchiang, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D51849
      97265f5f
  12. 21 Jul, 2015 2 commits
    • agiardullo's avatar
      Improved FileExists API · 06429408
      agiardullo authored
      Summary: Add new CheckFileExists method.  Considered changing the FileExists api but didn't want to break anyone's builds.
      
      Test Plan: unit tests
      
      Reviewers: yhchiang, igor, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D42003
      06429408
    • Islam AbdelRahman's avatar
      Skip unsupported tests in ROCKSDB_LITE · aa8ac644
      Islam AbdelRahman authored
      Summary:
      Skipping these tests in ROCKSDB_LITE since they are not supported
      json_document_test
      wal_manager_test
      ttl_test
      sst_dump_test
      deletefile_test
      compact_files_test
      prefix_test
      checkpoint_test
      
      Test Plan:
      json_document_test
      wal_manager_test
      ttl_test
      sst_dump_test
      deletefile_test
      compact_files_test
      prefix_test
      checkpoint_test
      
      Reviewers: igor, sdong, yhchiang, kradhakrishnan, anthony
      
      Reviewed By: anthony
      
      Subscribers: dhruba
      
      Differential Revision: https://reviews.facebook.net/D42573
      aa8ac644
  13. 18 Jul, 2015 1 commit
    • Igor Canadi's avatar
      Don't let flushes preempt compactions · 35ca5936
      Igor Canadi authored
      Summary:
      When we first started, max_background_flushes was 0 by default and compaction thread was executing flushes (since there was no flush thread). Then, we switched the default max_background_flushes to 1. However, we still support the case where there is no flush thread and flushes are done in compaction. This is making our code a bit more complicated. By not supporting this use-case we can make our code simpler.
      
      We have a special case that when you set max_background_flushes to 0, we
      schedule the flush to execute on the compaction thread.
      
      Test Plan: make check (there might be some unit tests that depend on this behavior)
      
      Reviewers: IslamAbdelRahman, yhchiang, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D41931
      35ca5936
  14. 18 Jun, 2015 1 commit
    • Islam AbdelRahman's avatar
      Use CompactRangeOptions for CompactRange · 12e030a9
      Islam AbdelRahman authored
      Summary:
      This diff update DB::CompactRange to use RangeCompactionOptions instead of using multiple parameters
      Old CompactRange is still available but deprecated
      
      Test Plan:
      make all check
      make rocksdbjava
      USE_CLANG=1 make all
      OPT=-DROCKSDB_LITE make release
      
      Reviewers: sdong, yhchiang, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba
      
      Differential Revision: https://reviews.facebook.net/D40209
      12e030a9
  15. 20 Mar, 2015 1 commit
    • Igor Sugak's avatar
      rocksdb: Remove #include "util/string_util.h" from util/testharness.h · 9405b5ef
      Igor Sugak authored
      Summary:
      1. Manually deleted #include "util/string_util.h" from util/testharness.h
      2.
      ```
      % USE_CLANG=1 make all -j55 -k 2> build.log
      % perl -naF: -E 'say $F[0] if /: error:/' build.log | sort -u | xargs sed -i '/#include "util\/testharness.h"/i #include "util\/string_util.h"'
      ```
      
      Test Plan:
      Make sure make all completes with no errors.
      ```
      % make all -j55
      ```
      
      Reviewers: meyering, igor, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D35493
      9405b5ef
  16. 18 Mar, 2015 1 commit
    • Igor Sugak's avatar
      rocksdb: switch to gtest · b4b69e4f
      Igor Sugak authored
      Summary:
      Our existing test notation is very similar to what is used in gtest. It makes it easy to adopt what is different.
      In this diff I modify existing [[ https://code.google.com/p/googletest/wiki/Primer#Test_Fixtures:_Using_the_Same_Data_Configuration_for_Multiple_Te | test fixture ]] classes to inherit from `testing::Test`. Also for unit tests that use fixture class, `TEST` is replaced with `TEST_F` as required in gtest.
      
      There are several custom `main` functions in our existing tests. To make this transition easier, I modify all `main` functions to fallow gtest notation. But eventually we can remove them and use implementation of `main` that gtest provides.
      
      ```lang=bash
      % cat ~/transform
      #!/bin/sh
      files=$(git ls-files '*test\.cc')
      for file in $files
      do
        if grep -q "rocksdb::test::RunAllTests()" $file
        then
          if grep -Eq '^class \w+Test {' $file
          then
            perl -pi -e 's/^(class \w+Test) {/${1}: public testing::Test {/g' $file
            perl -pi -e 's/^(TEST)/${1}_F/g' $file
          fi
          perl -pi -e 's/(int main.*\{)/${1}::testing::InitGoogleTest(&argc, argv);/g' $file
          perl -pi -e 's/rocksdb::test::RunAllTests/RUN_ALL_TESTS/g' $file
        fi
      done
      % sh ~/transform
      % make format
      ```
      
      Second iteration of this diff contains only scripted changes.
      
      Third iteration contains manual changes to fix last errors and make it compilable.
      
      Test Plan:
      Build and notice no errors.
      ```lang=bash
      % USE_CLANG=1 make check -j55
      ```
      Tests are still testing.
      
      Reviewers: meyering, sdong, rven, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D35157
      b4b69e4f
  17. 17 Mar, 2015 1 commit
    • Igor Sugak's avatar
      rocksdb: Replace ASSERT* with EXPECT* in functions that does not return void value · 9fd6edf8
      Igor Sugak authored
      Summary:
      gtest does not use exceptions to fail a unit test by design, and `ASSERT*`s are implemented using `return`. As a consequence we cannot use `ASSERT*` in a function that does not return `void` value ([[ https://code.google.com/p/googletest/wiki/AdvancedGuide#Assertion_Placement | 1]]), and have to fix our existing code. This diff does this in a generic way, with no manual changes.
      
      In order to detect all existing `ASSERT*` that are used in functions that doesn't return void value, I change the code to generate compile errors for such cases.
      
      In `util/testharness.h` I defined `EXPECT*` assertions, the same way as `ASSERT*`, and redefined `ASSERT*` to return `void`. Then executed:
      
      ```lang=bash
      % USE_CLANG=1 make all -j55 -k 2> build.log
      % perl -naF: -e 'print "-- -number=".$F[1]." ".$F[0]."\n" if  /: error:/' \
      build.log | xargs -L 1 perl -spi -e 's/ASSERT/EXPECT/g if $. == $number'
      % make format
      ```
      After that I reverted back change to `ASSERT*` in `util/testharness.h`. But preserved introduced `EXPECT*`, which is the same as `ASSERT*`. This will be deleted once switched to gtest.
      
      This diff is independent and contains manual changes only in `util/testharness.h`.
      
      Test Plan:
      Make sure all tests are passing.
      ```lang=bash
      % USE_CLANG=1 make check
      ```
      
      Reviewers: igor, lgalanis, sdong, yufei.zhu, rven, meyering
      
      Reviewed By: meyering
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D33333
      9fd6edf8
  18. 25 Nov, 2014 1 commit
  19. 22 Nov, 2014 1 commit
    • Yueh-Hsuan Chiang's avatar
      Fix leak when create_missing_column_families=true on ThreadStatus · 569853ed
      Yueh-Hsuan Chiang authored
      Summary:
      An entry of ConstantColumnFamilyInfo is created when:
      1. DB::Open
      2. CreateColumnFamily.
      
      However, there are cases that DB::Open could also call CreateColumnFamily
      when create_missing_column_families=true.  As a result, it will create
      duplicate ConstantColumnFamilyInfo and one of them would be leaked.
      
      Test Plan: ./deletefile_test
      
      Reviewers: igor, sdong, ljin
      
      Reviewed By: ljin
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D29307
      569853ed
  20. 22 Oct, 2014 1 commit
  21. 01 Oct, 2014 1 commit
  22. 19 Sep, 2014 1 commit
  23. 16 Nov, 2013 1 commit
    • Igor Canadi's avatar
      Start DeleteFileTest with clean plate · 21905dd4
      Igor Canadi authored
      Summary:
      Remove all the files from the test dir before the test. The test failed when there were some old files still in the directory, since it checks the file counts.
      This is what caused jenkins' test failures. It was running fine on my machine so it was hard to repro.
      
      Test Plan:
      1. create an extra 000001.log file in the test directory
      2. run a ./deletefile_test - test failes
      3. patch ./deletefile_test with this
      4. test succeeds
      
      Reviewers: haobo, dhruba
      
      Reviewed By: haobo
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D14097
      21905dd4
  24. 15 Nov, 2013 1 commit
    • Igor Canadi's avatar
      PurgeObsoleteFiles() unittest · a0ce3fd0
      Igor Canadi authored
      Summary:
      Created a unittest that verifies that automatic deletion performed by PurgeObsoleteFiles() works correctly.
      
      Also, few small fixes on the logic part -- call version_set_->GetObsoleteFiles() in FindObsoleteFiles() instead of on some arbitrary positions.
      
      Test Plan: Created a unit test
      
      Reviewers: dhruba, haobo, nkg-
      
      Reviewed By: haobo
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D14079
      a0ce3fd0
  25. 07 Nov, 2013 1 commit
    • shamdor's avatar
      WAL log retention policy based on archive size. · c2be2cba
      shamdor authored
      Summary:
      Archive cleaning will still happen every WAL_ttl seconds
      but archived logs will be deleted only if archive size
      is greater then a WAL_size_limit value.
      Empty archived logs will be deleted evety WAL_ttl.
      
      Test Plan:
      1. Unit tests pass.
      2. Benchmark.
      
      Reviewers: emayanke, dhruba, haobo, sdong, kailiu, igor
      
      Reviewed By: emayanke
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D13869
      c2be2cba
  26. 29 Oct, 2013 1 commit
    • Kai Liu's avatar
      Fix a valgrind warning · 7e91b86f
      Kai Liu authored
      Summary:
      A latest valgrind test found a recently added unit test has memory leak,
      which is because DB is not closed at the end of the test.
      
      Test Plan: re-run the valgrind locally and make sure there's no memory leakage any more.
      
      Reviewers: emayanke
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D13725
      7e91b86f
  27. 25 Oct, 2013 1 commit
    • Mayank Agarwal's avatar
      Unify DeleteFile and DeleteWalFiles · 56305221
      Mayank Agarwal authored
      Summary:
      This is to simplify rocksdb public APIs and improve the code quality.
      Created an additional parameter to ParseFileName for log sub type and improved the code for deleting a wal file.
      Wrote exhaustive unit-tests in delete_file_test
      Unification of other redundant APIs can be taken up in a separate diff
      
      Test Plan: Expanded delete_file test
      
      Reviewers: dhruba, haobo, kailiu, sdong
      
      Reviewed By: dhruba
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D13647
      56305221
  28. 23 Oct, 2013 1 commit
    • Mayank Agarwal's avatar
      Dbid feature · 9b50106f
      Mayank Agarwal authored
      Summary:
      Create a new type of file on startup if it doesn't already exist called DBID.
      This will store a unique number generated from boost library's uuid header file.
      The use-case is to identify the case of a db losing all its data and coming back up either empty or from an image(backup/live replica's recovery)
      the key point to note is that DBID is not stored in a backup or db snapshot
      It's preferable to use Boost for uuid because:
      1) A non-standard way of generating uuid is not good
      2) /proc/sys/kernel/random/uuid generates a uuid but only on linux environments and the solution would not be clean
      3) c++ doesn't have any direct way to get a uuid
      4) Boost is a very good library that was already having linkage in rocksdb from third-party
      Note: I had to update the TOOLCHAIN_REV in build files to get latest verison of boost from third-party as the older version had a bug.
      I had to put Wno-uninitialized in Makefile because boost-1.51 has an unitialized variable and rocksdb would not comiple otherwise. Latet open-source for boost is 1.54 but is not there in third-party. I have notified the concerned people in fbcode about it.
      @kailiu : While releasing to third-party, an additional dependency will need to be created for boost in TARGETS file. I can help identify.
      
      Test Plan:
      Expand db_test to test 2 cases
      1) Restarting db with Id file present - verify that no change to Id
      2)Restarting db with Id file deleted - verify that a different Id is there after reopen
      Also run make all check
      
      Reviewers: dhruba, haobo, kailiu, sdong
      
      Reviewed By: dhruba
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D13587
      9b50106f
  29. 17 Oct, 2013 1 commit
  30. 15 Oct, 2013 1 commit
    • Siying Dong's avatar
      Change Function names from Compaction->Flush When they really mean Flush · 88f2f890
      Siying Dong authored
      Summary: When I debug the unit test failures when enabling background flush thread, I feel the function names can be made clearer for people to understand. Also, if the names are fixed, in many places, some tests' bugs are obvious (and some of those tests are failing). This patch is to clean it up for future maintenance.
      
      Test Plan: Run test suites.
      
      Reviewers: haobo, dhruba, xjin
      
      Reviewed By: dhruba
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D13431
      88f2f890
  31. 05 Oct, 2013 1 commit
  32. 02 Sep, 2013 1 commit
    • Mayank Agarwal's avatar
      Fix build caused by DeleteFile not tolerating / at the beginning · ab5c5c28
      Mayank Agarwal authored
      Summary: db->DeleteFile calls ParseFileName to check name that was returned for sst file. Now, sst filename is returned using TableFileName which uses MakeFileName. This puts a / at the front of the name and ParseFileName doesn't like that. Changed ParseFileName to tolerate /s at the beginning. The test delet_file_test used to pass earlier because this behaviour of MakeFileName had been changed a while back to not return a / during which delete_file_test was checked in. But MakeFileName had to be reverted to add / at the front because GetLiveFiles used at many places outside rocksdb used the previous behaviour of MakeFileName.
      
      Test Plan: make;./delete_filetest;make all check
      
      Reviewers: dhruba, haobo, vamsi
      
      Reviewed By: dhruba
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D12663
      ab5c5c28
  33. 24 Aug, 2013 1 commit
  34. 23 Aug, 2013 1 commit
    • Simha Venkataramaiah's avatar
      Add APIs to query SST file metadata and to delete specific SST files · 60bf2b7d
      Simha Venkataramaiah authored
      Summary: An api to query the level, key ranges, size etc for each SST file and an api to delete a specific file from the db and all associated state in the bookkeeping datastructures.
      
      Notes: Editing the manifest version does not release the obsolete files right away. However deleting the file directly will mess up the iterator. We may need a more aggressive/timely file deletion api.
      
      I have used std::unique_ptr - will switch to boost:: since this is external. thoughts?
      
      Unit test is fragile right now as it expects the compaction at certain levels.
      
      Test Plan: unittest
      
      Reviewers: dhruba, vamsi, emayanke
      
      CC: zshao, leveldb, haobo
      
      Task ID: #
      
      Blame Rev:
      60bf2b7d