1. 21 Apr, 2018 1 commit
  2. 23 Feb, 2018 2 commits
  3. 22 Jul, 2017 2 commits
  4. 19 Jul, 2017 1 commit
  5. 16 Jul, 2017 1 commit
  6. 28 Apr, 2017 1 commit
  7. 04 Apr, 2017 1 commit
  8. 18 Nov, 2016 1 commit
  9. 17 Nov, 2016 1 commit
  10. 21 Jul, 2016 1 commit
    • omegaga's avatar
      Only cache level 0 indexes and filter when opening table reader · e70020e4
      omegaga authored
      Summary: In T8216281 we decided to disable prefetching the index and filter during opening table handlers during startup (max_open_files = -1).
      
      Test Plan: Rely on `IndexAndFilterBlocksOfNewTableAddedToCache` to guarantee L0 indexes and filters are still cached and change `PinL0IndexAndFilterBlocksTest` to make sure other levels are not cached (maybe add one more test to test we don't cache other levels?)
      
      Reviewers: sdong, andrewkr
      
      Reviewed By: andrewkr
      
      Subscribers: andrewkr, dhruba
      
      Differential Revision: https://reviews.facebook.net/D59913
      e70020e4
  11. 20 Jul, 2016 1 commit
    • John Alexander's avatar
      New Statistics to track Compression/Decompression (#1197) · 9430333f
      John Alexander authored
      * Added new statistics and refactored to allow ioptions to be passed around as required to access environment and statistics pointers (and, as a convenient side effect, info_log pointer).
      
      * Prevent incrementing compression counter when compression is turned off in options.
      
      * Prevent incrementing compression counter when compression is turned off in options.
      
      * Added two more supported compression types to test code in db_test.cc
      
      * Prevent incrementing compression counter when compression is turned off in options.
      
      * Added new StatsLevel that excludes compression timing.
      
      * Fixed casting error in coding.h
      
      * Fixed CompressionStatsTest for new StatsLevel.
      
      * Removed unused variable that was breaking the Linux build
      9430333f
  12. 07 Apr, 2016 1 commit
    • Andrew Kryczka's avatar
      Embed column family name in SST file · 2391ef72
      Andrew Kryczka authored
      Summary:
      Added the column family name to the properties block. This property
      is omitted only if the property is unavailable, such as when RepairDB()
      writes SST files.
      
      In a next diff, I will change RepairDB to use this new property for
      deciding to which column family an existing SST file belongs. If this
      property is missing, it will add it to the "unknown" column family (same
      as its existing behavior).
      
      Test Plan:
      New unit test:
      
        $ ./db_table_properties_test --gtest_filter=DBTablePropertiesTest.GetColumnFamilyNameProperty
      
      Reviewers: IslamAbdelRahman, yhchiang, sdong
      
      Reviewed By: sdong
      
      Subscribers: andrewkr, dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D55605
      2391ef72
  13. 10 Feb, 2016 1 commit
  14. 12 Jan, 2016 1 commit
  15. 09 Jan, 2016 1 commit
    • sdong's avatar
      plain table reader: non-mmap mode to keep two recent buffers · 9a8e3f73
      sdong authored
      Summary: In plain table reader's non-mmap mode, we only keep the most recent read buffer. However, for binary search, it is likely we come back to a location to read. To avoid one pread in such a case, we keep two read buffers. It should cover most of the cases.
      
      Test Plan:
      1. run tests
      2. check the optimization works through strace when running
      ./table_reader_bench -mmap_read=false --num_keys2=1 -num_keys1=5000 -table_factory=plain_table --iterator --through_db
      
      Reviewers: anthony, rven, kradhakrishnan, igor, yhchiang, IslamAbdelRahman
      
      Reviewed By: IslamAbdelRahman
      
      Subscribers: leveldb, dhruba
      
      Differential Revision: https://reviews.facebook.net/D51171
      9a8e3f73
  16. 24 Sep, 2015 1 commit
    • sdong's avatar
      PlainTableReader to support non-mmap mode · df34aea3
      sdong authored
      Summary:
      PlainTableReader now only allows mmap-mode. Add the support to non-mmap mode for more flexibility.
      Refactor the codes to move all logic of reading data to PlainTableKeyDecoder, and consolidate the calls to Read() call and ReadVarint32() call. Implement the calls for both of mmap and non-mmap case seperately. For non-mmap mode, make copy of keys in several places when we need to move the buffer after reading the keys.
      
      Test Plan: Add the mode of non-mmap case in plain_table_db_test. Run it in valgrind mode too.
      
      Subscribers: leveldb, dhruba
      
      Differential Revision: https://reviews.facebook.net/D47187
      df34aea3
  17. 12 Sep, 2015 1 commit
  18. 01 Sep, 2015 1 commit
  19. 21 Jul, 2015 1 commit
  20. 18 Jul, 2015 2 commits
    • sdong's avatar
      Move rate_limiter, write buffering, most perf context instrumentation and most... · 6e9fbeb2
      sdong authored
      Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env
      
      Summary: We want to keep Env a think layer for better portability. Less platform dependent codes should be moved out of Env. In this patch, I create a wrapper of file readers and writers, and put rate limiting, write buffering, as well as most perf context instrumentation and random kill out of Env. It will make it easier to maintain multiple Env in the future.
      
      Test Plan: Run all existing unit tests.
      
      Reviewers: anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, igor
      
      Reviewed By: igor
      
      Subscribers: leveldb, dhruba
      
      Differential Revision: https://reviews.facebook.net/D42321
      6e9fbeb2
    • Igor Canadi's avatar
      Don't let flushes preempt compactions · 35ca5936
      Igor Canadi authored
      Summary:
      When we first started, max_background_flushes was 0 by default and compaction thread was executing flushes (since there was no flush thread). Then, we switched the default max_background_flushes to 1. However, we still support the case where there is no flush thread and flushes are done in compaction. This is making our code a bit more complicated. By not supporting this use-case we can make our code simpler.
      
      We have a special case that when you set max_background_flushes to 0, we
      schedule the flush to execute on the compaction thread.
      
      Test Plan: make check (there might be some unit tests that depend on this behavior)
      
      Reviewers: IslamAbdelRahman, yhchiang, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D41931
      35ca5936
  21. 20 Mar, 2015 1 commit
    • Igor Sugak's avatar
      rocksdb: Remove #include "util/string_util.h" from util/testharness.h · 9405b5ef
      Igor Sugak authored
      Summary:
      1. Manually deleted #include "util/string_util.h" from util/testharness.h
      2.
      ```
      % USE_CLANG=1 make all -j55 -k 2> build.log
      % perl -naF: -E 'say $F[0] if /: error:/' build.log | sort -u | xargs sed -i '/#include "util\/testharness.h"/i #include "util\/string_util.h"'
      ```
      
      Test Plan:
      Make sure make all completes with no errors.
      ```
      % make all -j55
      ```
      
      Reviewers: meyering, igor, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D35493
      9405b5ef
  22. 18 Mar, 2015 1 commit
    • Igor Sugak's avatar
      rocksdb: switch to gtest · b4b69e4f
      Igor Sugak authored
      Summary:
      Our existing test notation is very similar to what is used in gtest. It makes it easy to adopt what is different.
      In this diff I modify existing [[ https://code.google.com/p/googletest/wiki/Primer#Test_Fixtures:_Using_the_Same_Data_Configuration_for_Multiple_Te | test fixture ]] classes to inherit from `testing::Test`. Also for unit tests that use fixture class, `TEST` is replaced with `TEST_F` as required in gtest.
      
      There are several custom `main` functions in our existing tests. To make this transition easier, I modify all `main` functions to fallow gtest notation. But eventually we can remove them and use implementation of `main` that gtest provides.
      
      ```lang=bash
      % cat ~/transform
      #!/bin/sh
      files=$(git ls-files '*test\.cc')
      for file in $files
      do
        if grep -q "rocksdb::test::RunAllTests()" $file
        then
          if grep -Eq '^class \w+Test {' $file
          then
            perl -pi -e 's/^(class \w+Test) {/${1}: public testing::Test {/g' $file
            perl -pi -e 's/^(TEST)/${1}_F/g' $file
          fi
          perl -pi -e 's/(int main.*\{)/${1}::testing::InitGoogleTest(&argc, argv);/g' $file
          perl -pi -e 's/rocksdb::test::RunAllTests/RUN_ALL_TESTS/g' $file
        fi
      done
      % sh ~/transform
      % make format
      ```
      
      Second iteration of this diff contains only scripted changes.
      
      Third iteration contains manual changes to fix last errors and make it compilable.
      
      Test Plan:
      Build and notice no errors.
      ```lang=bash
      % USE_CLANG=1 make check -j55
      ```
      Tests are still testing.
      
      Reviewers: meyering, sdong, rven, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D35157
      b4b69e4f
  23. 17 Mar, 2015 1 commit
    • Igor Sugak's avatar
      rocksdb: Replace ASSERT* with EXPECT* in functions that does not return void value · 9fd6edf8
      Igor Sugak authored
      Summary:
      gtest does not use exceptions to fail a unit test by design, and `ASSERT*`s are implemented using `return`. As a consequence we cannot use `ASSERT*` in a function that does not return `void` value ([[ https://code.google.com/p/googletest/wiki/AdvancedGuide#Assertion_Placement | 1]]), and have to fix our existing code. This diff does this in a generic way, with no manual changes.
      
      In order to detect all existing `ASSERT*` that are used in functions that doesn't return void value, I change the code to generate compile errors for such cases.
      
      In `util/testharness.h` I defined `EXPECT*` assertions, the same way as `ASSERT*`, and redefined `ASSERT*` to return `void`. Then executed:
      
      ```lang=bash
      % USE_CLANG=1 make all -j55 -k 2> build.log
      % perl -naF: -e 'print "-- -number=".$F[1]." ".$F[0]."\n" if  /: error:/' \
      build.log | xargs -L 1 perl -spi -e 's/ASSERT/EXPECT/g if $. == $number'
      % make format
      ```
      After that I reverted back change to `ASSERT*` in `util/testharness.h`. But preserved introduced `EXPECT*`, which is the same as `ASSERT*`. This will be deleted once switched to gtest.
      
      This diff is independent and contains manual changes only in `util/testharness.h`.
      
      Test Plan:
      Make sure all tests are passing.
      ```lang=bash
      % USE_CLANG=1 make check
      ```
      
      Reviewers: igor, lgalanis, sdong, yufei.zhu, rven, meyering
      
      Reviewed By: meyering
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D33333
      9fd6edf8
  24. 25 Nov, 2014 1 commit
  25. 12 Nov, 2014 1 commit
    • Igor Canadi's avatar
      Turn on -Wshorten-64-to-32 and fix all the errors · 767777c2
      Igor Canadi authored
      Summary:
      We need to turn on -Wshorten-64-to-32 for mobile. See D1671432 (internal phabricator) for details.
      
      This diff turns on the warning flag and fixes all the errors. There were also some interesting errors that I might call bugs, especially in plain table. Going forward, I think it makes sense to have this flag turned on and be very very careful when converting 64-bit to 32-bit variables.
      
      Test Plan: compiles
      
      Reviewers: ljin, rven, yhchiang, sdong
      
      Reviewed By: yhchiang
      
      Subscribers: bobbaldwin, dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D28689
      767777c2
  26. 30 Oct, 2014 1 commit
  27. 05 Sep, 2014 1 commit
    • Lei Jin's avatar
      introduce ImmutableOptions · 5665e5e2
      Lei Jin authored
      Summary:
      As a preparation to support updating some options dynamically, I'd like
      to first introduce ImmutableOptions, which is a subset of Options that
      cannot be changed during the course of a DB lifetime without restart.
      
      ColumnFamily will keep both Options and ImmutableOptions. Any component
      below ColumnFamily should only take ImmutableOptions in their
      constructor. Other options should be taken from APIs, which will be
      allowed to adjust dynamically.
      
      I am yet to make changes to memtable and other related classes to take
      ImmutableOptions in their ctor. That can be done in a seprate diff as
      this one is already pretty big.
      
      Test Plan: make all check
      
      Reviewers: yhchiang, igor, sdong
      
      Reviewed By: sdong
      
      Subscribers: leveldb, dhruba
      
      Differential Revision: https://reviews.facebook.net/D22545
      5665e5e2
  28. 07 Aug, 2014 1 commit
    • sdong's avatar
      Add DB property "rocksdb.estimate-table-readers-mem" · 1242bfca
      sdong authored
      Summary:
      Add a DB Property "rocksdb.estimate-table-readers-mem" to return estimated memory usage by all loaded table readers, other than allocated from block cache.
      
      Refactor the property codes to allow getting property from a version, with DB mutex not acquired.
      
      Test Plan: Add several checks of this new property in existing codes for various cases.
      
      Reviewers: yhchiang, ljin
      
      Reviewed By: ljin
      
      Subscribers: xjin, igor, leveldb
      
      Differential Revision: https://reviews.facebook.net/D20733
      1242bfca
  29. 19 Jul, 2014 1 commit
  30. 18 Jul, 2014 1 commit
  31. 11 Jul, 2014 1 commit
    • Stanislau Hlebik's avatar
      Removing NewTotalOrderPlainTableFactory · 30c81e77
      Stanislau Hlebik authored
      Summary:
      Seems like NewTotalOrderPlainTableFactory is useless and is semantically incorrect.
      Total order mode indicator is prefix_extractor == nullptr,
      but NewTotalOrderPlainTableFactory doesn't set it to be nullptr. That's why some tests
      in plain_table_db_tests is incorrect.
      
      Test Plan: make all check
      
      Reviewers: sdong
      
      Reviewed By: sdong
      
      Subscribers: leveldb
      
      Differential Revision: https://reviews.facebook.net/D19587
      30c81e77
  32. 02 Jul, 2014 1 commit
    • sdong's avatar
      HashLinkList memtable switches a bucket to a skip list to reduce performance outliers · 9c332aa1
      sdong authored
      Summary:
      In this patch, we enhance HashLinkList memtable to reduce performance outliers when a bucket contains too many entries. We switch to skip list for this case to enable binary search.
      
      Add threshold_use_skiplist parameter to determine when a bucket needs to switch to skip list.
      
      The new data structure is documented in comments in the codes.
      
      Test Plan:
      make all check
      set threshold_use_skiplist in several tests
      
      Reviewers: yhchiang, haobo, ljin
      
      Reviewed By: yhchiang, ljin
      
      Subscribers: nkg-, xjin, dhruba, yhchiang, leveldb
      
      Differential Revision: https://reviews.facebook.net/D19299
      9c332aa1
  33. 19 Jun, 2014 1 commit
  34. 18 Jun, 2014 3 commits
  35. 10 Jun, 2014 1 commit
    • sdong's avatar
      Clean PlainTableReader's variables for better data locality · 80f409ea
      sdong authored
      Summary:
      Clean PlainTableReader's data structures:
      (1) inline bloom_ (in order to do this, change DynamicBloom to allow lazy initialization)
      (2) remove some variables only used when initialization from the class
      (3) put variables not used in normal read code paths to the end of the class and reference prefix_extractor directly
      (4) make Options a reference.
      
      Test Plan: make all check
      
      Reviewers: haobo, ljin
      
      Reviewed By: ljin
      
      Subscribers: igor, yhchiang, dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D18891
      80f409ea