1. 20 Jun, 2015 2 commits
  2. 19 Jun, 2015 10 commits
    • Igor Canadi's avatar
      Disable CompressLevelCompaction() if Zlib is not supported · bf03f59c
      Igor Canadi authored
      Summary: CompressLevelCompaction() depends on Zlib. We should skip it when zlib is not present.
      
      Test Plan: `make check` without zlib
      
      Reviewers: yhchiang
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D40401
      bf03f59c
    • Yueh-Hsuan Chiang's avatar
      Make autovector_test runnable in ROCKSDB_LITE · df719d49
      Yueh-Hsuan Chiang authored
      Summary: Make autovector_test runnable in ROCKSDB_LITE
      
      Test Plan: autovector_test
      
      Reviewers: sdong, rven, anthony, kradhakrishnan, IslamAbdelRahman, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D40245
      df719d49
    • Yueh-Hsuan Chiang's avatar
      Block geodb_test in ROCKSDB_LITE · 4d6d4768
      Yueh-Hsuan Chiang authored
      Summary:
      Block geodb_test in ROCKSDB_LITE as geodb is not supported
      in ROCKSDB_LITE
      
      Test Plan: geodb_test
      
      Reviewers: sdong, rven, anthony, kradhakrishnan, IslamAbdelRahman, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D40335
      4d6d4768
    • Yueh-Hsuan Chiang's avatar
      Remove unused target --- compactor_test · 71b438c4
      Yueh-Hsuan Chiang authored
      Summary:
      Remove compactor_test, which depends on a directory not exist
      in our code base.
          make compactor_test
          GEN      util/build_version.cc
          GEN      util/build_version.cc
          make: *** No rule to make target `utilities/compaction/compactor_test.o', needed by `compactor_test'.  Stop.
      
      Test Plan: verify the output message of make compactor_test
      
      Reviewers: rven, anthony, kradhakrishnan, igor, IslamAbdelRahman, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D40341
      71b438c4
    • Yueh-Hsuan Chiang's avatar
      Block utilities/write_batch_with_index in ROCKSDB_LITE · eade498b
      Yueh-Hsuan Chiang authored
      Summary:
      Block utilities/write_batch_with_index in ROCKSDB_LITE as we
      don't include anly utilities in ROCKSDB_LITE
      
      Test Plan: write_batch_with_index_test
      
      Reviewers: rven, anthony, kradhakrishnan, IslamAbdelRahman, igor, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D40347
      eade498b
    • Igor Canadi's avatar
      Fail DB::Open() when the requested compression is not available · 760e9a94
      Igor Canadi authored
      Summary:
      Currently RocksDB silently ignores this issue and doesn't compress the data. Based on discussion, we agree that this is pretty bad because it can cause confusion for our users.
      
      This patch fails DB::Open() if we don't support the compression that is specified in the options.
      
      Test Plan: make check with LZ4 not present. If Snappy is not present all tests will just fail because Snappy is our default library. We should make Snappy the requirement, since without it our default DB::Open() fails.
      
      Reviewers: sdong, MarkCallaghan, rven, yhchiang
      
      Reviewed By: yhchiang
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D39687
      760e9a94
    • Aaron Feldman's avatar
      Add Cache.GetPinnedUsageUsage() · 69bb210d
      Aaron Feldman authored
      Summary:
        Add the funcion Cache.GetPinnedUsage() to return the memory size of entries
        that are in use by the system (that is, all the entries not in the LRU list).
      
      Test Plan:
        Run ./cache_test and examine PinnedUsageTest.
      
      Reviewers: tnovak, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba
      
      Differential Revision: https://reviews.facebook.net/D40305
      69bb210d
    • Islam AbdelRahman's avatar
      Skip bottommost level compaction if possible · 4eabbdb7
      Islam AbdelRahman authored
      Summary:
      This is https://reviews.facebook.net/D39999 but after introducing an option to force compaction the bottom most level
      
      Changes in this patch
      - Introduce force_bottommost_level_compaction to CompactRangeOptions that force compacting bottommost level during compaction
      - Skip bottommost level compaction if we dont have a compaction filter and force_bottommost_level_compaction options is not set
      
      Although tests pass on my machine but I suspect that there maybe some tests that I am not aware of that  should use force_bottommost_level_compaction to pass in a deterministic way
      
      Test Plan:
      make check
      adding new tests
      
      Reviewers: igor, sdong, yhchiang
      
      Reviewed By: yhchiang
      
      Subscribers: dhruba
      
      Differential Revision: https://reviews.facebook.net/D40059
      4eabbdb7
    • Igor Canadi's avatar
      Don't dump DBOptions for each column family · 4b8bb62f
      Igor Canadi authored
      Summary: Currently we dump DBOptions for each column family options we dump. This leads to duplicate lines in our LOG file. This diff fixes that.
      
      Test Plan: Check out the LOG
      
      Reviewers: sdong, rven, yhchiang
      
      Reviewed By: yhchiang
      
      Subscribers: IslamAbdelRahman, yoshinorim, dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D39729
      4b8bb62f
    • Poornima Chozhiyath Raman's avatar
      Merge branch 'master' of github.com:facebook/rocksdb · 176f0bed
      Poornima Chozhiyath Raman authored
      D40233: Replace %llu with format macros in ParsedInternalKey::DebugString())
      176f0bed
  3. 18 Jun, 2015 10 commits
    • Yueh-Hsuan Chiang's avatar
      Fixed a bug of CompactionStats in multi-level universal compaction case · bb1c74ce
      Yueh-Hsuan Chiang authored
      Summary:
      Universal compaction can involves in multiple levels.  However,
      the current implementation of bytes_readn and bytes_readnp1
      (and some other stats with postfix `n` and `np1`) assumes compaction
      can only have two levels.
      
      This patch fixes this bug and redefines bytes_readn and bytes_readnp1:
      * bytes_readnp1: the number of bytes read in the compaction output level.
      * bytes_readn: the total number of bytes read minus bytes_readnp1
      
      Test Plan: Add a test in compaction_job_stats_test
      
      Reviewers: igor, sdong, rven, anthony, kradhakrishnan, IslamAbdelRahman
      
      Reviewed By: IslamAbdelRahman
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D40239
      bb1c74ce
    • Poornima Chozhiyath Raman's avatar
      Merge branch 'master' of github.com:facebook/rocksdb · a66b8157
      Poornima Chozhiyath Raman authored
      D40233: Replace %llu with format macros in ParsedInternalKey::DebugString())
      a66b8157
    • Poornima Chozhiyath Raman's avatar
      Replace %llu with format macros in ParsedInternalKey::DebugString()) · f06be62f
      Poornima Chozhiyath Raman authored
      Test Plan: successfully compiled the code
      
      Reviewers: sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba
      
      Differential Revision: https://reviews.facebook.net/D40233
      f06be62f
    • Igor Canadi's avatar
      Add --benchmark_write_rate_limit option to db_bench · 2dc3910b
      Igor Canadi authored
      Summary:
      So far, we benchmarked RocksDB by writing as fast as possible. With this change, we're able to limit our write throughput, which should help us better understand how RocksDB performes under varying write workloads.
      
      Specifically, I'm currently interested in the shape of the graph that has write throughput on one axis and write rate on another. This should help us with designing our stall system, as we have started to do with D36351.
      
      Test Plan:
          $ ./db_bench --benchmarks=fillrandom --benchmark_write_rate_limit=1000000
          fillrandom   :     118.523 micros/op 8437 ops/sec;    0.9 MB/s
          $ ./db_bench --benchmarks=fillrandom --benchmark_write_rate_limit=2000000
          fillrandom   :      59.136 micros/op 16910 ops/sec;    1.9 MB/s
      
      Reviewers: MarkCallaghan, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D39759
      2dc3910b
    • Islam AbdelRahman's avatar
      Use CompactRangeOptions for CompactRange · 12e030a9
      Islam AbdelRahman authored
      Summary:
      This diff update DB::CompactRange to use RangeCompactionOptions instead of using multiple parameters
      Old CompactRange is still available but deprecated
      
      Test Plan:
      make all check
      make rocksdbjava
      USE_CLANG=1 make all
      OPT=-DROCKSDB_LITE make release
      
      Reviewers: sdong, yhchiang, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba
      
      Differential Revision: https://reviews.facebook.net/D40209
      12e030a9
    • Igor Canadi's avatar
      Move dockerbuild.sh to build_tools/ · c89369f5
      Igor Canadi authored
      Summary: That's where we keep build tools :)
      
      Test Plan: none
      
      Reviewers: sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D39741
      c89369f5
    • Igor Canadi's avatar
      Merge pull request #638 from HolodovAlexander/master · 4716ab4d
      Igor Canadi authored
      C api: human-readable statistics
      4716ab4d
    • Igor Canadi's avatar
      Clean up InstallSuperVersion · 25d60056
      Igor Canadi authored
      Summary:
      We go to great lengths to make sure MaybeScheduleFlushOrCompaction() is called outside of write thread. But anyway, it's still called in the mutex, so it's not that much cheaper.
      
      This diff removes the "optimization" and cleans up the code a bit.
      
      Test Plan: make check
      
      Reviewers: rven, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D40113
      25d60056
    • Yueh-Hsuan Chiang's avatar
      Only initialize the ThreadStatusData when necessary. · 1369f015
      Yueh-Hsuan Chiang authored
      Summary:
      Before this patch, any function call to ThreadStatusUtil might automatically initialize and register the thread status data.  However, if it is the user-thread making this call, the allocated thread-status-data will never be released as such threads are not managed by rocksdb.
      
      In this patch, I remove the automatic-initialization part.  Thread-status data is only initialized and uninitialized in Env during the thread creation and destruction.
      
      Test Plan:
      db_test
      thread_list_test
      listener_test
      
      Reviewers: igor, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D40017
      1369f015
    • Yueh-Hsuan Chiang's avatar
      Block c_test in ROCKSDB_LITE · 1a08d0be
      Yueh-Hsuan Chiang authored
      Summary: Block c_test in ROCKSDB_LITE as it's not supported in ROCKSDB_LITE.
      
      Test Plan: c_test
      
      Reviewers: sdong, rven, anthony, kradhakrishnan, IslamAbdelRahman, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D40257
      1a08d0be
  4. 17 Jun, 2015 1 commit
  5. 13 Jun, 2015 6 commits
  6. 12 Jun, 2015 6 commits
    • sdong's avatar
      Slow down writes by bytes written · 7842920b
      sdong authored
      Summary:
      We slow down data into the database to the rate of options.delayed_write_rate (a new option) with this patch.
      
      The thread synchronization approach I take is to still synchronize write controller by DB mutex and GetDelay() is inside DB mutex. Try to minimize the frequency of getting time in GetDelay(). I verified it through db_bench and it seems to work
      
      hard_rate_limit is deprecated.
      
      options.delayed_write_rate is still not dynamically changeable. Need to work on it as a follow-up.
      
      Test Plan: Add new unit tests in db_test
      
      Reviewers: yhchiang, rven, kradhakrishnan, anthony, MarkCallaghan, igor
      
      Reviewed By: igor
      
      Subscribers: ikabiljo, leveldb, dhruba
      
      Differential Revision: https://reviews.facebook.net/D36351
      7842920b
    • Igor Canadi's avatar
      Don't let two L0->L1 compactions run in parallel · a84df655
      Igor Canadi authored
      Summary: With experimental feature SuggestCompactRange() we don't restrict running two L0->L1 compactions in parallel. This diff fixes this.
      
      Test Plan: added a unit test to reproduce the failure. fixed the unit test
      
      Reviewers: yhchiang, rven, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D39981
      a84df655
    • Islam AbdelRahman's avatar
      Add largest sequence to FlushJobInfo · d6ce0f7c
      Islam AbdelRahman authored
      Summary:
      Adding largest sequence number to FlushJobInfo
      and passing flushed file metadata to NotifyOnFlushCompleted which include alot of other values that we may want to expose in FlushJobInfo
      
      Test Plan: make check
      
      Reviewers: igor, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba
      
      Differential Revision: https://reviews.facebook.net/D39927
      d6ce0f7c
    • Islam AbdelRahman's avatar
      fix clang build · ab455ce4
      Islam AbdelRahman authored
      ab455ce4
    • Yueh-Hsuan Chiang's avatar
      Add Env::GetThreadID(), which returns the ID of the current thread. · 3eddd1ab
      Yueh-Hsuan Chiang authored
      Summary:
      Add Env::GetThreadID(), which returns the ID of the current thread.
      
      In addition, make GetThreadList() and InfoLog use same unique ID for the same thread.
      
      Test Plan:
      db_test
      listener_test
      
      Reviewers: igor, rven, IslamAbdelRahman, kradhakrishnan, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba
      
      Differential Revision: https://reviews.facebook.net/D39735
      3eddd1ab
    • Islam AbdelRahman's avatar
      Handling edge cases for ReFitLevel · 73faa3d4
      Islam AbdelRahman authored
      Summary:
      Right now the level we pass to ReFitLevel is the maximum level with files (before compaction), there are multiple cases where this maximum level have changed after compaction
      - all files where in L0 (now maximum level is L1)
      - using kCompactionStyleUniversal (now maximum level in the last level)
      - level_compaction_dynamic_level_bytes ??
      
      We can handle each of these cases individually, but I felt it's safer to calculate max_level_with_files again if we want to do a ReFitLevel
      
      Test Plan:
      adding some tests
      make -j64 check
      
      Reviewers: igor, sdong
      
      Reviewed By: sdong
      
      Subscribers: ott, dhruba
      
      Differential Revision: https://reviews.facebook.net/D39663
      73faa3d4
  7. 11 Jun, 2015 5 commits