1. 22 Mar, 2018 5 commits
  2. 21 Mar, 2018 3 commits
  3. 20 Mar, 2018 2 commits
    • Andrew Kryczka's avatar
      fix db_compaction_test when compression disabled · d1b26507
      Andrew Kryczka authored
      Previously, the compaction in `DBCompactionTestWithParam.ForceBottommostLevelCompaction` generated multiple files in no-compression use case, andone file in compression use case. I increased `target_file_size_base` so it generates one file in both use cases.
      Closes https://github.com/facebook/rocksdb/pull/3625
      Differential Revision: D7311885
      Pulled By: ajkr
      fbshipit-source-id: 97f249fa83a9924ac34357a4bb3189c969ecb107
    • Tobias Tschinkowitz's avatar
      Enable compilation on OpenBSD · ccb76136
      Tobias Tschinkowitz authored
      I modified the Makefile so that we can compile rocksdb on OpenBSD.
      The instructions for building have been added to INSTALL.md.
      The whole compilation process works fine like this on OpenBSD-current
      Closes https://github.com/facebook/rocksdb/pull/3617
      Differential Revision: D7323754
      Pulled By: siying
      fbshipit-source-id: 990037d1cc69138d22f85bd77ef4dc8c1ba9edea
  4. 19 Mar, 2018 1 commit
    • Yanqin Jin's avatar
      Fix the command used to generate ctags · 1139422d
      Yanqin Jin authored
      In original $ROCKSDB_HOME/Makefile, the command used to generate ctags is
      ctags * -R
      However, this failed to generate tags for me.
      I did some search on the usage of ctags command and found that it should be
      ctags -R .
      ctags -R *
      After the change, I can find the tags in vim using `:ts <identifier>`.
      Closes https://github.com/facebook/rocksdb/pull/3626
      Reviewed By: ajkr
      Differential Revision: D7320217
      Pulled By: riversand963
      fbshipit-source-id: e4cd8f8a67842370a2343f0213df3cbd07754111
  5. 17 Mar, 2018 3 commits
  6. 16 Mar, 2018 2 commits
    • Niv Dayan's avatar
      allowing CompactFiles to return new file names · da82aab1
      Niv Dayan authored
      This is a small API extension to allow the CompactFiles method to return the names of files that were created during the compaction.
      Closes https://github.com/facebook/rocksdb/pull/3608
      Differential Revision: D7275789
      Pulled By: siying
      fbshipit-source-id: 1ec0c3954a0f10cd877efb5f29f9be6c7b59e9ba
    • Sagar Vemuri's avatar
      Update version · cc118b0e
      Sagar Vemuri authored
      We missed updating version.h on master when cutting 5.11.fb and 5.12.fb branches. It should be the same as the version in the latest release branch (or should it be one more?).
      I noticed this when trying to run some upgrade/downgrade tests from 5.11 to some new code on master.
      Closes https://github.com/facebook/rocksdb/pull/3611
      Differential Revision: D7282917
      Pulled By: sagar0
      fbshipit-source-id: 205ee75b77c5b6bbcea95a272760b427025a4aba
  7. 15 Mar, 2018 2 commits
    • Andrew Kryczka's avatar
      Fix WAL corruption from checkpoint/backup race condition · 0cdaa1a8
      Andrew Kryczka authored
      `Writer::WriteBuffer` was always called at the beginning of checkpoint/backup. But that log writer has no internal synchronization, which meant the same buffer could be flushed twice in a race condition case, causing a WAL entry to be duplicated. Then subsequent WAL entries would be at unexpected offsets, causing the 32KB block boundaries to be overlapped and manifesting as a corruption.
      This PR fixes the behavior to only use `WriteBuffer` (via `FlushWAL`) in checkpoint/backup when manual WAL flush is enabled. In that case, users are responsible for providing synchronization between WAL flushes. We can also consider removing the call entirely.
      Closes https://github.com/facebook/rocksdb/pull/3603
      Differential Revision: D7277447
      Pulled By: ajkr
      fbshipit-source-id: 1b15bd7fd930511222b075418c10de0aaa70a35a
    • Yi Wu's avatar
      Blob DB: remove unreacheable code · 449627f0
      Yi Wu authored
      Fixing #3604.
      Closes https://github.com/facebook/rocksdb/pull/3606
      Reviewed By: siying
      Differential Revision: D7276604
      Pulled By: yiwu-arbug
      fbshipit-source-id: 915c5897b010d28956f369989e49e64785d1161f
  8. 14 Mar, 2018 5 commits
  9. 09 Mar, 2018 6 commits
  10. 08 Mar, 2018 2 commits
  11. 07 Mar, 2018 9 commits
    • Stuart's avatar
      Add rocksdb_open_with_ttl function in C API · f021f1d9
      Stuart authored
      Change-Id: Ie6f9b10bce459f6bf0ade0e5877264b4e10da3f5
      Signed-off-by: default avatarStuart <Stuart.Hu@emc.com>
      Closes https://github.com/facebook/rocksdb/pull/3553
      Differential Revision: D7144833
      Pulled By: sagar0
      fbshipit-source-id: 815225fa6e560d8a5bc47ffd0a98118b107ce264
    • amytai's avatar
      Disallow compactions if there isn't enough free space · 0a3db28d
      amytai authored
      This diff handles cases where compaction causes an ENOSPC error.
      This does not handle corner cases where another background job is started while compaction is running, and the other background job triggers ENOSPC, although we do allow the user to provision for these background jobs with SstFileManager::SetCompactionBufferSize.
      It also does not handle the case where compaction has finished and some other background job independently triggers ENOSPC.
      Usage: Functionality is inside SstFileManager. In particular, users should set SstFileManager::SetMaxAllowedSpaceUsage, which is the reference highwatermark for determining whether to cancel compactions.
      Closes https://github.com/facebook/rocksdb/pull/3449
      Differential Revision: D7016941
      Pulled By: amytai
      fbshipit-source-id: 8965ab8dd8b00972e771637a41b4e6c645450445
    • Andrew Kryczka's avatar
      Enable subcompactions in manual level-based compaction · 20c508c1
      Andrew Kryczka authored
      This is the simplest way I could think of to speed up `CompactRange`. It works but isn't that optimal because it relies on the same `max_compaction_bytes` and `max_subcompactions` options that are used in other places. If it turns out to be useful we can allow overriding these in `CompactRangeOptions` in the future.
      Closes https://github.com/facebook/rocksdb/pull/3549
      Differential Revision: D7117634
      Pulled By: ajkr
      fbshipit-source-id: d0cd03d6bd0d2fd7ea3fb13cd3b8bf7c47d11e42
    • Fosco Marotto's avatar
      Add dual-license info to README.md · 3462c94b
      Fosco Marotto authored
      From #3417 and after talking to both GitHub and our open source legal team, the recommended approach was to explicitly state the dual-license in the readme.
      Changing the license files to accommodate the auto-detection is too much of a pain, would involve editing every code file header.
      Closes https://github.com/facebook/rocksdb/pull/3541
      Differential Revision: D7171111
      Pulled By: gfosco
      fbshipit-source-id: 0ee7b134446015228249efe991fa5e76526ca0b0
    • Andrew Kryczka's avatar
      support multiple db_paths in SstFileManager · 6a3eebba
      Andrew Kryczka authored
      Now that files scheduled for deletion are kept in the same directory, we don't need to constrain deletion scheduler to `db_paths[0]`. Previously this was done because there was a separate trash directory, and this constraint prevented files from being accidentally copied to another filesystem when they're scheduled for deletion.
      Closes https://github.com/facebook/rocksdb/pull/3544
      Differential Revision: D7093786
      Pulled By: ajkr
      fbshipit-source-id: 202f5c92d925eafebec1281fb95bb5828d33414f
    • Fosco Marotto's avatar
      uint64_t and size_t changes to compile for iOS · d518fe1d
      Fosco Marotto authored
      In attempting to build a static lib for use in iOS, I ran in to lots of type errors between uint64_t and size_t.  This PR contains the changes I made to get `TARGET_OS=IOS make static_lib` to succeed while also getting Xcode to build successfully with the resulting `librocksdb.a` library imported.
      This also compiles for me on macOS and tests fine, but I'm really not sure if I made the correct decisions about where to `static_cast` and where to change types.
      Also up for discussion: is iOS worth supporting?  Getting the static lib is just part one, we aren't providing any bridging headers or wrappers like the ObjectiveRocks project, it won't be a great experience.
      Closes https://github.com/facebook/rocksdb/pull/3503
      Differential Revision: D7106457
      Pulled By: gfosco
      fbshipit-source-id: 82ac2073de7e1f09b91f6b4faea91d18bd311f8e
    • Siying Dong's avatar
      Update TARGETS · 8bc41f4f
      Siying Dong authored
      Watch the build
      Closes https://github.com/facebook/rocksdb/pull/3533
      Differential Revision: D7063777
      Pulled By: siying
      fbshipit-source-id: db9cdfc362a8d281dada6513ab034a6d6f0d552e
    • Dmitri Smirnov's avatar
      Windows cumulative patch · c364eb42
      Dmitri Smirnov authored
      This patch addressed several issues.
        Portability including db_test std::thread -> port::Thread Cc: @
        and %z to ROCKSDB portable macro. Cc: maysamyabandeh
        Implement Env::AreFilesSame
        Make the implementation of file unique number more robust
        Get rid of C-runtime and go directly to Windows API when dealing
        with file primitives.
        Implement GetSectorSize() and aling unbuffered read on the value if
        Adjust Windows Logger for the new interface, implement CloseImpl() Cc: anand1976
        Fix test running script issue where $status var was of incorrect scope
        so the failures were swallowed and not reported.
        DestroyDB() creates a logger and opens a LOG file in the directory
        being cleaned up. This holds a lock on the folder and the cleanup is
        prevented. This fails one of the checkpoin tests. We observe the same in production.
        We close the log file in this change.
       Fix DBTest2.ReadAmpBitmapLiveInCacheAfterDBClose failure where the test
       attempts to open a directory with NewRandomAccessFile which does not
       work on Windows.
        Fix DBTest.SoftLimit as it is dependent on thread timing. CC: yiwu-arbug
      Closes https://github.com/facebook/rocksdb/pull/3552
      Differential Revision: D7156304
      Pulled By: siying
      fbshipit-source-id: 43db0a757f1dfceffeb2b7988043156639173f5b
    • Yi Wu's avatar
      Blob DB: Improve FIFO eviction · b864bc9b
      Yi Wu authored
      Improving blob db FIFO eviction with the following changes,
      * Change blob_dir_size to max_db_size. Take into account SST file size when computing DB size.
      * FIFO now only take into account live sst files and live blob files. It is normal for disk usage to go over max_db_size because there are obsolete sst files and blob files pending deletion.
      * FIFO eviction now also evict TTL blob files that's still open. It doesn't evict non-TTL blob files.
      * If FIFO is triggered, it will pass an expiration and the current sequence number to compaction filter. Compaction filter will then filter inlined keys to evict those with an earlier expiration and smaller sequence number. So call LSM FIFO.
      * Compaction filter also filter those blob indexes where corresponding blob file is gone.
      * Add an event listener to listen compaction/flush event and update sst file size.
      * Implement DB::Close() to make sure base db, as well as event listener and compaction filter, destruct before blob db.
      * More blob db statistics around FIFO.
      * Fix some locking issue when accessing a blob file.
      Closes https://github.com/facebook/rocksdb/pull/3556
      Differential Revision: D7139328
      Pulled By: yiwu-arbug
      fbshipit-source-id: ea5edb07b33dfceacb2682f4789bea61de28bbfa