1. 26 May, 2018 3 commits
    • Fosco Marotto's avatar
      Update history for release of 5.13.2 · d25ca5fd
      Fosco Marotto authored
    • Adam Retter's avatar
      Fix an issue with unnecessary capture in lambda expressions · 2da2286d
      Adam Retter authored
      Closes https://github.com/facebook/rocksdb/issues/3900
      Replaces https://github.com/facebook/rocksdb/pull/3901
      I needed this to build v5.12.4 on Mac OS X (10.13.3).
      Closes https://github.com/facebook/rocksdb/pull/3904
      Differential Revision: D8169357
      Pulled By: sagar0
      fbshipit-source-id: 85faac42168796e7def9250d0c221a9a03b84476
    • Yanqin Jin's avatar
      Fix segfault caused by object premature destruction · 83973ddd
      Yanqin Jin authored
      Please refer to earlier discussion in [issue 3609](https://github.com/facebook/rocksdb/issues/3609).
      There was also an alternative fix in [PR 3888](https://github.com/facebook/rocksdb/pull/3888), but the proposed solution requires complex change.
      To summarize the cause of the problem. Upon creation of a column family, a `BlockBasedTableFactory` object is `new`ed and encapsulated by a `std::shared_ptr`. Since there is no other `std::shared_ptr` pointing to this `BlockBasedTableFactory`, when the column family is dropped, the `ColumnFamilyData` is `delete`d, causing the destructor of `std::shared_ptr`. Since there is no other `std::shared_ptr`, the underlying memory is also freed.
      Later when the db exits, it releases all the table readers, including the table readers that have been operating on the dropped column family. This needs to access the `table_options` owned by `BlockBasedTableFactory` that has already been deleted. Therefore, a segfault is raised.
      Previous workaround is to purge all obsolete files upon `ColumnFamilyData` destruction, which leads to a force release of table readers of the dropped column family. However this does not work when the user disables file deletion.
      Our solution in this PR is making a copy of `table_options` in `BlockBasedTable::Rep`. This solution increases memory copy and usage, but is much simpler.
      Test plan
      $ make -j16
      $ ./column_family_test --gtest_filter=ColumnFamilyTest.CreateDropAndDestroy:ColumnFamilyTest.CreateDropAndDestroyWithoutFileDeletion
      Expected behavior:
      All tests should pass.
      Closes https://github.com/facebook/rocksdb/pull/3898
      Differential Revision: D8149421
      Pulled By: riversand963
      fbshipit-source-id: eaecc2e064057ef607fbdd4cc275874f866c3438
  2. 24 May, 2018 2 commits
  3. 01 May, 2018 2 commits
    • Andrew Kryczka's avatar
      update history and version · c60df9d9
      Andrew Kryczka authored
    • Andrew Kryczka's avatar
      Avoid directory renames in BackupEngine · 747c8532
      Andrew Kryczka authored
      We used to name private directories like "1.tmp" while BackupEngine populated them, and then rename without the ".tmp" suffix (i.e., rename "1.tmp" to "1") after all files were copied. On glusterfs, directory renames like this require operations across many hosts, and partial failures have caused operational problems.
      Fortunately we don't need to rename private directories. We already have a meta-file that uses the tempfile-rename pattern to commit a backup atomically after all its files have been successfully copied. So we can copy private files directly to their final location, so now there's no directory rename.
      Closes https://github.com/facebook/rocksdb/pull/3749
      Differential Revision: D7705610
      Pulled By: ajkr
      fbshipit-source-id: fd724a28dd2bf993ce323a5f2cb7e7d6980cc346
  4. 25 Apr, 2018 1 commit
    • Gabriel Wicke's avatar
      Support lowering CPU priority of background threads · f5ee207c
      Gabriel Wicke authored
      Background activities like compaction can negatively affect
      latency of higher-priority tasks like request processing. To avoid this,
      rocksdb already lowers the IO priority of background threads on Linux
      systems. While this takes care of typical IO-bound systems, it does not
      help much when CPU (temporarily) becomes the bottleneck. This is
      especially likely when using more expensive compression settings.
      This patch adds an API to allow for lowering the CPU priority of
      background threads, modeled on the IO priority API. Benchmarks (see
      below) show significant latency and throughput improvements when CPU
      bound. As a result, workloads with some CPU usage bursts should benefit
      from lower latencies at a given utilization, or should be able to push
      utilization higher at a given request latency target.
      A useful side effect is that compaction CPU usage is now easily visible
      in common tools, allowing for an easier estimation of the contribution
      of compaction vs. request processing threads.
      As with IO priority, the implementation is limited to Linux, degrading
      to a no-op on other systems.
      Closes https://github.com/facebook/rocksdb/pull/3763
      Differential Revision: D7740096
      Pulled By: gwicke
      fbshipit-source-id: e5d32373e8dc403a7b0c2227023f9ce4f22b413c
  5. 17 Apr, 2018 2 commits
  6. 24 Mar, 2018 4 commits
  7. 21 Mar, 2018 1 commit
  8. 20 Mar, 2018 2 commits
    • Andrew Kryczka's avatar
      fix db_compaction_test when compression disabled · d1b26507
      Andrew Kryczka authored
      Previously, the compaction in `DBCompactionTestWithParam.ForceBottommostLevelCompaction` generated multiple files in no-compression use case, andone file in compression use case. I increased `target_file_size_base` so it generates one file in both use cases.
      Closes https://github.com/facebook/rocksdb/pull/3625
      Differential Revision: D7311885
      Pulled By: ajkr
      fbshipit-source-id: 97f249fa83a9924ac34357a4bb3189c969ecb107
    • Tobias Tschinkowitz's avatar
      Enable compilation on OpenBSD · ccb76136
      Tobias Tschinkowitz authored
      I modified the Makefile so that we can compile rocksdb on OpenBSD.
      The instructions for building have been added to INSTALL.md.
      The whole compilation process works fine like this on OpenBSD-current
      Closes https://github.com/facebook/rocksdb/pull/3617
      Differential Revision: D7323754
      Pulled By: siying
      fbshipit-source-id: 990037d1cc69138d22f85bd77ef4dc8c1ba9edea
  9. 19 Mar, 2018 1 commit
    • Yanqin Jin's avatar
      Fix the command used to generate ctags · 1139422d
      Yanqin Jin authored
      In original $ROCKSDB_HOME/Makefile, the command used to generate ctags is
      ctags * -R
      However, this failed to generate tags for me.
      I did some search on the usage of ctags command and found that it should be
      ctags -R .
      ctags -R *
      After the change, I can find the tags in vim using `:ts <identifier>`.
      Closes https://github.com/facebook/rocksdb/pull/3626
      Reviewed By: ajkr
      Differential Revision: D7320217
      Pulled By: riversand963
      fbshipit-source-id: e4cd8f8a67842370a2343f0213df3cbd07754111
  10. 17 Mar, 2018 3 commits
  11. 16 Mar, 2018 2 commits
    • Niv Dayan's avatar
      allowing CompactFiles to return new file names · da82aab1
      Niv Dayan authored
      This is a small API extension to allow the CompactFiles method to return the names of files that were created during the compaction.
      Closes https://github.com/facebook/rocksdb/pull/3608
      Differential Revision: D7275789
      Pulled By: siying
      fbshipit-source-id: 1ec0c3954a0f10cd877efb5f29f9be6c7b59e9ba
    • Sagar Vemuri's avatar
      Update version · cc118b0e
      Sagar Vemuri authored
      We missed updating version.h on master when cutting 5.11.fb and 5.12.fb branches. It should be the same as the version in the latest release branch (or should it be one more?).
      I noticed this when trying to run some upgrade/downgrade tests from 5.11 to some new code on master.
      Closes https://github.com/facebook/rocksdb/pull/3611
      Differential Revision: D7282917
      Pulled By: sagar0
      fbshipit-source-id: 205ee75b77c5b6bbcea95a272760b427025a4aba
  12. 15 Mar, 2018 2 commits
    • Andrew Kryczka's avatar
      Fix WAL corruption from checkpoint/backup race condition · 0cdaa1a8
      Andrew Kryczka authored
      `Writer::WriteBuffer` was always called at the beginning of checkpoint/backup. But that log writer has no internal synchronization, which meant the same buffer could be flushed twice in a race condition case, causing a WAL entry to be duplicated. Then subsequent WAL entries would be at unexpected offsets, causing the 32KB block boundaries to be overlapped and manifesting as a corruption.
      This PR fixes the behavior to only use `WriteBuffer` (via `FlushWAL`) in checkpoint/backup when manual WAL flush is enabled. In that case, users are responsible for providing synchronization between WAL flushes. We can also consider removing the call entirely.
      Closes https://github.com/facebook/rocksdb/pull/3603
      Differential Revision: D7277447
      Pulled By: ajkr
      fbshipit-source-id: 1b15bd7fd930511222b075418c10de0aaa70a35a
    • Yi Wu's avatar
      Blob DB: remove unreacheable code · 449627f0
      Yi Wu authored
      Fixing #3604.
      Closes https://github.com/facebook/rocksdb/pull/3606
      Reviewed By: siying
      Differential Revision: D7276604
      Pulled By: yiwu-arbug
      fbshipit-source-id: 915c5897b010d28956f369989e49e64785d1161f
  13. 14 Mar, 2018 5 commits
  14. 09 Mar, 2018 6 commits
  15. 08 Mar, 2018 2 commits
  16. 07 Mar, 2018 2 commits
    • Stuart's avatar
      Add rocksdb_open_with_ttl function in C API · f021f1d9
      Stuart authored
      Change-Id: Ie6f9b10bce459f6bf0ade0e5877264b4e10da3f5
      Signed-off-by: default avatarStuart <Stuart.Hu@emc.com>
      Closes https://github.com/facebook/rocksdb/pull/3553
      Differential Revision: D7144833
      Pulled By: sagar0
      fbshipit-source-id: 815225fa6e560d8a5bc47ffd0a98118b107ce264
    • amytai's avatar
      Disallow compactions if there isn't enough free space · 0a3db28d
      amytai authored
      This diff handles cases where compaction causes an ENOSPC error.
      This does not handle corner cases where another background job is started while compaction is running, and the other background job triggers ENOSPC, although we do allow the user to provision for these background jobs with SstFileManager::SetCompactionBufferSize.
      It also does not handle the case where compaction has finished and some other background job independently triggers ENOSPC.
      Usage: Functionality is inside SstFileManager. In particular, users should set SstFileManager::SetMaxAllowedSpaceUsage, which is the reference highwatermark for determining whether to cancel compactions.
      Closes https://github.com/facebook/rocksdb/pull/3449
      Differential Revision: D7016941
      Pulled By: amytai
      fbshipit-source-id: 8965ab8dd8b00972e771637a41b4e6c645450445