Skip to content
GitLab
  1. 01 May, 2018 1 commit
    • Andrew Kryczka's avatar
      Avoid directory renames in BackupEngine · 747c8532
      Andrew Kryczka authored 
      Summary:
We used to name private directories like "1.tmp" while BackupEngine populated them, and then rename without the ".tmp" suffix (i.e., rename "1.tmp" to "1") after all files were copied. On glusterfs, directory renames like this require operations across many hosts, and partial failures have caused operational problems.

Fortunately we don't need to rename private directories. We already have a meta-file that uses the tempfile-rename pattern to commit a backup atomically after all its files have been successfully copied. So we can copy private files directly to their final location, so now there's no directory rename.
Closes https://github.com/facebook/rocksdb/pull/3749

Differential Revision: D7705610

Pulled By: ajkr

fbshipit-source-id: fd724a28dd2bf993ce323a5f2cb7e7d6980cc346
      747c8532
  3. 25 Apr, 2018 1 commit
    • Gabriel Wicke's avatar
      Support lowering CPU priority of background threads · f5ee207c
      Gabriel Wicke authored 
      Summary:
Background activities like compaction can negatively affect
latency of higher-priority tasks like request processing. To avoid this,
rocksdb already lowers the IO priority of background threads on Linux
systems. While this takes care of typical IO-bound systems, it does not
help much when CPU (temporarily) becomes the bottleneck. This is
especially likely when using more expensive compression settings.

This patch adds an API to allow for lowering the CPU priority of
background threads, modeled on the IO priority API. Benchmarks (see
below) show significant latency and throughput improvements when CPU
bound. As a result, workloads with some CPU usage bursts should benefit
from lower latencies at a given utilization, or should be able to push
utilization higher at a given request latency target.

A useful side effect is that compaction CPU usage is now easily visible
in common tools, allowing for an easier estimation of the contribution
of compaction vs. request processing threads.

As with IO priority, the implementation is limited to Linux, degrading
to a no-op on other systems.
Closes https://github.com/facebook/rocksdb/pull/3763

Differential Revision: D7740096

Pulled By: gwicke

fbshipit-source-id: e5d32373e8dc403a7b0c2227023f9ce4f22b413c
      f5ee207c
  5. 17 Apr, 2018 2 commits
  7. 24 Mar, 2018 4 commits
  9. 21 Mar, 2018 1 commit
  11. 20 Mar, 2018 2 commits
    • Andrew Kryczka's avatar
      fix db_compaction_test when compression disabled · d1b26507
      Andrew Kryczka authored 
      Summary:
Previously, the compaction in `DBCompactionTestWithParam.ForceBottommostLevelCompaction` generated multiple files in no-compression use case, andone file in compression use case. I increased `target_file_size_base` so it generates one file in both use cases.
Closes https://github.com/facebook/rocksdb/pull/3625

Differential Revision: D7311885

Pulled By: ajkr

fbshipit-source-id: 97f249fa83a9924ac34357a4bb3189c969ecb107
      d1b26507
    • Tobias Tschinkowitz's avatar
      Enable compilation on OpenBSD · ccb76136
      Tobias Tschinkowitz authored 
      Summary:
I modified the Makefile so that we can compile rocksdb on OpenBSD.
The instructions for building have been added to INSTALL.md.
The whole compilation process works fine like this on OpenBSD-current
Closes https://github.com/facebook/rocksdb/pull/3617

Differential Revision: D7323754

Pulled By: siying

fbshipit-source-id: 990037d1cc69138d22f85bd77ef4dc8c1ba9edea
      ccb76136
  13. 19 Mar, 2018 1 commit
    • Yanqin Jin's avatar
      Fix the command used to generate ctags · 1139422d
      Yanqin Jin authored 
      Summary:
In original $ROCKSDB_HOME/Makefile, the command used to generate ctags is
```
ctags * -R
```
However, this failed to generate tags for me.
I did some search on the usage of ctags command and found that it should be
```
ctags -R .
```
or
```
ctags -R *
```
After the change, I can find the tags in vim using `:ts <identifier>`.
Closes https://github.com/facebook/rocksdb/pull/3626

Reviewed By: ajkr

Differential Revision: D7320217

Pulled By: riversand963

fbshipit-source-id: e4cd8f8a67842370a2343f0213df3cbd07754111
      1139422d
  15. 17 Mar, 2018 3 commits
  17. 16 Mar, 2018 2 commits
    • Niv Dayan's avatar
      allowing CompactFiles to return new file names · da82aab1
      Niv Dayan authored 
      Summary:
This is a small API extension to allow the CompactFiles method to return the names of files that were created during the compaction.
Closes https://github.com/facebook/rocksdb/pull/3608

Differential Revision: D7275789

Pulled By: siying

fbshipit-source-id: 1ec0c3954a0f10cd877efb5f29f9be6c7b59e9ba
      da82aab1
    • Sagar Vemuri's avatar
      Update version · cc118b0e
      Sagar Vemuri authored 
      Summary:
We missed updating version.h on master when cutting 5.11.fb and 5.12.fb branches. It should be the same as the version in the latest release branch (or should it be one more?).

I noticed this when trying to run some upgrade/downgrade tests from 5.11 to some new code on master.
Closes https://github.com/facebook/rocksdb/pull/3611

Differential Revision: D7282917

Pulled By: sagar0

fbshipit-source-id: 205ee75b77c5b6bbcea95a272760b427025a4aba
      cc118b0e
  19. 15 Mar, 2018 2 commits
    • Andrew Kryczka's avatar
      Fix WAL corruption from checkpoint/backup race condition · 0cdaa1a8
      Andrew Kryczka authored 
      Summary:
`Writer::WriteBuffer` was always called at the beginning of checkpoint/backup. But that log writer has no internal synchronization, which meant the same buffer could be flushed twice in a race condition case, causing a WAL entry to be duplicated. Then subsequent WAL entries would be at unexpected offsets, causing the 32KB block boundaries to be overlapped and manifesting as a corruption.

This PR fixes the behavior to only use `WriteBuffer` (via `FlushWAL`) in checkpoint/backup when manual WAL flush is enabled. In that case, users are responsible for providing synchronization between WAL flushes. We can also consider removing the call entirely.
Closes https://github.com/facebook/rocksdb/pull/3603

Differential Revision: D7277447

Pulled By: ajkr

fbshipit-source-id: 1b15bd7fd930511222b075418c10de0aaa70a35a
      0cdaa1a8
    • Yi Wu's avatar
      Blob DB: remove unreacheable code · 449627f0
      Yi Wu authored 
      Summary:
Fixing #3604.
Closes https://github.com/facebook/rocksdb/pull/3606

Reviewed By: siying

Differential Revision: D7276604

Pulled By: yiwu-arbug

fbshipit-source-id: 915c5897b010d28956f369989e49e64785d1161f
      449627f0
  21. 14 Mar, 2018 5 commits
  23. 09 Mar, 2018 6 commits
  25. 08 Mar, 2018 2 commits
  27. 07 Mar, 2018 8 commits
    • Stuart's avatar
      Add rocksdb_open_with_ttl function in C API · f021f1d9
      Stuart authored 
      

Summary:
Change-Id: Ie6f9b10bce459f6bf0ade0e5877264b4e10da3f5
Signed-off-by: default avatarStuart <Stuart.Hu@emc.com>
Closes https://github.com/facebook/rocksdb/pull/3553

Differential Revision: D7144833

Pulled By: sagar0

fbshipit-source-id: 815225fa6e560d8a5bc47ffd0a98118b107ce264
      f021f1d9
    • amytai's avatar
      Disallow compactions if there isn't enough free space · 0a3db28d
      amytai authored 
      Summary:
This diff handles cases where compaction causes an ENOSPC error.
This does not handle corner cases where another background job is started while compaction is running, and the other background job triggers ENOSPC, although we do allow the user to provision for these background jobs with SstFileManager::SetCompactionBufferSize.
It also does not handle the case where compaction has finished and some other background job independently triggers ENOSPC.

Usage: Functionality is inside SstFileManager. In particular, users should set SstFileManager::SetMaxAllowedSpaceUsage, which is the reference highwatermark for determining whether to cancel compactions.
Closes https://github.com/facebook/rocksdb/pull/3449

Differential Revision: D7016941

Pulled By: amytai

fbshipit-source-id: 8965ab8dd8b00972e771637a41b4e6c645450445
      0a3db28d
    • Andrew Kryczka's avatar
      Enable subcompactions in manual level-based compaction · 20c508c1
      Andrew Kryczka authored 
      Summary:
This is the simplest way I could think of to speed up `CompactRange`. It works but isn't that optimal because it relies on the same `max_compaction_bytes` and `max_subcompactions` options that are used in other places. If it turns out to be useful we can allow overriding these in `CompactRangeOptions` in the future.
Closes https://github.com/facebook/rocksdb/pull/3549

Differential Revision: D7117634

Pulled By: ajkr

fbshipit-source-id: d0cd03d6bd0d2fd7ea3fb13cd3b8bf7c47d11e42
      20c508c1
    • Fosco Marotto's avatar
      Add dual-license info to README.md · 3462c94b
      Fosco Marotto authored 
      Summary:
From #3417 and after talking to both GitHub and our open source legal team, the recommended approach was to explicitly state the dual-license in the readme.

Changing the license files to accommodate the auto-detection is too much of a pain, would involve editing every code file header.
Closes https://github.com/facebook/rocksdb/pull/3541

Differential Revision: D7171111

Pulled By: gfosco

fbshipit-source-id: 0ee7b134446015228249efe991fa5e76526ca0b0
      3462c94b
    • Andrew Kryczka's avatar
      support multiple db_paths in SstFileManager · 6a3eebba
      Andrew Kryczka authored 
      Summary:
Now that files scheduled for deletion are kept in the same directory, we don't need to constrain deletion scheduler to `db_paths[0]`. Previously this was done because there was a separate trash directory, and this constraint prevented files from being accidentally copied to another filesystem when they're scheduled for deletion.
Closes https://github.com/facebook/rocksdb/pull/3544

Differential Revision: D7093786

Pulled By: ajkr

fbshipit-source-id: 202f5c92d925eafebec1281fb95bb5828d33414f
      6a3eebba
    • Fosco Marotto's avatar
      uint64_t and size_t changes to compile for iOS · d518fe1d
      Fosco Marotto authored 
      Summary:
In attempting to build a static lib for use in iOS, I ran in to lots of type errors between uint64_t and size_t.  This PR contains the changes I made to get `TARGET_OS=IOS make static_lib` to succeed while also getting Xcode to build successfully with the resulting `librocksdb.a` library imported.

This also compiles for me on macOS and tests fine, but I'm really not sure if I made the correct decisions about where to `static_cast` and where to change types.

Also up for discussion: is iOS worth supporting?  Getting the static lib is just part one, we aren't providing any bridging headers or wrappers like the ObjectiveRocks project, it won't be a great experience.
Closes https://github.com/facebook/rocksdb/pull/3503

Differential Revision: D7106457

Pulled By: gfosco

fbshipit-source-id: 82ac2073de7e1f09b91f6b4faea91d18bd311f8e
      d518fe1d
    • Siying Dong's avatar
      Update TARGETS · 8bc41f4f
      Siying Dong authored 
      Summary:
Watch the build
Closes https://github.com/facebook/rocksdb/pull/3533

Differential Revision: D7063777

Pulled By: siying

fbshipit-source-id: db9cdfc362a8d281dada6513ab034a6d6f0d552e
      8bc41f4f
    • Dmitri Smirnov's avatar
      Windows cumulative patch · c364eb42
      Dmitri Smirnov authored 
      Summary:
This patch addressed several issues.
  Portability including db_test std::thread -> port::Thread Cc: @
  and %z to ROCKSDB portable macro. Cc: maysamyabandeh

  Implement Env::AreFilesSame

  Make the implementation of file unique number more robust

  Get rid of C-runtime and go directly to Windows API when dealing
  with file primitives.

  Implement GetSectorSize() and aling unbuffered read on the value if
  available.

  Adjust Windows Logger for the new interface, implement CloseImpl() Cc: anand1976

  Fix test running script issue where $status var was of incorrect scope
  so the failures were swallowed and not reported.

  DestroyDB() creates a logger and opens a LOG file in the directory
  being cleaned up. This holds a lock on the folder and the cleanup is
  prevented. This fails one of the checkpoin tests. We observe the same in production.
  We close the log file in this change.

 Fix DBTest2.ReadAmpBitmapLiveInCacheAfterDBClose failure where the test
 attempts to open a directory with NewRandomAccessFile which does not
 work on Windows.
  Fix DBTest.SoftLimit as it is dependent on thread timing. CC: yiwu-arbug
Closes https://github.com/facebook/rocksdb/pull/3552

Differential Revision: D7156304

Pulled By: siying

fbshipit-source-id: 43db0a757f1dfceffeb2b7988043156639173f5b
      c364eb42