1. 31 May, 2019 10 commits
    • Siying Dong's avatar
      Move some memory related files from util/ to memory/ (#5382) · 8843129e
      Siying Dong authored
      Summary:
      Move arena, allocator, and memory tools under util to a separate memory/ directory.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5382
      
      Differential Revision: D15564655
      
      Pulled By: siying
      
      fbshipit-source-id: 9cd6b5d0d3d52b39606e19221fa154596e5852a5
      8843129e
    • Yanqin Jin's avatar
      Add class-level comments to version-related classes (#5348) · f1302eba
      Yanqin Jin authored
      Summary:
      As title.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5348
      
      Differential Revision: D15564595
      
      Pulled By: riversand963
      
      fbshipit-source-id: dd45aa86a70e0343c2e9ef702fad165163f548e6
      f1302eba
    • Sagar Vemuri's avatar
      Fix flaky DBTest2.PresetCompressionDict test (#5378) · 1b59a490
      Sagar Vemuri authored
      Summary:
      Fix flaky DBTest2.PresetCompressionDict test.
      
      This PR fixes two issues with the test:
      1. Replaces `GetSstFiles` with `TotalSize`, which is based on `DB::GetColumnFamilyMetaData` so that only the size of the live SST files is taken into consideration when computing the total size of all sst files. Earlier, with `GetSstFiles`, even obsolete files were getting picked up.
      1. In ZSTD compression, it is sometimes possible that using a trained dictionary is not better than using an untrained one. Using a trained dictionary performs well in 99% of the cases, but still in the remaining ~1% of the cases (out of 10000 runs) using an untrained dictionary gets better compression results.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5378
      
      Differential Revision: D15559100
      
      Pulled By: sagar0
      
      fbshipit-source-id: c35adbf13871f520a2cec48f8bad9ff27ff7a0b4
      1b59a490
    • Vijay Nadimpalli's avatar
      Organizing rocksdb/table directory by format · 50e47079
      Vijay Nadimpalli authored
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5373
      
      Differential Revision: D15559425
      
      Pulled By: vjnadimpalli
      
      fbshipit-source-id: 5d6d6d615582bedd96a4b879bb25d429a6de8b55
      50e47079
    • Sagar Vemuri's avatar
      Fix env_options_for_read spelling in CompactionJob · e6298626
      Sagar Vemuri authored
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5380
      
      Differential Revision: D15563386
      
      Pulled By: sagar0
      
      fbshipit-source-id: 8b26aef47cfc40ff8016daf815582f21cdd40df2
      e6298626
    • Levi Tamasi's avatar
      Move the index readers out of the block cache (#5298) · 1e355842
      Levi Tamasi authored
      Summary:
      Currently, when the block cache is used for index blocks as well, it is
      not really the index block that is stored in the cache but an
      IndexReader object. Since this object is not pure data (it has, for
      instance, pointers that might dangle), it's not really sharable. To
      avoid the issues around this, the current code uses a dummy unique cache
      key for each TableReader to store the IndexReader, and erases the
      IndexReader entry when the TableReader is closed. Instead of doing this,
      the new code moves the IndexReader out of the cache altogether. In
      particular, instead of the TableReader owning, or caching/pinning the
      IndexReader based on the customer's settings, the TableReader
      unconditionally owns the IndexReader, which in turn owns/caches/pins
      the index block (which is itself sharable and thus can be safely put in
      the cache without any hacks).
      
      Note: the change has two side effects:
      1) Partitions of partitioned indexes no longer affect the read
      amplification statistics.
      2) Eviction statistics for index blocks are temporarily broken. We plan to fix
      this in a separate phase.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5298
      
      Differential Revision: D15303203
      
      Pulled By: ltamasi
      
      fbshipit-source-id: 935a69ba59d87d5e44f42e2310619b790c366e47
      1e355842
    • anand76's avatar
      Fix reopen voting logic in db_stress when using MultiGet (#5374) · bd44ec20
      anand76 authored
      Summary:
      When the --reopen option is non-zero, the DB is reopened after every ops_per_thread/(reopen+1) ops, with the check being done after every op. With MultiGet, we might do multiple ops in one iteration, which broke the logic that checked when to synchronize among the threads and reopen the DB. This PR fixes that logic.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5374
      
      Differential Revision: D15559780
      
      Pulled By: anand1976
      
      fbshipit-source-id: ee6563a68045df7f367eca3cbc2500d3e26359ef
      bd44ec20
    • Siying Dong's avatar
      Move test related files under util/ to test_util/ (#5377) · e9e0101c
      Siying Dong authored
      Summary:
      There are too many types of files under util/. Some test related files don't belong to there or just are just loosely related. Mo
      ve them to a new directory test_util/, so that util/ is cleaner.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5377
      
      Differential Revision: D15551366
      
      Pulled By: siying
      
      fbshipit-source-id: 0f5c8653832354ef8caa31749c0143815d719e2c
      e9e0101c
    • anand76's avatar
      Increase Trash/DB size ratio in DBSSTTest.RateLimitedWALDelete (#5366) · a984040f
      anand76 authored
      Summary:
      By increasing the ratio, we ensure that all files go through background deletion and eliminate flakiness due to timing of deletions.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5366
      
      Differential Revision: D15549992
      
      Pulled By: anand1976
      
      fbshipit-source-id: d137375cd791fc1a802841412755d6e2b8fd7688
      a984040f
    • Zhongyi Xie's avatar
      Fix FIFO dynamic options sanitization (#5367) · 87fe4bca
      Zhongyi Xie authored
      Summary:
      When dynamically setting options, we check the option type info and skip options that are marked deprecated. However this check is only done at top level, which results in bugs where SetOptions will corrupt option values and cause unexpected system behavior iff a deprecated second level option is set dynamically.
      For exmaple, the following call:
      ```
      dbfull()->SetOptions(
          {{"compaction_options_fifo",
              "{allow_compaction=true;max_table_files_size=1024;ttl=731;}"}});
      ```
      was from pre 6.0 release when `ttl` was part of `compaction_options_fifo`. Now that it got moved out of `compaction_options_fifo`, this call will incorrectly set `compaction_options_fifo.max_table_files_size` to 731 (as `max_table_files_size` is the first one in `OptionsHelper::fifo_compaction_options_type_info` struct) and cause files to gett evicted much faster than expected.
      
      This PR adds verification to second level options like `compaction_options_fifo.ttl` or `compaction_options_fifo.max_table_files_size` when set dynamically, and filter out those marked as deprecated.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5367
      
      Differential Revision: D15530998
      
      Pulled By: miasantreble
      
      fbshipit-source-id: 818258be5c3abe09cd82d62f3c083572d70fecdd
      87fe4bca
  2. 30 May, 2019 1 commit
  3. 29 May, 2019 4 commits
    • Maysam Yabandeh's avatar
      WritePrepared: skip_concurrency_control option (#5330) · eab4f49a
      Maysam Yabandeh authored
      Summary:
      This enables the user to set TransactionDBOptions::skip_concurrency_control so the standard `DB::Write(const WriteOptions& opts, WriteBatch* updates)` would skip the concurrency control. This would give higher throughput to the users who know their use case doesn't need concurrency control.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5330
      
      Differential Revision: D15525932
      
      Pulled By: maysamyabandeh
      
      fbshipit-source-id: 68421ac1ba34f549a4a8de9ce4c2dccf6fb4b06b
      eab4f49a
    • Maysam Yabandeh's avatar
      WritePrepared: disableWAL in commit without prepare (#5327) · f5576c33
      Maysam Yabandeh authored
      Summary:
      When committing a transaction without prepare, WritePrepared simply writes the batch to db and add the commit entry to CommitCache. When two_write_queues=true, following the rule of committing only from 2nd write queue, the first write, writes the batch and the only thing the 2nd write does is to write the commit entry to CommitCache. Currently the write batch in 2nd write is set to an empty LogData entry, while the write to the WAL could simply be entirely disabled.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5327
      
      Differential Revision: D15424546
      
      Pulled By: maysamyabandeh
      
      fbshipit-source-id: 3d9ea3922d5196984c584d62a3ed57e1f7ca7b9f
      f5576c33
    • Siying Dong's avatar
      Add comments in compaction_picker.h · 4d0c3b1f
      Siying Dong authored
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5357
      
      Differential Revision: D15522825
      
      Pulled By: siying
      
      fbshipit-source-id: d775386b9d10c7179f5d3af2c821ed213abfacdf
      4d0c3b1f
    • Yanqin Jin's avatar
      Fix a clang analyze error (#5365) · b5e4ee2e
      Yanqin Jin authored
      Summary:
      The analyzer thinks max_allowed_ space can be 0. In that case, free_space will
      be assigned as free_space. It fails to realize that the function call
      GetFreeSpace actually sets the free_space variable properly, which is possibly
      due to lack of inter-function call analysis.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5365
      
      Differential Revision: D15521790
      
      Pulled By: riversand963
      
      fbshipit-source-id: 839d0a285a1c8773a28a385f0c3be4bb7fbe32cb
      b5e4ee2e
  4. 25 May, 2019 15 commits
  5. 24 May, 2019 8 commits
  6. 23 May, 2019 2 commits