Skip to content
GitLab
  1. 10 Feb, 2018 1 commit
  3. 08 Feb, 2018 4 commits
  5. 07 Feb, 2018 4 commits
    • Maysam Yabandeh's avatar
      Add skip_cc option to TransactionDB::Write · 8feee280
      Maysam Yabandeh authored 
      Summary:
Compared to DB::Write, TransactionDB::Write has the additional overhead of creating and initializing an internal transaction object, as well as the overhead of locking/unlocking the keys. This patch extends the TransactionDB::Write with an skip_cc option to allow the users to indicate that the write batch do not conflict with others and the concurrency control and its overhead can be skipped. TransactionDB::Write by default calls DB::Write when skip_cc is set, which works for WriteCommitted WritePolicy. Any other flavor of TransactionDB that is not compatible with this default behavior (such as WritePreparedTxnDB) can extend ::Write and implement their own approach for taking into account the skip_cc optimization.
Closes https://github.com/facebook/rocksdb/pull/3457

Differential Revision: D6877318

Pulled By: maysamyabandeh

fbshipit-source-id: 56f4e21db87ff71492db4e376fb7c2b03dfeab6b
      8feee280
    • Maysam Yabandeh's avatar
      Fix leak report by asan on DuplicateKeys test · 8f8eb4f1
      Maysam Yabandeh authored 
      Summary:
Deletes the transaction object at the end of the test.
Verified by:
- COMPILE_WITH_ASAN=1 make -j32 transaction_test
- ./transaction_test --gtest_filter="DBA**Duplicate*"
Closes https://github.com/facebook/rocksdb/pull/3470

Differential Revision: D6916473

Pulled By: maysamyabandeh

fbshipit-source-id: 8303df25408635d5d3ac2b25f309a3d15957c937
      8f8eb4f1
    • Yi Wu's avatar
      WritePrepared Txn: update compaction_iterator_test and db_iterator_test · 81736d8a
      Yi Wu authored 
      Summary:
Update compaction_iterator_test with write-prepared transaction DB related tests. Transaction related tests are group in CompactionIteratorWithSnapshotCheckerTest. The existing test are duplicated to make them also test with dummy SnapshotChecker that will say every key is visible to every snapshot (this is okay, we still compare sequence number to verify visibility). Merge related tests are disabled and will be revisit in another PR.

Existing db_iterator_tests are also duplicated to test with dummy read_callback that will say every key is committed.
Closes https://github.com/facebook/rocksdb/pull/3466

Differential Revision: D6909253

Pulled By: yiwu-arbug

fbshipit-source-id: 2ae4656b843a55e2e9ff8beecf21f2832f96cd25
      81736d8a
    • Zhongyi Xie's avatar
      split RandomizedHarnessTest more ways · 2f299917
      Zhongyi Xie authored 
      Summary:
RandomizedHarnessTest enumerates different combinations of test type, compression type, restart interval, etc. For some combinations it takes very long to finish, causing the test to time out in test infrastructure.
This PR split the test input into smaller trunks in the hope that they will fit in the timeout window. Another possibility is to reduce `num_entries` of course
Closes https://github.com/facebook/rocksdb/pull/3467

Differential Revision: D6910235

Pulled By: miasantreble

fbshipit-source-id: 717246ee5d21a8a48ad82d4d9c04f9051a66f07f
      2f299917
  7. 06 Feb, 2018 2 commits
    • Maysam Yabandeh's avatar
      WritePrepared Txn: Duplicate Keys, Txn Part · 88d8b2a2
      Maysam Yabandeh authored 
      Summary:
This patch takes advantage of memtable being able to detect duplicate <key,seq> and returning TryAgain to handle duplicate keys in WritePrepared Txns. Through WriteBatchWithIndex's index it detects existence of at least a duplicate key in the write batch. If duplicate key was reported, it then pays the cost of counting the number of sub-patches by iterating over the write batch and pass it to DBImpl::Write. DB will make use of the provided batch_count to assign proper sequence numbers before sending them to the WAL. When later inserting the batch to the memtable, it increases the seq each time memtbale reports a duplicate (a sub-patch in our counting) and tries again.
Closes https://github.com/facebook/rocksdb/pull/3455

Differential Revision: D6873699

Pulled By: maysamyabandeh

fbshipit-source-id: db8487526c3a5dc1ddda0ea49f0f979b26ae648d
      88d8b2a2
    • Anand Ananthabhotla's avatar
      Handle error return from WriteBuffer() · 4b124fb9
      Anand Ananthabhotla authored 
      Summary:
There are a couple of places where we swallow any error from
WriteBuffer() - in SwitchMemtable() and DBImpl::CloseImpl(). Propagate
the error up in those cases rather than ignoring it.
Closes https://github.com/facebook/rocksdb/pull/3404

Differential Revision: D6879954

Pulled By: anand1976

fbshipit-source-id: 2ef88b554be5286b0a8bad7384ba17a105395bdb
      4b124fb9
  9. 04 Feb, 2018 1 commit
  11. 03 Feb, 2018 3 commits
  13. 02 Feb, 2018 4 commits
    • Prashant D's avatar
      options: Fix coverity issues · 6e5b341e
      Prashant D authored 
      Summary:
options/cf_options.cc:
 77      memtable_insert_with_hint_prefix_extractor(

CID 1396208 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member info_log_level is not initialized in this constructor nor in any functions that it calls.
Closes https://github.com/facebook/rocksdb/pull/3106

Differential Revision: D6874689

Pulled By: sagar0

fbshipit-source-id: b5cd2d13915fd86d87260050f9c5d117615bbe30
      6e5b341e
    • Jun Wu's avatar
      crc32: suppress -Wimplicit-fallthrough warnings · e502839e
      Jun Wu authored 
      Summary:
Workaround a bunch of "implicit-fallthrough" compiler errors, like:

```
util/crc32c.cc:533:7: error: this statement may fall through [-Werror=implicit-fallthrough=]
   crc = _mm_crc32_u64(crc, *(uint64_t*)(buf + offset));
       ^
util/crc32c.cc:1016:9: note: in expansion of macro ‘CRCsinglet’
         CRCsinglet(crc0, next, -2 * 8);
         ^~~~~~~~~~
util/crc32c.cc:1017:7: note: here
       case 1:
```
Closes https://github.com/facebook/rocksdb/pull/3339

Reviewed By: sagar0

Differential Revision: D6874736

Pulled By: quark-zju

fbshipit-source-id: eec9f3bc135e12fca336928d01711006d5c3cb16
      e502839e
    • Fosco Marotto's avatar
      Upgrade Appveyor to VS2017 · ba8aa8fd
      Fosco Marotto authored 
      Summary:
Per some discussions, this will switch our Appveyor testing to use Visual Studio 2017.
Closes https://github.com/facebook/rocksdb/pull/3445

Differential Revision: D6874918

Pulled By: gfosco

fbshipit-source-id: c5a0032ca9f37f0d3baeae35c59d850d528c3176
      ba8aa8fd
    • Andrew Kryczka's avatar
      fix ReadaheadRandomAccessFile/iterator prefetch bug · b78ed046
      Andrew Kryczka authored 
      Summary:
`ReadaheadRandomAccessFile` is used by iterators for file reads in several cases, like in compaction when `compaction_readahead_size > 0` or `use_direct_io_for_flush_and_compaction == true`, or in user iterator when `ReadOptions::readahead_size > 0`. `ReadaheadRandomAccessFile` maintains an internal buffer for readahead data. It assumes that, if the buffer's length is less than `ReadaheadRandomAccessFile::readahead_size_`, which is fixed in the constructor, then EOF has been reached so it doesn't try reading further.

Recently, d938226a started calling `RandomAccessFile::Prefetch` with various lengths: 8KB, 16KB, etc. When the `RandomAccessFile` is a `ReadaheadRandomAccessFile`, it triggers the above condition and incorrectly determines EOF. If a block is partially in the readahead buffer and EOF is incorrectly decided, the result is a truncated data block.

The problem is reproducible:

```
TEST_TMPDIR=/data/compaction_bench ./db_bench -benchmarks=fillrandom -write_buffer_size=1048576 -target_file_size_base=1048576 -block_size=18384 -use_direct_io_for_flush_and_compaction=true
...
put error: Corruption: truncated block read from /data/compaction_bench/dbbench/000014.sst offset 20245, expected 10143 bytes, got 8427
```
Closes https://github.com/facebook/rocksdb/pull/3454

Differential Revision: D6869405

Pulled By: ajkr

fbshipit-source-id: 87001c299e7600a37c0dcccbd0368e0954c929cf
      b78ed046
  15. 01 Feb, 2018 6 commits
  17. 31 Jan, 2018 4 commits
  19. 30 Jan, 2018 10 commits
    • Andrew Kryczka's avatar
      fix for checkpoint directory with trailing slash(es) · f3fe6f88
      Andrew Kryczka authored 
      Summary:
previously if `checkpoint_dir` contained a trailing slash, we'd attempt to create the `.tmp` directory under `checkpoint_dir` due to simply concatenating `checkpoint_dir + ".tmp"`. This failed because `checkpoint_dir` hadn't been created yet and our directory creation is non-recursive. This PR fixes the issue by always creating the `.tmp` directory in the same parent as `checkpoint_dir` by stripping trailing slashes before concatenating.
Closes https://github.com/facebook/rocksdb/pull/3275

Differential Revision: D6574952

Pulled By: ajkr

fbshipit-source-id: a6daa6777a901eac2460cd0140c9515f7241aefc
      f3fe6f88
    • Yi Wu's avatar
      Fix DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge dead lock · 4bdf06e7
      Yi Wu authored 
      Summary:
In the test, there can be a dead lock between background flush thread and foreground main thread as following:
* background flush thread:
  - holding db mutex, while
  - waiting on "DBImpl::FlushMemTableToOutputFile:BeforeInstallSV" sync point.
* foreground thread:
  - waiting for db mutex to write "key2"

Fixing by let background flush thread wait without holding db mutex.
Closes https://github.com/facebook/rocksdb/pull/3436

Differential Revision: D6841334

Pulled By: yiwu-arbug

fbshipit-source-id: b020768ac94e166e40953c5d09e505515a5f244d
      4bdf06e7
    • Maysam Yabandeh's avatar
      Split SnapshotConcurrentAccessTest into 20 sub tests · 3073b1c5
      Maysam Yabandeh authored 
      Summary:
SnapshotConcurrentAccessTest sometimes times out when running on the test infra. This patch splits the test into smaller sub-tests to avoid the timeout. It also benefits from lower run-time of each sub-test and increases the coverage of the test. The overall run-time of each final sub-test is at most half of the original test so we should no longer see a timeout.
Closes https://github.com/facebook/rocksdb/pull/3435

Differential Revision: D6839427

Pulled By: maysamyabandeh

fbshipit-source-id: d53fdb157109e2438ca7fe447d0cf4b71f304bd8
      3073b1c5
    • Sagar Vemuri's avatar
      Tests for dynamic universal compaction options · e6605e53
      Sagar Vemuri authored 
      Summary:
Added a test for three dynamic universal compaction options, in the realm of read amplification:
- size_ratio
- min_merge_width
- max_merge_width

Also updated DynamicUniversalCompactionSizeAmplification by adding a check on compaction reason.
Found a bug in compaction reason setting while working on this PR, and fixed in #3412 .

TODO for later: Still to add tests for these options: compression_size_percent, stop_style and trivial_move.
Closes https://github.com/facebook/rocksdb/pull/3419

Differential Revision: D6822217

Pulled By: sagar0

fbshipit-source-id: 074573fca6389053cbac229891a0163f38bb56c4
      e6605e53
    • Zhongyi Xie's avatar
      Use block cache to track memory usage when ReadOptions.fill_cache=false · 3fe09371
      Zhongyi Xie authored 
      Summary:
ReadOptions.fill_cache is set in compaction inputs and can be set by users in their queries too. It tells RocksDB not to put a data block used to block cache.

The memory used by the data block is, however, not trackable by users.

To make the system more manageable, we can cost the block to block cache while using it, and then release it after using.
Closes https://github.com/facebook/rocksdb/pull/3333

Differential Revision: D6670230

Pulled By: miasantreble

fbshipit-source-id: ab848d3ed286bd081a13ee1903de357b56cbc308
      3fe09371
    • Siying Dong's avatar
      db_bench: sanity check CuckooTable with mmap_read option · e2d4b0ef
      Siying Dong authored 
      Summary:
This is to avoid run time error. Fail the db_bench immediately if cuckoo table is used but mmap_read is not specified.
Closes https://github.com/facebook/rocksdb/pull/3420

Differential Revision: D6838284

Pulled By: siying

fbshipit-source-id: 20893fa28d40fadc31e4ff154bed02f5a1bad341
      e2d4b0ef
    • Mark Isaacson's avatar
      Suppress lint in old files · b8eb32f8
      Mark Isaacson authored 
      Summary: Grandfather in super old lint issues to make a clean slate for moving forward that allows us to have stronger enforcement on new issues.

Reviewed By: yiwu-arbug

Differential Revision: D6821806

fbshipit-source-id: 22797d31ec58e9eb0255d3b66fedfcfcb0dc127c
      b8eb32f8
    • Andrew Kryczka's avatar
      fix db_bench filluniquerandom key count assertion · 9f7ccc84
      Andrew Kryczka authored 
      Summary:
It failed every time. I guess people usually ran with assertions disabled.
Closes https://github.com/facebook/rocksdb/pull/3422

Differential Revision: D6822984

Pulled By: ajkr

fbshipit-source-id: 2e90db75618b26ac1c46ddfa9e03c095c7bf16e3
      9f7ccc84
    • Mamy Ratsimbazafy's avatar
      Add Nim to the list of language bindings · 3f666f79
      Mamy Ratsimbazafy authored 
      Summary: Closes https://github.com/facebook/rocksdb/pull/3428

Differential Revision: D6834061

Pulled By: maysamyabandeh

fbshipit-source-id: edca5b5b8330e0fee646c7434b9631da76670240
      3f666f79
    • Ben Darnell's avatar
      Rewrite comments on use_fsync option · 65cd6cd4
      Ben Darnell authored 
      Summary:
This replaces a vague warning about the mostly-obsolete ext3 filesystem with
a more detailed note about a historical bug in the still-relevant ext4.

Fixes #3410
Closes https://github.com/facebook/rocksdb/pull/3421

Differential Revision: D6834881

Pulled By: siying

fbshipit-source-id: 7771ef5c89a54c0ac17821680779c48178d0b400
      65cd6cd4
  21. 29 Jan, 2018 1 commit