1. 10 Feb, 2018 1 commit
  2. 08 Feb, 2018 4 commits
  3. 07 Feb, 2018 4 commits
    • Maysam Yabandeh's avatar
      Add skip_cc option to TransactionDB::Write · 8feee280
      Maysam Yabandeh authored
      Summary:
      Compared to DB::Write, TransactionDB::Write has the additional overhead of creating and initializing an internal transaction object, as well as the overhead of locking/unlocking the keys. This patch extends the TransactionDB::Write with an skip_cc option to allow the users to indicate that the write batch do not conflict with others and the concurrency control and its overhead can be skipped. TransactionDB::Write by default calls DB::Write when skip_cc is set, which works for WriteCommitted WritePolicy. Any other flavor of TransactionDB that is not compatible with this default behavior (such as WritePreparedTxnDB) can extend ::Write and implement their own approach for taking into account the skip_cc optimization.
      Closes https://github.com/facebook/rocksdb/pull/3457
      
      Differential Revision: D6877318
      
      Pulled By: maysamyabandeh
      
      fbshipit-source-id: 56f4e21db87ff71492db4e376fb7c2b03dfeab6b
      8feee280
    • Maysam Yabandeh's avatar
      Fix leak report by asan on DuplicateKeys test · 8f8eb4f1
      Maysam Yabandeh authored
      Summary:
      Deletes the transaction object at the end of the test.
      Verified by:
      - COMPILE_WITH_ASAN=1 make -j32 transaction_test
      - ./transaction_test --gtest_filter="DBA**Duplicate*"
      Closes https://github.com/facebook/rocksdb/pull/3470
      
      Differential Revision: D6916473
      
      Pulled By: maysamyabandeh
      
      fbshipit-source-id: 8303df25408635d5d3ac2b25f309a3d15957c937
      8f8eb4f1
    • Yi Wu's avatar
      WritePrepared Txn: update compaction_iterator_test and db_iterator_test · 81736d8a
      Yi Wu authored
      Summary:
      Update compaction_iterator_test with write-prepared transaction DB related tests. Transaction related tests are group in CompactionIteratorWithSnapshotCheckerTest. The existing test are duplicated to make them also test with dummy SnapshotChecker that will say every key is visible to every snapshot (this is okay, we still compare sequence number to verify visibility). Merge related tests are disabled and will be revisit in another PR.
      
      Existing db_iterator_tests are also duplicated to test with dummy read_callback that will say every key is committed.
      Closes https://github.com/facebook/rocksdb/pull/3466
      
      Differential Revision: D6909253
      
      Pulled By: yiwu-arbug
      
      fbshipit-source-id: 2ae4656b843a55e2e9ff8beecf21f2832f96cd25
      81736d8a
    • Zhongyi Xie's avatar
      split RandomizedHarnessTest more ways · 2f299917
      Zhongyi Xie authored
      Summary:
      RandomizedHarnessTest enumerates different combinations of test type, compression type, restart interval, etc. For some combinations it takes very long to finish, causing the test to time out in test infrastructure.
      This PR split the test input into smaller trunks in the hope that they will fit in the timeout window. Another possibility is to reduce `num_entries` of course
      Closes https://github.com/facebook/rocksdb/pull/3467
      
      Differential Revision: D6910235
      
      Pulled By: miasantreble
      
      fbshipit-source-id: 717246ee5d21a8a48ad82d4d9c04f9051a66f07f
      2f299917
  4. 06 Feb, 2018 2 commits
    • Maysam Yabandeh's avatar
      WritePrepared Txn: Duplicate Keys, Txn Part · 88d8b2a2
      Maysam Yabandeh authored
      Summary:
      This patch takes advantage of memtable being able to detect duplicate <key,seq> and returning TryAgain to handle duplicate keys in WritePrepared Txns. Through WriteBatchWithIndex's index it detects existence of at least a duplicate key in the write batch. If duplicate key was reported, it then pays the cost of counting the number of sub-patches by iterating over the write batch and pass it to DBImpl::Write. DB will make use of the provided batch_count to assign proper sequence numbers before sending them to the WAL. When later inserting the batch to the memtable, it increases the seq each time memtbale reports a duplicate (a sub-patch in our counting) and tries again.
      Closes https://github.com/facebook/rocksdb/pull/3455
      
      Differential Revision: D6873699
      
      Pulled By: maysamyabandeh
      
      fbshipit-source-id: db8487526c3a5dc1ddda0ea49f0f979b26ae648d
      88d8b2a2
    • Anand Ananthabhotla's avatar
      Handle error return from WriteBuffer() · 4b124fb9
      Anand Ananthabhotla authored
      Summary:
      There are a couple of places where we swallow any error from
      WriteBuffer() - in SwitchMemtable() and DBImpl::CloseImpl(). Propagate
      the error up in those cases rather than ignoring it.
      Closes https://github.com/facebook/rocksdb/pull/3404
      
      Differential Revision: D6879954
      
      Pulled By: anand1976
      
      fbshipit-source-id: 2ef88b554be5286b0a8bad7384ba17a105395bdb
      4b124fb9
  5. 04 Feb, 2018 1 commit
  6. 03 Feb, 2018 3 commits
  7. 02 Feb, 2018 4 commits
    • Prashant D's avatar
      options: Fix coverity issues · 6e5b341e
      Prashant D authored
      Summary:
      options/cf_options.cc:
       77      memtable_insert_with_hint_prefix_extractor(
      
      CID 1396208 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
      2. uninit_member: Non-static class member info_log_level is not initialized in this constructor nor in any functions that it calls.
      Closes https://github.com/facebook/rocksdb/pull/3106
      
      Differential Revision: D6874689
      
      Pulled By: sagar0
      
      fbshipit-source-id: b5cd2d13915fd86d87260050f9c5d117615bbe30
      6e5b341e
    • Jun Wu's avatar
      crc32: suppress -Wimplicit-fallthrough warnings · e502839e
      Jun Wu authored
      Summary:
      Workaround a bunch of "implicit-fallthrough" compiler errors, like:
      
      ```
      util/crc32c.cc:533:7: error: this statement may fall through [-Werror=implicit-fallthrough=]
         crc = _mm_crc32_u64(crc, *(uint64_t*)(buf + offset));
             ^
      util/crc32c.cc:1016:9: note: in expansion of macro ‘CRCsinglet’
               CRCsinglet(crc0, next, -2 * 8);
               ^~~~~~~~~~
      util/crc32c.cc:1017:7: note: here
             case 1:
      ```
      Closes https://github.com/facebook/rocksdb/pull/3339
      
      Reviewed By: sagar0
      
      Differential Revision: D6874736
      
      Pulled By: quark-zju
      
      fbshipit-source-id: eec9f3bc135e12fca336928d01711006d5c3cb16
      e502839e
    • Fosco Marotto's avatar
      Upgrade Appveyor to VS2017 · ba8aa8fd
      Fosco Marotto authored
      Summary:
      Per some discussions, this will switch our Appveyor testing to use Visual Studio 2017.
      Closes https://github.com/facebook/rocksdb/pull/3445
      
      Differential Revision: D6874918
      
      Pulled By: gfosco
      
      fbshipit-source-id: c5a0032ca9f37f0d3baeae35c59d850d528c3176
      ba8aa8fd
    • Andrew Kryczka's avatar
      fix ReadaheadRandomAccessFile/iterator prefetch bug · b78ed046
      Andrew Kryczka authored
      Summary:
      `ReadaheadRandomAccessFile` is used by iterators for file reads in several cases, like in compaction when `compaction_readahead_size > 0` or `use_direct_io_for_flush_and_compaction == true`, or in user iterator when `ReadOptions::readahead_size > 0`. `ReadaheadRandomAccessFile` maintains an internal buffer for readahead data. It assumes that, if the buffer's length is less than `ReadaheadRandomAccessFile::readahead_size_`, which is fixed in the constructor, then EOF has been reached so it doesn't try reading further.
      
      Recently, d938226a started calling `RandomAccessFile::Prefetch` with various lengths: 8KB, 16KB, etc. When the `RandomAccessFile` is a `ReadaheadRandomAccessFile`, it triggers the above condition and incorrectly determines EOF. If a block is partially in the readahead buffer and EOF is incorrectly decided, the result is a truncated data block.
      
      The problem is reproducible:
      
      ```
      TEST_TMPDIR=/data/compaction_bench ./db_bench -benchmarks=fillrandom -write_buffer_size=1048576 -target_file_size_base=1048576 -block_size=18384 -use_direct_io_for_flush_and_compaction=true
      ...
      put error: Corruption: truncated block read from /data/compaction_bench/dbbench/000014.sst offset 20245, expected 10143 bytes, got 8427
      ```
      Closes https://github.com/facebook/rocksdb/pull/3454
      
      Differential Revision: D6869405
      
      Pulled By: ajkr
      
      fbshipit-source-id: 87001c299e7600a37c0dcccbd0368e0954c929cf
      b78ed046
  8. 01 Feb, 2018 6 commits
  9. 31 Jan, 2018 4 commits
  10. 30 Jan, 2018 10 commits
    • Andrew Kryczka's avatar
      fix for checkpoint directory with trailing slash(es) · f3fe6f88
      Andrew Kryczka authored
      Summary:
      previously if `checkpoint_dir` contained a trailing slash, we'd attempt to create the `.tmp` directory under `checkpoint_dir` due to simply concatenating `checkpoint_dir + ".tmp"`. This failed because `checkpoint_dir` hadn't been created yet and our directory creation is non-recursive. This PR fixes the issue by always creating the `.tmp` directory in the same parent as `checkpoint_dir` by stripping trailing slashes before concatenating.
      Closes https://github.com/facebook/rocksdb/pull/3275
      
      Differential Revision: D6574952
      
      Pulled By: ajkr
      
      fbshipit-source-id: a6daa6777a901eac2460cd0140c9515f7241aefc
      f3fe6f88
    • Yi Wu's avatar
      Fix DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge dead lock · 4bdf06e7
      Yi Wu authored
      Summary:
      In the test, there can be a dead lock between background flush thread and foreground main thread as following:
      * background flush thread:
        - holding db mutex, while
        - waiting on "DBImpl::FlushMemTableToOutputFile:BeforeInstallSV" sync point.
      * foreground thread:
        - waiting for db mutex to write "key2"
      
      Fixing by let background flush thread wait without holding db mutex.
      Closes https://github.com/facebook/rocksdb/pull/3436
      
      Differential Revision: D6841334
      
      Pulled By: yiwu-arbug
      
      fbshipit-source-id: b020768ac94e166e40953c5d09e505515a5f244d
      4bdf06e7
    • Maysam Yabandeh's avatar
      Split SnapshotConcurrentAccessTest into 20 sub tests · 3073b1c5
      Maysam Yabandeh authored
      Summary:
      SnapshotConcurrentAccessTest sometimes times out when running on the test infra. This patch splits the test into smaller sub-tests to avoid the timeout. It also benefits from lower run-time of each sub-test and increases the coverage of the test. The overall run-time of each final sub-test is at most half of the original test so we should no longer see a timeout.
      Closes https://github.com/facebook/rocksdb/pull/3435
      
      Differential Revision: D6839427
      
      Pulled By: maysamyabandeh
      
      fbshipit-source-id: d53fdb157109e2438ca7fe447d0cf4b71f304bd8
      3073b1c5
    • Sagar Vemuri's avatar
      Tests for dynamic universal compaction options · e6605e53
      Sagar Vemuri authored
      Summary:
      Added a test for three dynamic universal compaction options, in the realm of read amplification:
      - size_ratio
      - min_merge_width
      - max_merge_width
      
      Also updated DynamicUniversalCompactionSizeAmplification by adding a check on compaction reason.
      Found a bug in compaction reason setting while working on this PR, and fixed in #3412 .
      
      TODO for later: Still to add tests for these options: compression_size_percent, stop_style and trivial_move.
      Closes https://github.com/facebook/rocksdb/pull/3419
      
      Differential Revision: D6822217
      
      Pulled By: sagar0
      
      fbshipit-source-id: 074573fca6389053cbac229891a0163f38bb56c4
      e6605e53
    • Zhongyi Xie's avatar
      Use block cache to track memory usage when ReadOptions.fill_cache=false · 3fe09371
      Zhongyi Xie authored
      Summary:
      ReadOptions.fill_cache is set in compaction inputs and can be set by users in their queries too. It tells RocksDB not to put a data block used to block cache.
      
      The memory used by the data block is, however, not trackable by users.
      
      To make the system more manageable, we can cost the block to block cache while using it, and then release it after using.
      Closes https://github.com/facebook/rocksdb/pull/3333
      
      Differential Revision: D6670230
      
      Pulled By: miasantreble
      
      fbshipit-source-id: ab848d3ed286bd081a13ee1903de357b56cbc308
      3fe09371
    • Siying Dong's avatar
      db_bench: sanity check CuckooTable with mmap_read option · e2d4b0ef
      Siying Dong authored
      Summary:
      This is to avoid run time error. Fail the db_bench immediately if cuckoo table is used but mmap_read is not specified.
      Closes https://github.com/facebook/rocksdb/pull/3420
      
      Differential Revision: D6838284
      
      Pulled By: siying
      
      fbshipit-source-id: 20893fa28d40fadc31e4ff154bed02f5a1bad341
      e2d4b0ef
    • Mark Isaacson's avatar
      Suppress lint in old files · b8eb32f8
      Mark Isaacson authored
      Summary: Grandfather in super old lint issues to make a clean slate for moving forward that allows us to have stronger enforcement on new issues.
      
      Reviewed By: yiwu-arbug
      
      Differential Revision: D6821806
      
      fbshipit-source-id: 22797d31ec58e9eb0255d3b66fedfcfcb0dc127c
      b8eb32f8
    • Andrew Kryczka's avatar
      fix db_bench filluniquerandom key count assertion · 9f7ccc84
      Andrew Kryczka authored
      Summary:
      It failed every time. I guess people usually ran with assertions disabled.
      Closes https://github.com/facebook/rocksdb/pull/3422
      
      Differential Revision: D6822984
      
      Pulled By: ajkr
      
      fbshipit-source-id: 2e90db75618b26ac1c46ddfa9e03c095c7bf16e3
      9f7ccc84
    • Mamy Ratsimbazafy's avatar
      Add Nim to the list of language bindings · 3f666f79
      Mamy Ratsimbazafy authored
      Summary: Closes https://github.com/facebook/rocksdb/pull/3428
      
      Differential Revision: D6834061
      
      Pulled By: maysamyabandeh
      
      fbshipit-source-id: edca5b5b8330e0fee646c7434b9631da76670240
      3f666f79
    • Ben Darnell's avatar
      Rewrite comments on use_fsync option · 65cd6cd4
      Ben Darnell authored
      Summary:
      This replaces a vague warning about the mostly-obsolete ext3 filesystem with
      a more detailed note about a historical bug in the still-relevant ext4.
      
      Fixes #3410
      Closes https://github.com/facebook/rocksdb/pull/3421
      
      Differential Revision: D6834881
      
      Pulled By: siying
      
      fbshipit-source-id: 7771ef5c89a54c0ac17821680779c48178d0b400
      65cd6cd4
  11. 29 Jan, 2018 1 commit