1. 10 Feb, 2018 1 commit
  2. 08 Feb, 2018 4 commits
  3. 07 Feb, 2018 4 commits
    • Maysam Yabandeh's avatar
      Add skip_cc option to TransactionDB::Write · 8feee280
      Maysam Yabandeh authored
      Compared to DB::Write, TransactionDB::Write has the additional overhead of creating and initializing an internal transaction object, as well as the overhead of locking/unlocking the keys. This patch extends the TransactionDB::Write with an skip_cc option to allow the users to indicate that the write batch do not conflict with others and the concurrency control and its overhead can be skipped. TransactionDB::Write by default calls DB::Write when skip_cc is set, which works for WriteCommitted WritePolicy. Any other flavor of TransactionDB that is not compatible with this default behavior (such as WritePreparedTxnDB) can extend ::Write and implement their own approach for taking into account the skip_cc optimization.
      Closes https://github.com/facebook/rocksdb/pull/3457
      Differential Revision: D6877318
      Pulled By: maysamyabandeh
      fbshipit-source-id: 56f4e21db87ff71492db4e376fb7c2b03dfeab6b
    • Maysam Yabandeh's avatar
      Fix leak report by asan on DuplicateKeys test · 8f8eb4f1
      Maysam Yabandeh authored
      Deletes the transaction object at the end of the test.
      Verified by:
      - COMPILE_WITH_ASAN=1 make -j32 transaction_test
      - ./transaction_test --gtest_filter="DBA**Duplicate*"
      Closes https://github.com/facebook/rocksdb/pull/3470
      Differential Revision: D6916473
      Pulled By: maysamyabandeh
      fbshipit-source-id: 8303df25408635d5d3ac2b25f309a3d15957c937
    • Yi Wu's avatar
      WritePrepared Txn: update compaction_iterator_test and db_iterator_test · 81736d8a
      Yi Wu authored
      Update compaction_iterator_test with write-prepared transaction DB related tests. Transaction related tests are group in CompactionIteratorWithSnapshotCheckerTest. The existing test are duplicated to make them also test with dummy SnapshotChecker that will say every key is visible to every snapshot (this is okay, we still compare sequence number to verify visibility). Merge related tests are disabled and will be revisit in another PR.
      Existing db_iterator_tests are also duplicated to test with dummy read_callback that will say every key is committed.
      Closes https://github.com/facebook/rocksdb/pull/3466
      Differential Revision: D6909253
      Pulled By: yiwu-arbug
      fbshipit-source-id: 2ae4656b843a55e2e9ff8beecf21f2832f96cd25
    • Zhongyi Xie's avatar
      split RandomizedHarnessTest more ways · 2f299917
      Zhongyi Xie authored
      RandomizedHarnessTest enumerates different combinations of test type, compression type, restart interval, etc. For some combinations it takes very long to finish, causing the test to time out in test infrastructure.
      This PR split the test input into smaller trunks in the hope that they will fit in the timeout window. Another possibility is to reduce `num_entries` of course
      Closes https://github.com/facebook/rocksdb/pull/3467
      Differential Revision: D6910235
      Pulled By: miasantreble
      fbshipit-source-id: 717246ee5d21a8a48ad82d4d9c04f9051a66f07f
  4. 06 Feb, 2018 2 commits
    • Maysam Yabandeh's avatar
      WritePrepared Txn: Duplicate Keys, Txn Part · 88d8b2a2
      Maysam Yabandeh authored
      This patch takes advantage of memtable being able to detect duplicate <key,seq> and returning TryAgain to handle duplicate keys in WritePrepared Txns. Through WriteBatchWithIndex's index it detects existence of at least a duplicate key in the write batch. If duplicate key was reported, it then pays the cost of counting the number of sub-patches by iterating over the write batch and pass it to DBImpl::Write. DB will make use of the provided batch_count to assign proper sequence numbers before sending them to the WAL. When later inserting the batch to the memtable, it increases the seq each time memtbale reports a duplicate (a sub-patch in our counting) and tries again.
      Closes https://github.com/facebook/rocksdb/pull/3455
      Differential Revision: D6873699
      Pulled By: maysamyabandeh
      fbshipit-source-id: db8487526c3a5dc1ddda0ea49f0f979b26ae648d
    • Anand Ananthabhotla's avatar
      Handle error return from WriteBuffer() · 4b124fb9
      Anand Ananthabhotla authored
      There are a couple of places where we swallow any error from
      WriteBuffer() - in SwitchMemtable() and DBImpl::CloseImpl(). Propagate
      the error up in those cases rather than ignoring it.
      Closes https://github.com/facebook/rocksdb/pull/3404
      Differential Revision: D6879954
      Pulled By: anand1976
      fbshipit-source-id: 2ef88b554be5286b0a8bad7384ba17a105395bdb
  5. 04 Feb, 2018 1 commit
  6. 03 Feb, 2018 3 commits
  7. 02 Feb, 2018 4 commits
    • Prashant D's avatar
      options: Fix coverity issues · 6e5b341e
      Prashant D authored
       77      memtable_insert_with_hint_prefix_extractor(
      CID 1396208 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
      2. uninit_member: Non-static class member info_log_level is not initialized in this constructor nor in any functions that it calls.
      Closes https://github.com/facebook/rocksdb/pull/3106
      Differential Revision: D6874689
      Pulled By: sagar0
      fbshipit-source-id: b5cd2d13915fd86d87260050f9c5d117615bbe30
    • Jun Wu's avatar
      crc32: suppress -Wimplicit-fallthrough warnings · e502839e
      Jun Wu authored
      Workaround a bunch of "implicit-fallthrough" compiler errors, like:
      util/crc32c.cc:533:7: error: this statement may fall through [-Werror=implicit-fallthrough=]
         crc = _mm_crc32_u64(crc, *(uint64_t*)(buf + offset));
      util/crc32c.cc:1016:9: note: in expansion of macro ‘CRCsinglet’
               CRCsinglet(crc0, next, -2 * 8);
      util/crc32c.cc:1017:7: note: here
             case 1:
      Closes https://github.com/facebook/rocksdb/pull/3339
      Reviewed By: sagar0
      Differential Revision: D6874736
      Pulled By: quark-zju
      fbshipit-source-id: eec9f3bc135e12fca336928d01711006d5c3cb16
    • Fosco Marotto's avatar
      Upgrade Appveyor to VS2017 · ba8aa8fd
      Fosco Marotto authored
      Per some discussions, this will switch our Appveyor testing to use Visual Studio 2017.
      Closes https://github.com/facebook/rocksdb/pull/3445
      Differential Revision: D6874918
      Pulled By: gfosco
      fbshipit-source-id: c5a0032ca9f37f0d3baeae35c59d850d528c3176
    • Andrew Kryczka's avatar
      fix ReadaheadRandomAccessFile/iterator prefetch bug · b78ed046
      Andrew Kryczka authored
      `ReadaheadRandomAccessFile` is used by iterators for file reads in several cases, like in compaction when `compaction_readahead_size > 0` or `use_direct_io_for_flush_and_compaction == true`, or in user iterator when `ReadOptions::readahead_size > 0`. `ReadaheadRandomAccessFile` maintains an internal buffer for readahead data. It assumes that, if the buffer's length is less than `ReadaheadRandomAccessFile::readahead_size_`, which is fixed in the constructor, then EOF has been reached so it doesn't try reading further.
      Recently, d938226a started calling `RandomAccessFile::Prefetch` with various lengths: 8KB, 16KB, etc. When the `RandomAccessFile` is a `ReadaheadRandomAccessFile`, it triggers the above condition and incorrectly determines EOF. If a block is partially in the readahead buffer and EOF is incorrectly decided, the result is a truncated data block.
      The problem is reproducible:
      TEST_TMPDIR=/data/compaction_bench ./db_bench -benchmarks=fillrandom -write_buffer_size=1048576 -target_file_size_base=1048576 -block_size=18384 -use_direct_io_for_flush_and_compaction=true
      put error: Corruption: truncated block read from /data/compaction_bench/dbbench/000014.sst offset 20245, expected 10143 bytes, got 8427
      Closes https://github.com/facebook/rocksdb/pull/3454
      Differential Revision: D6869405
      Pulled By: ajkr
      fbshipit-source-id: 87001c299e7600a37c0dcccbd0368e0954c929cf
  8. 01 Feb, 2018 6 commits
  9. 31 Jan, 2018 4 commits
  10. 30 Jan, 2018 10 commits
    • Andrew Kryczka's avatar
      fix for checkpoint directory with trailing slash(es) · f3fe6f88
      Andrew Kryczka authored
      previously if `checkpoint_dir` contained a trailing slash, we'd attempt to create the `.tmp` directory under `checkpoint_dir` due to simply concatenating `checkpoint_dir + ".tmp"`. This failed because `checkpoint_dir` hadn't been created yet and our directory creation is non-recursive. This PR fixes the issue by always creating the `.tmp` directory in the same parent as `checkpoint_dir` by stripping trailing slashes before concatenating.
      Closes https://github.com/facebook/rocksdb/pull/3275
      Differential Revision: D6574952
      Pulled By: ajkr
      fbshipit-source-id: a6daa6777a901eac2460cd0140c9515f7241aefc
    • Yi Wu's avatar
      Fix DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge dead lock · 4bdf06e7
      Yi Wu authored
      In the test, there can be a dead lock between background flush thread and foreground main thread as following:
      * background flush thread:
        - holding db mutex, while
        - waiting on "DBImpl::FlushMemTableToOutputFile:BeforeInstallSV" sync point.
      * foreground thread:
        - waiting for db mutex to write "key2"
      Fixing by let background flush thread wait without holding db mutex.
      Closes https://github.com/facebook/rocksdb/pull/3436
      Differential Revision: D6841334
      Pulled By: yiwu-arbug
      fbshipit-source-id: b020768ac94e166e40953c5d09e505515a5f244d
    • Maysam Yabandeh's avatar
      Split SnapshotConcurrentAccessTest into 20 sub tests · 3073b1c5
      Maysam Yabandeh authored
      SnapshotConcurrentAccessTest sometimes times out when running on the test infra. This patch splits the test into smaller sub-tests to avoid the timeout. It also benefits from lower run-time of each sub-test and increases the coverage of the test. The overall run-time of each final sub-test is at most half of the original test so we should no longer see a timeout.
      Closes https://github.com/facebook/rocksdb/pull/3435
      Differential Revision: D6839427
      Pulled By: maysamyabandeh
      fbshipit-source-id: d53fdb157109e2438ca7fe447d0cf4b71f304bd8
    • Sagar Vemuri's avatar
      Tests for dynamic universal compaction options · e6605e53
      Sagar Vemuri authored
      Added a test for three dynamic universal compaction options, in the realm of read amplification:
      - size_ratio
      - min_merge_width
      - max_merge_width
      Also updated DynamicUniversalCompactionSizeAmplification by adding a check on compaction reason.
      Found a bug in compaction reason setting while working on this PR, and fixed in #3412 .
      TODO for later: Still to add tests for these options: compression_size_percent, stop_style and trivial_move.
      Closes https://github.com/facebook/rocksdb/pull/3419
      Differential Revision: D6822217
      Pulled By: sagar0
      fbshipit-source-id: 074573fca6389053cbac229891a0163f38bb56c4
    • Zhongyi Xie's avatar
      Use block cache to track memory usage when ReadOptions.fill_cache=false · 3fe09371
      Zhongyi Xie authored
      ReadOptions.fill_cache is set in compaction inputs and can be set by users in their queries too. It tells RocksDB not to put a data block used to block cache.
      The memory used by the data block is, however, not trackable by users.
      To make the system more manageable, we can cost the block to block cache while using it, and then release it after using.
      Closes https://github.com/facebook/rocksdb/pull/3333
      Differential Revision: D6670230
      Pulled By: miasantreble
      fbshipit-source-id: ab848d3ed286bd081a13ee1903de357b56cbc308
    • Siying Dong's avatar
      db_bench: sanity check CuckooTable with mmap_read option · e2d4b0ef
      Siying Dong authored
      This is to avoid run time error. Fail the db_bench immediately if cuckoo table is used but mmap_read is not specified.
      Closes https://github.com/facebook/rocksdb/pull/3420
      Differential Revision: D6838284
      Pulled By: siying
      fbshipit-source-id: 20893fa28d40fadc31e4ff154bed02f5a1bad341
    • Mark Isaacson's avatar
      Suppress lint in old files · b8eb32f8
      Mark Isaacson authored
      Summary: Grandfather in super old lint issues to make a clean slate for moving forward that allows us to have stronger enforcement on new issues.
      Reviewed By: yiwu-arbug
      Differential Revision: D6821806
      fbshipit-source-id: 22797d31ec58e9eb0255d3b66fedfcfcb0dc127c
    • Andrew Kryczka's avatar
      fix db_bench filluniquerandom key count assertion · 9f7ccc84
      Andrew Kryczka authored
      It failed every time. I guess people usually ran with assertions disabled.
      Closes https://github.com/facebook/rocksdb/pull/3422
      Differential Revision: D6822984
      Pulled By: ajkr
      fbshipit-source-id: 2e90db75618b26ac1c46ddfa9e03c095c7bf16e3
    • Mamy Ratsimbazafy's avatar
      Add Nim to the list of language bindings · 3f666f79
      Mamy Ratsimbazafy authored
      Summary: Closes https://github.com/facebook/rocksdb/pull/3428
      Differential Revision: D6834061
      Pulled By: maysamyabandeh
      fbshipit-source-id: edca5b5b8330e0fee646c7434b9631da76670240
    • Ben Darnell's avatar
      Rewrite comments on use_fsync option · 65cd6cd4
      Ben Darnell authored
      This replaces a vague warning about the mostly-obsolete ext3 filesystem with
      a more detailed note about a historical bug in the still-relevant ext4.
      Fixes #3410
      Closes https://github.com/facebook/rocksdb/pull/3421
      Differential Revision: D6834881
      Pulled By: siying
      fbshipit-source-id: 7771ef5c89a54c0ac17821680779c48178d0b400
  11. 29 Jan, 2018 1 commit