1. 01 May, 2020 1 commit
    • anand76's avatar
      Pass a timeout to FileSystem for random reads (#6751) · ab13d43e
      anand76 authored
      Summary:
      Calculate ```IOOptions::timeout``` using ```ReadOptions::deadline``` and pass it to ```FileSystem::Read/FileSystem::MultiRead```. This allows us to impose a tighter bound on the time taken by Get/MultiGet on FileSystem/Envs that support IO timeouts. Even on those that don't support, check in ```RandomAccessFileReader::Read``` and ```MultiRead``` and return ```Status::TimedOut()``` if the deadline is exceeded.
      
      For now, TableReader creation, which might do file opens and reads, are not covered. It will be implemented in another PR.
      
      Tests:
      Update existing unit tests to verify the correct timeout value is being passed
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6751
      
      Reviewed By: riversand963
      
      Differential Revision: D21285631
      
      Pulled By: anand1976
      
      fbshipit-source-id: d89af843e5a91ece866e87aa29438b52a65a8567
      ab13d43e
  2. 28 Mar, 2020 1 commit
    • Zhichao Cao's avatar
      Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) · 42468881
      Zhichao Cao authored
      Summary:
      In the current code base, we use Status to get and store the returned status from the call. Specifically, for IO related functions, the current Status cannot reflect the IO Error details such as error scope, error retryable attribute, and others. With the implementation of https://github.com/facebook/rocksdb/issues/5761, we have the new Wrapper for IO, which returns IOStatus instead of Status. However, the IOStatus is purged at the lower level of write path and transferred to Status.
      
      The first job of this PR is to pass the IOStatus to the write path (flush, WAL write, and Compaction). The second job is to identify the Retryable IO Error as HardError, and set the bg_error_ as HardError. In this case, the DB Instance becomes read only. User is informed of the Status and need to take actions to deal with it (e.g., call db->Resume()).
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6487
      
      Test Plan: Added the testing case to error_handler_fs_test. Pass make asan_check
      
      Reviewed By: anand1976
      
      Differential Revision: D20685017
      
      Pulled By: zhichao-cao
      
      fbshipit-source-id: ff85f042896243abcd6ef37877834e26f36b6eb0
      42468881
  3. 07 Mar, 2020 1 commit
    • Cheng Chang's avatar
      Remove memcpy from RandomAccessFileReader::Read in direct IO mode (#6455) · 0a0151fb
      Cheng Chang authored
      Summary:
      In direct IO mode, RandomAccessFileReader::Read allocates an internal aligned buffer, and then copies the result into the scratch buffer. If the result is only temporarily used inside a function, there is no need to do the memcpy and just let the result Slice refer to the internally allocated buffer.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6455
      
      Test Plan: make check
      
      Differential Revision: D20106753
      
      Pulled By: cheng-chang
      
      fbshipit-source-id: 44f505843837bba47a56e3fa2c4dd3bd76486b58
      0a0151fb
  4. 21 Feb, 2020 1 commit
    • sdong's avatar
      Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) · fdf882de
      sdong authored
      Summary:
      When dynamically linking two binaries together, different builds of RocksDB from two sources might cause errors. To provide a tool for user to solve the problem, the RocksDB namespace is changed to a flag which can be overridden in build time.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6433
      
      Test Plan: Build release, all and jtest. Try to build with ROCKSDB_NAMESPACE with another flag.
      
      Differential Revision: D19977691
      
      fbshipit-source-id: aa7f2d0972e1c31d75339ac48478f34f6cfcfb3e
      fdf882de
  5. 17 Sep, 2019 1 commit
    • sdong's avatar
      Divide file_reader_writer.h and .cc (#5803) · b931f84e
      sdong authored
      Summary:
      file_reader_writer.h and .cc contain several files and helper function, and it's hard to navigate. Separate it to multiple files and put them under file/
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5803
      
      Test Plan: Build whole project using make and cmake.
      
      Differential Revision: D17374550
      
      fbshipit-source-id: 10efca907721e7a78ed25bbf74dc5410dea05987
      b931f84e
  6. 31 May, 2019 2 commits
  7. 06 Mar, 2018 1 commit
  8. 23 Feb, 2018 2 commits
  9. 22 Jul, 2017 2 commits
  10. 16 Jul, 2017 1 commit
  11. 28 Apr, 2017 1 commit
  12. 05 Apr, 2017 1 commit
  13. 10 Feb, 2016 1 commit
  14. 09 Jan, 2016 1 commit
    • sdong's avatar
      plain table reader: non-mmap mode to keep two recent buffers · 9a8e3f73
      sdong authored
      Summary: In plain table reader's non-mmap mode, we only keep the most recent read buffer. However, for binary search, it is likely we come back to a location to read. To avoid one pread in such a case, we keep two read buffers. It should cover most of the cases.
      
      Test Plan:
      1. run tests
      2. check the optimization works through strace when running
      ./table_reader_bench -mmap_read=false --num_keys2=1 -num_keys1=5000 -table_factory=plain_table --iterator --through_db
      
      Reviewers: anthony, rven, kradhakrishnan, igor, yhchiang, IslamAbdelRahman
      
      Reviewed By: IslamAbdelRahman
      
      Subscribers: leveldb, dhruba
      
      Differential Revision: https://reviews.facebook.net/D51171
      9a8e3f73
  15. 24 Sep, 2015 2 commits
    • sdong's avatar
      Fix the bug of using freed memory introduced by recent plain table reader patch · a70d08ec
      sdong authored
      Summary: Recent patch introduced a bug that if non-mmap mode is used, in prefix encoding case, there is a resizing of cur_key_ within the same prefix, we still read prefix from the released buffer. It fails ASAN tests and this commit fixes it.
      
      Test Plan: Run the ASAN tests for the failing test case.
      
      Reviewers: IslamAbdelRahman, yhchiang, anthony, igor, kradhakrishnan, rven
      
      Subscribers: leveldb, dhruba
      
      Differential Revision: https://reviews.facebook.net/D47457
      a70d08ec
    • sdong's avatar
      PlainTableReader to support non-mmap mode · df34aea3
      sdong authored
      Summary:
      PlainTableReader now only allows mmap-mode. Add the support to non-mmap mode for more flexibility.
      Refactor the codes to move all logic of reading data to PlainTableKeyDecoder, and consolidate the calls to Read() call and ReadVarint32() call. Implement the calls for both of mmap and non-mmap case seperately. For non-mmap mode, make copy of keys in several places when we need to move the buffer after reading the keys.
      
      Test Plan: Add the mode of non-mmap case in plain_table_db_test. Run it in valgrind mode too.
      
      Subscribers: leveldb, dhruba
      
      Differential Revision: https://reviews.facebook.net/D47187
      df34aea3
  16. 18 Jul, 2015 1 commit
    • sdong's avatar
      Move rate_limiter, write buffering, most perf context instrumentation and most... · 6e9fbeb2
      sdong authored
      Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env
      
      Summary: We want to keep Env a think layer for better portability. Less platform dependent codes should be moved out of Env. In this patch, I create a wrapper of file readers and writers, and put rate limiting, write buffering, as well as most perf context instrumentation and random kill out of Env. It will make it easier to maintain multiple Env in the future.
      
      Test Plan: Run all existing unit tests.
      
      Reviewers: anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, igor
      
      Reviewed By: igor
      
      Subscribers: leveldb, dhruba
      
      Differential Revision: https://reviews.facebook.net/D42321
      6e9fbeb2
  17. 07 Apr, 2015 1 commit
    • sdong's avatar
      Avoid naming conflict of EntryType · a45e7581
      sdong authored
      Summary:
      Fix build break on travis build:
      
      $ OPT=-DTRAVIS V=1 make unity && make clean && OPT=-DTRAVIS V=1 make db_test && ./db_test
      
      ......
      
      In file included from unity.cc:65:0:
      ./table/plain_table_key_coding.cc: In member function ‘rocksdb::Status rocksdb::PlainTableKeyDecoder::NextPrefixEncodingKey(const char*, const char*, rocksdb::ParsedInternalKey*, rocksdb::Slice*, size_t*, bool*)’:
      ./table/plain_table_key_coding.cc:224:3: error: reference to ‘EntryType’ is ambiguous
         EntryType entry_type;
         ^
      In file included from ./db/table_properties_collector.h:9:0,
                       from ./db/builder.h:11,
                       from ./db/builder.cc:10,
                       from unity.cc:1:
      ./include/rocksdb/table_properties.h:81:6: note: candidates are: enum rocksdb::EntryType
       enum EntryType {
            ^
      In file included from unity.cc:65:0:
      ./table/plain_table_key_coding.cc:16:6: note:                 enum rocksdb::{anonymous}::EntryType
       enum EntryType : unsigned char {
            ^
      ./table/plain_table_key_coding.cc:231:51: error: ‘entry_type’ was not declared in this scope
           const char* pos = DecodeSize(key_ptr, limit, &entry_type, &size);
                                                         ^
      make: *** [unity.o] Error 1
      
      Test Plan:
      OPT=-DTRAVIS V=1 make unity
      
      And make sure it doesn't break anymore.
      
      Reviewers: yhchiang, kradhakrishnan, igor
      
      Reviewed By: igor
      
      Subscribers: leveldb, dhruba
      
      Differential Revision: https://reviews.facebook.net/D36549
      a45e7581
  18. 24 Feb, 2015 1 commit
  19. 21 Feb, 2015 1 commit
    • Jim Meyering's avatar
      build: remove always-true assertions · 9283c7af
      Jim Meyering authored
      Summary:
      Remove some always-true assertions.
      They provoke these compilation failures:
      
        table/plain_table_key_coding.cc:279:20: error: comparison of unsigned expression >= 0 is always true [-Werror=type-limits]
        db/version_set.cc:336:15: error: comparison of unsigned expression >= 0 is always true [-Werror=type-limits]
      
      * table/plain_table_key_coding.cc (rocksdb): Remove assertion that
      unsigned type variable is >= 0.
      * db/version_set.cc (DoGenerateLevelFilesBrief): Likewise.
      
      Test Plan:
        Run "make EXTRA_CXXFLAGS='-W -Wextra'" and see fewer errors.
      
      Reviewers: ljin, sdong, igor.sugak, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba
      
      Differential Revision: https://reviews.facebook.net/D33747
      9283c7af
  20. 12 Nov, 2014 1 commit
    • Igor Canadi's avatar
      Turn on -Wshorten-64-to-32 and fix all the errors · 767777c2
      Igor Canadi authored
      Summary:
      We need to turn on -Wshorten-64-to-32 for mobile. See D1671432 (internal phabricator) for details.
      
      This diff turns on the warning flag and fixes all the errors. There were also some interesting errors that I might call bugs, especially in plain table. Going forward, I think it makes sense to have this flag turned on and be very very careful when converting 64-bit to 32-bit variables.
      
      Test Plan: compiles
      
      Reviewers: ljin, rven, yhchiang, sdong
      
      Reviewed By: yhchiang
      
      Subscribers: bobbaldwin, dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D28689
      767777c2
  21. 08 Sep, 2014 1 commit
  22. 06 Sep, 2014 1 commit
  23. 04 Jul, 2014 1 commit
  24. 19 Jun, 2014 1 commit