1. 06 Jun, 2020 1 commit
  2. 04 Jun, 2020 1 commit
    • Levi Tamasi's avatar
      Improve consistency checks in VersionBuilder (#6901) · 78e291b1
      Levi Tamasi authored
      Summary:
      The patch cleans up the code and improves the consistency checks around
      adding/deleting table files in `VersionBuilder`. Namely, it makes the checks
      stricter and improves them in the following ways:
      1) A table file can now only be deleted from the LSM tree using the level it
      resides on. Earlier, there was some unnecessary wiggle room for
      trivially moved files (they could be deleted using a lower level number than
      the actual one).
      2) A table file cannot be added to the tree if it is already present in the tree
      on any level (not just the target level). The earlier code only had an assertion
      (which is a no-op in release builds) that the newly added file is not already
      present on the target level.
      3) The above consistency checks around state transitions are now mandatory,
      as opposed to the earlier `CheckConsistencyForDeletes`, which was a no-op
      in release mode unless `force_consistency_checks` was set to `true`. The rationale
      here is that assuming that the initial state is consistent, a valid transition leads to a
      next state that is also consistent; however, an *invalid* transition offers no such
      guarantee. Hence it makes sense to validate the transitions unconditionally,
      and save `force_consistency_checks` for the paranoid checks that re-validate
      the entire state.
      4) The new checks build on the mechanism introduced in https://github.com/facebook/rocksdb/pull/6862,
      which enables us to efficiently look up the location (level and position within level)
      of files in a `Version` by file number. This makes the consistency checks much more
      efficient than the earlier `CheckConsistencyForDeletes`, which essentially
      performed a linear search.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6901
      
      Test Plan:
      Extended the unit tests and ran:
      
      `make check`
      `make whitebox_crash_test`
      
      Reviewed By: ajkr
      
      Differential Revision: D21822714
      
      Pulled By: ltamasi
      
      fbshipit-source-id: e2b29c8b6da1bf0f59004acc889e4870b2d18215
      78e291b1
  3. 20 May, 2020 1 commit
    • Levi Tamasi's avatar
      Refactor the blob file related logic in VersionBuilder (#6835) · 1551f101
      Levi Tamasi authored
      Summary:
      This patch is groundwork for an upcoming change to store the set of
      linked SSTs in `BlobFileMetaData`. With the current code, a new
      `BlobFileMetaData` object is created each time a `VersionEdit` touches
      a certain blob file. This is fine as long as these objects are lightweight
      and cheap to create; however, with the addition of the linked SST set, it would
      be very inefficient since the set would have to be copied over and over again.
      Note that this is the same kind of problem that `VersionBuilder` is solving
      w/r/t `Version`s and files, and we can apply the same solution; that is, we can
      accumulate the changes in a different mutable object, and apply the delta in
      one shot when the changes are committed. The patch does exactly that by
      adding a new `BlobFileMetaDataDelta` class to `VersionBuilder`. In addition,
      it turns the existing `GetBlobFileMetaData` helper into `IsBlobFileInVersion`
      (which is fine since that's the only thing the method's clients care about now),
      and adds a couple of helper methods that can create a `BlobFileMetaData`
      object from the `BlobFileMetaData` in the base (if applicable) and the delta
      when the `Version` is saved.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6835
      
      Test Plan: `make check`
      
      Reviewed By: riversand963
      
      Differential Revision: D21505187
      
      Pulled By: ltamasi
      
      fbshipit-source-id: d81a48c5f2ca7b79d7124c935332a6bcf3d5d988
      1551f101
  4. 06 May, 2020 2 commits
  5. 05 May, 2020 1 commit
    • sdong's avatar
      Avoid Swallowing Some File Consistency Checking Bugs (#6793) · 680c4163
      sdong authored
      Summary:
      We are swallowing some file consistency checking failures. This is not expected. We are fixing two cases: DB reopen and manifest dump.
      More places are not fixed and need follow-up.
      
      Error from CheckConsistencyForDeletes() is also swallowed, which is not fixed in this PR.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6793
      
      Test Plan: Add a unit test to cover the reopen case.
      
      Reviewed By: riversand963
      
      Differential Revision: D21366525
      
      fbshipit-source-id: eb438a322237814e8d5125f916a3c6de97f39ded
      680c4163
  6. 01 May, 2020 1 commit
    • Levi Tamasi's avatar
      Keep track of obsolete blob files in VersionSet (#6755) · fe238e54
      Levi Tamasi authored
      Summary:
      The patch adds logic to keep track of obsolete blob files. A blob file becomes
      obsolete when the last `shared_ptr` that points to the corresponding
      `SharedBlobFileMetaData` object goes away, which, in turn, happens when the
      last `Version` that contains the blob file is destroyed. No longer needed blob
      files are added to the obsolete list in `VersionSet` using a custom deleter to
      avoid unnecessary coupling between `SharedBlobFileMetaData` and `VersionSet`.
      Obsolete blob files are returned by `VersionSet::GetObsoleteFiles` and stored
      in `JobContext`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6755
      
      Test Plan: `make check`
      
      Reviewed By: riversand963
      
      Differential Revision: D21233155
      
      Pulled By: ltamasi
      
      fbshipit-source-id: 47757e06fdc0127f27ed57f51abd27893d9a7b7a
      fe238e54
  7. 24 Apr, 2020 1 commit
    • Levi Tamasi's avatar
      Make sure (Shared)BlobFileMetaData are owned by shared_ptrs (#6749) · bc51e33d
      Levi Tamasi authored
      Summary:
      The patch makes a couple of small cleanups to `SharedBlobFileMetaData` and `BlobFileMetaData`:
      * It makes the constructors private and introduces factory methods to ensure these objects are always owned by `shared_ptr`s. Note that `SharedBlobFileMetaData` has an additional factory that takes a deleter object; we can utilize this to e.g. notify `VersionSet` when a blob file becomes obsolete (which is exactly when `SharedBlobFileMetaData` is destroyed).
      * It disables move operations explicitly instead of relying on them being suppressed because of a user-declared destructor.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6749
      
      Test Plan: `make check`
      
      Reviewed By: riversand963
      
      Differential Revision: D21206947
      
      Pulled By: ltamasi
      
      fbshipit-source-id: 9094c14cc335b3e226f883e5a0df4f87a5cdeb95
      bc51e33d
  8. 27 Mar, 2020 1 commit
    • Levi Tamasi's avatar
      Add blob files to VersionStorageInfo/VersionBuilder (#6597) · 6f62322f
      Levi Tamasi authored
      Summary:
      The patch adds a couple of classes to represent metadata about
      blob files: `SharedBlobFileMetaData` contains the information elements
      that are immutable (once the blob file is closed), e.g. blob file number,
      total number and size of blob files, checksum method/value, while
      `BlobFileMetaData` contains attributes that can vary across versions like
      the amount of garbage in the file. There is a single `SharedBlobFileMetaData`
      for each blob file, which is jointly owned by the `BlobFileMetaData` objects
      that point to it; `BlobFileMetaData` objects, in turn, are owned by `Version`s
      and can also be shared if the (immutable _and_ mutable) state of the blob file
      is the same in two versions.
      
      In addition, the patch adds the blob file metadata to `VersionStorageInfo`, and extends
      `VersionBuilder` so that it can apply blob file related `VersionEdit`s (i.e. those
      containing `BlobFileAddition`s and/or `BlobFileGarbage`), and save blob file metadata
      to a new `VersionStorageInfo`. Consistency checks are also extended to ensure
      that table files point to blob files that are part of the `Version`, and that all blob files
      that are part of any given `Version` have at least some _non_-garbage data in them.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6597
      
      Test Plan: `make check`
      
      Reviewed By: riversand963
      
      Differential Revision: D20656803
      
      Pulled By: ltamasi
      
      fbshipit-source-id: f1f74d135045b3b42d0146f03ee576ef0a4bfd80
      6f62322f
  9. 21 Mar, 2020 1 commit
    • Yanqin Jin's avatar
      Attempt to recover from db with missing table files (#6334) · fb09ef05
      Yanqin Jin authored
      Summary:
      There are situations when RocksDB tries to recover, but the db is in an inconsistent state due to SST files referenced in the MANIFEST being missing. In this case, previous RocksDB will just fail the recovery and return a non-ok status.
      This PR enables another possibility. During recovery, RocksDB checks possible MANIFEST files, and try to recover to the most recent state without missing table file. `VersionSet::Recover()` applies version edits incrementally and "materializes" a version only when this version does not reference any missing table file. After processing the entire MANIFEST, the version created last will be the latest version.
      `DBImpl::Recover()` calls `VersionSet::Recover()`. Afterwards, WAL replay will *not* be performed.
      To use this capability, set `options.best_efforts_recovery = true` when opening the db. Best-efforts recovery is currently incompatible with atomic flush.
      
      Test plan (on devserver):
      ```
      $make check
      $COMPILE_WITH_ASAN=1 make all && make check
      ```
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6334
      
      Reviewed By: anand1976
      
      Differential Revision: D19778960
      
      Pulled By: riversand963
      
      fbshipit-source-id: c27ea80f29bc952e7d3311ecf5ee9c54393b40a8
      fb09ef05
  10. 20 Mar, 2020 1 commit
    • Levi Tamasi's avatar
      Clean up VersionBuilder a bit (#6556) · 44240455
      Levi Tamasi authored
      Summary:
      The whole point of the pimpl idiom is to hide implementation details.
      Internal helper methods like `CheckConsistency`, `CheckConsistencyForDeletes`,
      and `MaybeAddFile` do not belong in the public interface of the class.
      In addition, the patch switches to `unique_ptr` for the implementation
      object instead of using a raw `delete`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6556
      
      Test Plan: `make check`
      
      Reviewed By: riversand963
      
      Differential Revision: D20523568
      
      Pulled By: ltamasi
      
      fbshipit-source-id: 5bbb0ccebd0c47a33b815398c7f9cfe13bd775ac
      44240455
  11. 12 Mar, 2020 1 commit
    • Cheng Chang's avatar
      Cache result of GetLogicalBufferSize in Linux (#6457) · 2d9efc9a
      Cheng Chang authored
      Summary:
      In Linux, when reopening DB with many SST files, profiling shows that 100% system cpu time spent for a couple of seconds for `GetLogicalBufferSize`. This slows down MyRocks' recovery time when site is down.
      
      This PR introduces two new APIs:
      1. `Env::RegisterDbPaths` and `Env::UnregisterDbPaths` lets `DB` tell the env when it starts or stops using its database directories . The `PosixFileSystem` takes this opportunity to set up a cache from database directories to the corresponding logical block sizes.
      2. `LogicalBlockSizeCache` is defined only for OS_LINUX to cache the logical block sizes.
      
      Other modifications:
      1. rename `logical buffer size` to `logical block size` to be consistent with Linux terms.
      2. declare `GetLogicalBlockSize` in `PosixHelper` to expose it to `PosixFileSystem`.
      3. change the functions `IOError` and `IOStatus` in `env/io_posix.h` to have external linkage since they are used in other translation units too.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6457
      
      Test Plan:
      1. A new unit test is added for `LogicalBlockSizeCache` in `env/io_posix_test.cc`.
      2. A new integration test is added for `DB` operations related to the cache in `db/db_logical_block_size_cache_test.cc`.
      
      `make check`
      
      Differential Revision: D20131243
      
      Pulled By: cheng-chang
      
      fbshipit-source-id: 3077c50f8065c0bffb544d8f49fb10bba9408d04
      2d9efc9a
  12. 21 Feb, 2020 1 commit
    • sdong's avatar
      Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) · fdf882de
      sdong authored
      Summary:
      When dynamically linking two binaries together, different builds of RocksDB from two sources might cause errors. To provide a tool for user to solve the problem, the RocksDB namespace is changed to a flag which can be overridden in build time.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6433
      
      Test Plan: Build release, all and jtest. Try to build with ROCKSDB_NAMESPACE with another flag.
      
      Differential Revision: D19977691
      
      fbshipit-source-id: aa7f2d0972e1c31d75339ac48478f34f6cfcfb3e
      fdf882de
  13. 08 Feb, 2020 1 commit
    • Levi Tamasi's avatar
      Clean up VersionEdit a bit (#6383) · 752c87af
      Levi Tamasi authored
      Summary:
      This is a bunch of small improvements to `VersionEdit`. Namely, the patch
      
      * Makes the names and order of variables, methods, and code chunks related
        to the various information elements more consistent, and adds missing
        getters for the sake of completeness.
      * Initializes previously uninitialized stack variables.
      * Marks all getters const to improve const correctness.
      * Adds in-class initializers and removes the default ctor that would
        create an object with uninitialized built-in fields and call `Clear`
        afterwards.
      * Adds a new type alias for new files and changes the existing `typedef`
        for deleted files into a type alias as well.
      * Makes the helper method `DecodeNewFile4From` private.
      * Switches from long-winded iterator syntax to range based loops in a
        couple of places.
      * Fixes a couple of assignments where an integer 0 was assigned to
        boolean members.
      * Fixes a getter which used to return a `const std::string` instead of
      the intended `const std::string&`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6383
      
      Test Plan: make check
      
      Differential Revision: D19780537
      
      Pulled By: ltamasi
      
      fbshipit-source-id: b0b4f09fee0ec0e7c7b7a6d76bfe5346e91824d0
      752c87af
  14. 14 Dec, 2019 1 commit
    • anand76's avatar
      Introduce a new storage specific Env API (#5761) · afa2420c
      anand76 authored
      Summary:
      The current Env API encompasses both storage/file operations, as well as OS related operations. Most of the APIs return a Status, which does not have enough metadata about an error, such as whether its retry-able or not, scope (i.e fault domain) of the error etc., that may be required in order to properly handle a storage error. The file APIs also do not provide enough control over the IO SLA, such as timeout, prioritization, hinting about placement and redundancy etc.
      
      This PR separates out the file/storage APIs from Env into a new FileSystem class. The APIs are updated to return an IOStatus with metadata about the error, as well as to take an IOOptions structure as input in order to allow more control over the IO.
      
      The user can set both ```options.env``` and ```options.file_system``` to specify that RocksDB should use the former for OS related operations and the latter for storage operations. Internally, a ```CompositeEnvWrapper``` has been introduced that inherits from ```Env``` and redirects individual methods to either an ```Env``` implementation or the ```FileSystem``` as appropriate. When options are sanitized during ```DB::Open```, ```options.env``` is replaced with a newly allocated ```CompositeEnvWrapper``` instance if both env and file_system have been specified. This way, the rest of the RocksDB code can continue to function as before.
      
      This PR also ports PosixEnv to the new API by splitting it into two - PosixEnv and PosixFileSystem. PosixEnv is defined as a sub-class of CompositeEnvWrapper, and threading/time functions are overridden with Posix specific implementations in order to avoid an extra level of indirection.
      
      The ```CompositeEnvWrapper``` translates ```IOStatus``` return code to ```Status```, and sets the severity to ```kSoftError``` if the io_status is retryable. The error handling code in RocksDB can then recover the DB automatically.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5761
      
      Differential Revision: D18868376
      
      Pulled By: anand1976
      
      fbshipit-source-id: 39efe18a162ea746fabac6360ff529baba48486f
      afa2420c
  15. 08 Oct, 2019 1 commit
  16. 21 Sep, 2019 1 commit
  17. 30 Aug, 2019 1 commit
    • Pratik Dhandharia's avatar
      Lower the risk for users to run options.force_consistency_checks = true (#5744) · a2818223
      Pratik Dhandharia authored
      Summary:
      Open-source users recently reported two occurrences of LSM-tree corruption (https://github.com/facebook/rocksdb/issues/5558 is one), which would be caught by options.force_consistency_checks = true. options.force_consistency_checks has a usability limitation because it crashes the service once inconsistency is detected. This makes the feature hard to use. Most users serve from multiple RocksDB shards per server and the impacts of crashing the service is higher than it should be.
      
      Instead, we just pass the error back to users without killing the service, and ask them to deal with the problem accordingly.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5744
      
      Differential Revision: D17096940
      
      Pulled By: pdhandharia
      
      fbshipit-source-id: b6780039044e265f26ed2ad03c51f4abbe8b603c
      a2818223
  18. 07 Jun, 2019 1 commit
  19. 27 Mar, 2019 1 commit
    • Yanqin Jin's avatar
      Support for single-primary, multi-secondary instances (#4899) · 9358178e
      Yanqin Jin authored
      Summary:
      This PR allows RocksDB to run in single-primary, multi-secondary process mode.
      The writer is a regular RocksDB (e.g. an `DBImpl`) instance playing the role of a primary.
      Multiple `DBImplSecondary` processes (secondaries) share the same set of SST files, MANIFEST, WAL files with the primary. Secondaries tail the MANIFEST of the primary and apply updates to their own in-memory state of the file system, e.g. `VersionStorageInfo`.
      
      This PR has several components:
      1. (Originally in #4745). Add a `PathNotFound` subcode to `IOError` to denote the failure when a secondary tries to open a file which has been deleted by the primary.
      
      2. (Similar to #4602). Add `FragmentBufferedReader` to handle partially-read, trailing record at the end of a log from where future read can continue.
      
      3. (Originally in #4710 and #4820). Add implementation of the secondary, i.e. `DBImplSecondary`.
      3.1 Tail the primary's MANIFEST during recovery.
      3.2 Tail the primary's MANIFEST during normal processing by calling `ReadAndApply`.
      3.3 Tailing WAL will be in a future PR.
      
      4. Add an example in 'examples/multi_processes_example.cc' to demonstrate the usage of secondary RocksDB instance in a multi-process setting. Instructions to run the example can be found at the beginning of the source code.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4899
      
      Differential Revision: D14510945
      
      Pulled By: riversand963
      
      fbshipit-source-id: 4ac1c5693e6012ad23f7b4b42d3c374fecbe8886
      9358178e
  20. 29 Dec, 2018 1 commit
    • Siying Dong's avatar
      Preload some files even if options.max_open_files (#3340) · f0dda35d
      Siying Dong authored
      Summary:
      Choose to preload some files if options.max_open_files != -1. This can slightly narrow the gap of performance between options.max_open_files is -1 and a large number. To avoid a significant regression to DB reopen speed if options.max_open_files != -1. Limit the files to preload in DB open time to 16.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/3340
      
      Differential Revision: D6686945
      
      Pulled By: siying
      
      fbshipit-source-id: 8ec11bbdb46e3d0cdee7b6ad5897a09c5a07869f
      f0dda35d
  21. 15 Sep, 2018 1 commit
  22. 21 Aug, 2018 1 commit
  23. 28 Jul, 2018 1 commit
    • Yanqin Jin's avatar
      Remove random writes from SST file ingestion (#4172) · 54de5684
      Yanqin Jin authored
      Summary:
      RocksDB used to store global_seqno in external SST files written by
      SstFileWriter. During file ingestion, RocksDB uses `pwrite` to update the
      `global_seqno`. Since random write is not supported in some non-POSIX compliant
      file systems, external SST file ingestion is not supported on these file
      systems. To address this limitation, we no longer update `global_seqno` during
      file ingestion. Later RocksDB uses the MANIFEST and other information in table
      properties to deduce global seqno for externally-ingested SST files.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4172
      
      Differential Revision: D8961465
      
      Pulled By: riversand963
      
      fbshipit-source-id: 4382ec85270a96be5bc0cf33758ca2b167b05071
      54de5684
  24. 16 Jun, 2018 1 commit
  25. 22 May, 2018 1 commit
    • Zhongyi Xie's avatar
      Move prefix_extractor to MutableCFOptions · c3ebc758
      Zhongyi Xie authored
      Summary:
      Currently it is not possible to change bloom filter config without restart the db, which is causing a lot of operational complexity for users.
      This PR aims to make it possible to dynamically change bloom filter config.
      Closes https://github.com/facebook/rocksdb/pull/3601
      
      Differential Revision: D7253114
      
      Pulled By: miasantreble
      
      fbshipit-source-id: f22595437d3e0b86c95918c484502de2ceca120c
      c3ebc758
  26. 09 Mar, 2018 1 commit
  27. 06 Mar, 2018 1 commit
  28. 23 Feb, 2018 2 commits
  29. 08 Dec, 2017 1 commit
    • Prashant D's avatar
      Fix coverity issues version, write_batch · 34aa245d
      Prashant D authored
      Summary:
      db/version_builder.cc:
      117        base_vstorage_->InternalComparator();
      
      CID 1351713 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
      2. uninit_member: Non-static class member field level_zero_cmp_.internal_comparator is not initialized in this constructor nor in any functions that it calls.
      
      db/version_edit.h:
      145  FdWithKeyRange()
      146      : fd(),
      147        smallest_key(),
      148        largest_key() {
      
      CID 1418254 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
      2. uninit_member: Non-static class member file_metadata is not initialized in this constructor nor in any functions that it calls.
      149  }
      
      db/version_set.cc:
      120    }
      
      CID 1322789 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
      4. uninit_member: Non-static class member curr_file_level_ is not initialized in this constructor nor in any functions that it calls.
      121  }
      
      db/write_batch.cc:
       939    assert(cf_mems_);
      
      CID 1419862 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
      3. uninit_member: Non-static class member rebuilding_trx_seq_ is not initialized in this constructor nor in any functions that it calls.
       940  }
      Closes https://github.com/facebook/rocksdb/pull/3092
      
      Differential Revision: D6505666
      
      Pulled By: yiwu-arbug
      
      fbshipit-source-id: fd2c68948a0280772691a419d72ac7e190951d86
      34aa245d
  30. 18 Oct, 2017 1 commit
  31. 25 Aug, 2017 1 commit
    • Yi Wu's avatar
      Allow DB reopen with reduced options.num_levels · 3c840d1a
      Yi Wu authored
      Summary:
      Allow user to reduce number of levels in LSM by issue a full CompactRange() and put the result in a lower level, and then reopen DB with reduced options.num_levels. Previous this will fail on reopen on when recovery replaying the previous MANIFEST and found a historical file was on a higher level than the new options.num_levels. The workaround was after CompactRange(), reopen the DB with old num_levels, which will create a new MANIFEST, and then reopen the DB again with new num_levels.
      
      This patch relax the check of levels during recovery. It allows DB to open if there was a historical file on level > options.num_levels, but was also deleted.
      Closes https://github.com/facebook/rocksdb/pull/2740
      
      Differential Revision: D5629354
      
      Pulled By: yiwu-arbug
      
      fbshipit-source-id: 545903f6b36b6083e8cbaf777176aef2f488021d
      3c840d1a
  32. 22 Jul, 2017 2 commits
  33. 16 Jul, 2017 1 commit
  34. 28 Apr, 2017 1 commit
  35. 07 Feb, 2017 1 commit
    • Dmitri Smirnov's avatar
      Windows thread · 0a4cdde5
      Dmitri Smirnov authored
      Summary:
      introduce new methods into a public threadpool interface,
      - allow submission of std::functions as they allow greater flexibility.
      - add Joining methods to the implementation to join scheduled and submitted jobs with
        an option to cancel jobs that did not start executing.
      - Remove ugly `#ifdefs` between pthread and std implementation, make it uniform.
      - introduce pimpl for a drop in replacement of the implementation
      - Introduce rocksdb::port::Thread typedef which is a replacement for std::thread.  On Posix Thread defaults as before std::thread.
      - Implement WindowsThread that allocates memory in a more controllable manner than windows std::thread with a replaceable implementation.
      - should be no functionality changes.
      Closes https://github.com/facebook/rocksdb/pull/1823
      
      Differential Revision: D4492902
      
      Pulled By: siying
      
      fbshipit-source-id: c74cb11
      0a4cdde5
  36. 17 Dec, 2016 1 commit
    • Daniel Black's avatar
      gcc-7 requires include <functional> for std::function · 816c1e30
      Daniel Black authored
      Summary:
      Fixes compile error:
      
      In file included from ./util/statistics.h:17:0,
                       from ./util/stop_watch.h:8,
                       from ./util/perf_step_timer.h:9,
                       from ./util/iostats_context_imp.h:8,
                       from ./util/posix_logger.h:27,
                       from ./port/util_logger.h:18,
                       from ./db/auto_roll_logger.h:15,
                       from db/auto_roll_logger.cc:6:
      ./util/thread_local.h:65:16: error: 'function' in namespace 'std' does not name a template type
         typedef std::function<void(void*, void*)> FoldFunc;
      Closes https://github.com/facebook/rocksdb/pull/1656
      
      Differential Revision: D4318702
      
      Pulled By: yiwu-arbug
      
      fbshipit-source-id: 8c5d17a
      816c1e30
  37. 21 Oct, 2016 1 commit
    • Islam AbdelRahman's avatar
      Support IngestExternalFile (remove AddFile restrictions) · 869ae5d7
      Islam AbdelRahman authored
      Summary:
      Changes in the diff
      
      API changes:
      - Introduce IngestExternalFile to replace AddFile (I think this make the API more clear)
      - Introduce IngestExternalFileOptions (This struct will encapsulate the options for ingesting the external file)
      - Deprecate AddFile() API
      
      Logic changes:
      - If our file overlap with the memtable we will flush the memtable
      - We will find the first level in the LSM tree that our file key range overlap with the keys in it
      - We will find the lowest level in the LSM tree above the the level we found in step 2 that our file can fit in and ingest our file in it
      - We will assign a global sequence number to our new file
      - Remove AddFile restrictions by using global sequence numbers
      
      Other changes:
      - Refactor all AddFile logic to be encapsulated in ExternalSstFileIngestionJob
      
      Test Plan:
      unit tests (still need to add more)
      addfile_stress (https://reviews.facebook.net/D65037)
      
      Reviewers: yiwu, andrewkr, lightmark, yhchiang, sdong
      
      Reviewed By: sdong
      
      Subscribers: jkedgar, hcz, andrewkr, dhruba
      
      Differential Revision: https://reviews.facebook.net/D65061
      869ae5d7