1. 10 7月, 2020 1 次提交
    • Yanqin Jin's avatar
      First step towards handling MANIFEST write error (#6949) · dd63f04c
      Yanqin Jin 创作于
      Summary:
      This PR provides preliminary support for handling IO error during MANIFEST write.
      File write/sync is not guaranteed to be atomic. If we encounter an IOError while writing/syncing to the MANIFEST file, we cannot be sure about the state of the MANIFEST file. The version edits may or may not have reached the file. During cleanup, if we delete the newly-generated SST files referenced by the pending version edit(s), but the version edit(s) actually are persistent in the MANIFEST, then next recovery attempt will process the version edits(s) and then fail since the SST files have already been deleted.
      One approach is to truncate the MANIFEST after write/sync error, so that it is safe to delete the SST files. However, file truncation may not be supported on certain file systems. Therefore, we take the following approach.
      If an IOError is detected during MANIFEST write/sync, we disable file deletions for the faulty database. Depending on whether the IOError is retryable (set by underlying file system), either RocksDB or application can call `DB::Resume()`, or simply shutdown and restart. During `Resume()`, RocksDB will try to switch to a new MANIFEST and write all existing in-memory version storage in the new file. If this succeeds, then RocksDB may proceed. If all recovery is completed, then file deletions will be re-enabled.
      Note that multiple threads can call `LogAndApply()` at the same time, though only one of them will be going through the process MANIFEST write, possibly batching the version edits of other threads. When the leading MANIFEST writer finishes, all of the MANIFEST writing threads in this batch will have the same IOError. They will all call `ErrorHandler::SetBGError()` in which file deletion will be disabled.
      
      Possible future directions:
      - Add an `ErrorContext` structure so that it is easier to pass more info to `ErrorHandler`. Currently, as in this example, a new `BackgroundErrorReason` has to be added.
      
      Test plan (dev server):
      make check
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6949
      
      Reviewed By: anand1976
      
      Differential Revision: D22026020
      
      Pulled By: riversand963
      
      fbshipit-source-id: f3c68a2ef45d9b505d0d625c7c5e0c88495b91c8
      dd63f04c
  2. 05 5月, 2020 2 次提交
    • Yanqin Jin's avatar
      Fix db_stress when GetLiveFiles() flushes dropped CF (#6805) · 5a61e786
      Yanqin Jin 创作于
      Summary:
      Current impl. of db_stress will abort verification and report failure if
      GetLiveFiles() causes a dropped column family to be flushed. This is not
      desired.
      To fix, this PR makes the following change:
      In GetLiveFiles, if flush is triggered and returns
      Status::IsColumnFamilyDropped(), then set status to Status::OK().
      This is OK because dropped column families will be skipped during the rest of
      this function, and valid column families will have their live files returned to
      caller.
      
      Test plan (dev server):
      make check
      ./db_stress -ops_per_thread=1000 -get_live_files_one_in=100 -clear_column_family_one_in=100
      ./db_stress -disable_wal=1 -reopen=0 -ops_per_thread=1000 -get_live_files_one_in=100 -clear_column_family_one_in=100
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6805
      
      Reviewed By: ltamasi
      
      Differential Revision: D21390044
      
      Pulled By: riversand963
      
      fbshipit-source-id: de67846b95a4f1b88aa0a30c3d70c43cc68625b9
      5a61e786
    • Levi Tamasi's avatar
      Expose the set of live blob files from Version/VersionSet (#6785) · a00ddf15
      Levi Tamasi 创作于
      Summary:
      The patch adds logic that returns the set of live blob files from
      `Version::AddLiveFiles` and `VersionSet::AddLiveFiles` (in addition to
      live table files), and also cleans up the code a bit, for example, by
      exposing only the numbers of table files as opposed to the earlier
      `FileDescriptor`s that no clients used. Moreover, the patch extends
      the `GetLiveFiles` API so that it also exposes blob files in the current version.
      Similarly to https://github.com/facebook/rocksdb/pull/6755,
      this is a building block for identifying and purging obsolete blob files.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6785
      
      Test Plan: `make check`
      
      Reviewed By: riversand963
      
      Differential Revision: D21336210
      
      Pulled By: ltamasi
      
      fbshipit-source-id: fc1aede8a49eacd03caafbc5f6f9ce43b6270821
      a00ddf15
  3. 21 2月, 2020 1 次提交
    • sdong's avatar
      Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) · fdf882de
      sdong 创作于
      Summary:
      When dynamically linking two binaries together, different builds of RocksDB from two sources might cause errors. To provide a tool for user to solve the problem, the RocksDB namespace is changed to a flag which can be overridden in build time.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/6433
      
      Test Plan: Build release, all and jtest. Try to build with ROCKSDB_NAMESPACE with another flag.
      
      Differential Revision: D19977691
      
      fbshipit-source-id: aa7f2d0972e1c31d75339ac48478f34f6cfcfb3e
      fdf882de
  4. 13 12月, 2019 1 次提交
  5. 21 9月, 2019 1 次提交
  6. 05 9月, 2019 1 次提交
    • Affan Dar's avatar
      Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765) · 229e6fbe
      Affan Dar 创作于
      Summary:
      Adding a light weight API to get last live WAL file name and size. Meant to be used as a helper for backup/restore tooling in a larger ecosystem such as MySQL with a MyRocks storage engine.
      
      Specifically within MySQL's backup/restore mechanism, this call can be made with a write lock on the mysql db to get a transactionally consistent snapshot of the current WAL file position along with other non-rocksdb log/data files.
      
      Without this, the alternative would be to take the aforementioned lock, scan the WAL dir for all files, find the last file and note its exact size as the rocksdb 'checkpoint'.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5765
      
      Differential Revision: D17172717
      
      Pulled By: affandar
      
      fbshipit-source-id: f2fabafd4c0e6fc45f126670c8c88a9f84cb8a37
      229e6fbe
  7. 14 6月, 2019 1 次提交
    • Levi Tamasi's avatar
      Add missing check before calling PurgeObsoleteFiles in EnableFileDeletions (#5448) · a3b8c76d
      Levi Tamasi 创作于
      Summary:
      Calling PurgeObsoleteFiles with a JobContext for which HaveSomethingToDelete
      is false is a precondition violation. This would trigger an assertion in debug builds;
      however, in release builds with assertions disabled, this can result in the
      pending_purge_obsolete_files_ counter in DBImpl underflowing, which in turn can lead
      to the process hanging during database close.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/5448
      
      Differential Revision: D15792569
      
      Pulled By: ltamasi
      
      fbshipit-source-id: 82d92c9b4f6a9efcdc69dbb3d5a52a1ae2dd2472
      a3b8c76d
  8. 07 6月, 2019 1 次提交
  9. 01 6月, 2019 1 次提交
  10. 31 5月, 2019 2 次提交
  11. 30 5月, 2019 1 次提交
  12. 13 11月, 2018 1 次提交
    • Yanqin Jin's avatar
      Remove redundant member var and set options (#4631) · 05dec0c7
      Yanqin Jin 创作于
      Summary:
      In the past, both `DBImpl::atomic_flush_` and
      `DBImpl::immutable_db_options_.atomic_flush` exist. However, we fail to set
      `immutable_db_options_.atomic_flush`, but use `DBImpl::atomic_flush_` which is
      set correctly. This does not lead to incorrect behavior, but is a duplicate of
      information.
      
      Since `immutable_db_options_` is always there and has `atomic_flush`, we should
      use it as source of truth and remove `DBImpl::atomic_flush_`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4631
      
      Differential Revision: D12928371
      
      Pulled By: riversand963
      
      fbshipit-source-id: f85a811959d3828aad4a3a1b05f71facf19c636d
      05dec0c7
  13. 31 10月, 2018 1 次提交
  14. 27 10月, 2018 1 次提交
  15. 10 10月, 2018 1 次提交
  16. 22 3月, 2018 1 次提交
  17. 10 2月, 2018 1 次提交
  18. 18 1月, 2018 1 次提交
    • Andrew Kryczka's avatar
      fix live WALs purged while file deletions disabled · 46e599fc
      Andrew Kryczka 创作于
      Summary:
      When calling `DisableFileDeletions` followed by `GetSortedWalFiles`, we guarantee the files returned by the latter call won't be deleted until after file deletions are re-enabled. However, `GetSortedWalFiles` didn't omit files already planned for deletion via `PurgeObsoleteFiles`, so the guarantee could be broken.
      
      We fix it by making `GetSortedWalFiles` wait for the number of pending purges to hit zero if file deletions are disabled. This condition is eventually met since `PurgeObsoleteFiles` is guaranteed to be called for the existing pending purges, and new purges cannot be scheduled while file deletions are disabled. Once the condition is met, `GetSortedWalFiles` simply returns the content of DB and archive directories, which nobody can delete (except for deletion scheduler, for which I plan to fix this bug later) until deletions are re-enabled.
      Closes https://github.com/facebook/rocksdb/pull/3341
      
      Differential Revision: D6681131
      
      Pulled By: ajkr
      
      fbshipit-source-id: 90b1e2f2362ea9ef715623841c0826611a817634
      46e599fc
  19. 19 7月, 2017 1 次提交
  20. 16 7月, 2017 1 次提交
  21. 28 4月, 2017 1 次提交
  22. 04 4月, 2017 1 次提交
  23. 16 3月, 2017 1 次提交
    • Islam AbdelRahman's avatar
      Add macros to include file name and line number during Logging · e1916368
      Islam AbdelRahman 创作于
      Summary:
      current logging
      ```
      2017/03/14-14:20:30.393432 7fedde9f5700 (Original Log Time 2017/03/14-14:20:30.393414) [default] Level summary: base level 1 max bytes base 268435456 files[1 0 0 0 0 0 0] max score 0.25
      2017/03/14-14:20:30.393438 7fedde9f5700 [JOB 2] Try to delete WAL files size 61417909, prev total WAL file size 73820858, number of live WAL files 2.
      2017/03/14-14:20:30.393464 7fedde9f5700 [DEBUG] [JOB 2] Delete /dev/shm/old_logging//MANIFEST-000001 type=3 #1 -- OK
      2017/03/14-14:20:30.393472 7fedde9f5700 [DEBUG] [JOB 2] Delete /dev/shm/old_logging//000003.log type=0 #3 -- OK
      2017/03/14-14:20:31.427103 7fedd49f1700 [default] New memtable created with log file: #9. Immutable memtables: 0.
      2017/03/14-14:20:31.427179 7fedde9f5700 [JOB 3] Syncing log #6
      2017/03/14-14:20:31.427190 7fedde9f5700 (Original Log Time 2017/03/14-14:20:31.427170) Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
      2017/03/14-14:20:31.
      Closes https://github.com/facebook/rocksdb/pull/1990
      
      Differential Revision: D4708695
      
      Pulled By: IslamAbdelRahman
      
      fbshipit-source-id: cb8968f
      e1916368
  24. 10 12月, 2016 1 次提交
  25. 24 9月, 2016 1 次提交
    • Yi Wu's avatar
      Split DBOptions into ImmutableDBOptions and MutableDBOptions · 9ed928e7
      Yi Wu 创作于
      Summary: Use ImmutableDBOptions/MutableDBOptions internally and DBOptions only for user-facing APIs. MutableDBOptions is barely a placeholder for now. I'll start to move options to MutableDBOptions in following diffs.
      
      Test Plan:
        make all check
      
      Reviewers: yhchiang, IslamAbdelRahman, sdong
      
      Reviewed By: sdong
      
      Subscribers: andrewkr, dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D64065
      9ed928e7
  26. 10 6月, 2016 1 次提交
    • Wanning Jiang's avatar
      Backup Options · 56887f6c
      Wanning Jiang 创作于
      Summary: Backup options file to private directory
      
      Test Plan:
      backupable_db_test.cc, BackupOptions
      	   Modify DB options by calling OpenDB for 3 times. Check the latest options file is in the right place. Also check no redundent files are backuped.
      
      Reviewers: andrewkr
      
      Reviewed By: andrewkr
      
      Subscribers: leveldb, dhruba, andrewkr
      
      Differential Revision: https://reviews.facebook.net/D59373
      56887f6c
  27. 10 2月, 2016 1 次提交
  28. 20 6月, 2015 1 次提交
    • Venkatesh Radhakrishnan's avatar
      Add wal files to Checkpoint for multiple column families. · 04251e1e
      Venkatesh Radhakrishnan 创作于
      Summary:
      When there are multiple column families, the flush in
      GetLiveFiles is not atomic, so that there are entries in the wal files
      which are needed to get a consisten RocksDB. We now add the log files to
      the checkpoint.
      
      Test Plan:
      CheckpointCF - This test forces more data to be written to
      the other column families after the flush of the first column family but
      before the second.
      
      Reviewers: igor, yhchiang, IslamAbdelRahman, anthony, kradhakrishnan, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D40323
      04251e1e
  29. 20 3月, 2015 1 次提交
    • Igor Canadi's avatar
      Don't delete files when column family is dropped · b088c83e
      Igor Canadi 创作于
      Summary:
      To understand the bug read t5943287 and check out the new test in column_family_test (ReadDroppedColumnFamily), iter 0.
      
      RocksDB contract allowes you to read a drop column family as long as there is a live reference. However, since our iteration ignores dropped column families, AddLiveFiles() didn't mark files of a dropped column families as live. So we deleted them.
      
      In this patch I no longer ignore dropped column families in the iteration. I think this behavior was confusing and it also led to this bug. Now if an iterator client wants to ignore dropped column families, he needs to do it explicitly.
      
      Test Plan: Added a new unit test that is failing on master. Unit test succeeds now.
      
      Reviewers: sdong, rven, yhchiang
      
      Reviewed By: yhchiang
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D32535
      b088c83e
  30. 13 2月, 2015 1 次提交
    • Igor Canadi's avatar
      Introduce job_id for flush and compaction · e7ea51a8
      Igor Canadi 创作于
      Summary:
      It would be good to assing background job their IDs. Two benefits:
      1) makes LOGs more readable
      2) I might use it in my EventLogger, which will try to make our LOG easier to read/query/visualize
      
      Test Plan: ran rocksdb, read the LOG
      
      Reviewers: sdong, rven, yhchiang
      
      Reviewed By: yhchiang
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D31617
      e7ea51a8
  31. 05 2月, 2015 1 次提交
    • Yueh-Hsuan Chiang's avatar
      Add a counter for collecting the wait time on db mutex. · 181191a1
      Yueh-Hsuan Chiang 创作于
      Summary:
      Add a counter for collecting the wait time on db mutex.
      Also add MutexWrapper and CondVarWrapper for measuring wait time.
      
      Test Plan:
      ./db_test
      export ROCKSDB_TESTS=MutexWaitStats
      ./db_test
      
      verify stats output using db_bench
      make clean
      make release
      ./db_bench --statistics=1 --benchmarks=fillseq,readwhilewriting --num=10000 --threads=10
      
      Sample output:
          rocksdb.db.mutex.wait.micros COUNT : 7546866
      
      Reviewers: MarkCallaghan, rven, sdong, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D32787
      181191a1
  32. 21 11月, 2014 1 次提交
  33. 15 11月, 2014 3 次提交
    • Igor Canadi's avatar
      Clean job_context · 23295b74
      Igor Canadi 创作于
      23295b74
    • Igor Canadi's avatar
      Explicitly clean JobContext · 5c04acda
      Igor Canadi 创作于
      Summary: This way we can gurantee that old MemTables get destructed before DBImpl gets destructed, which might be useful if we want to make them depend on state from DBImpl.
      
      Test Plan: make check with asserts in JobContext's destructor
      
      Reviewers: ljin, sdong, yhchiang, rven, jonahcohen
      
      Reviewed By: jonahcohen
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D28959
      5c04acda
    • Venkatesh Radhakrishnan's avatar
      Provide openable snapshots · 6c1b040c
      Venkatesh Radhakrishnan 创作于
      Summary: Store links to live files in directory on same disk
      
      Test Plan:
      Take snapshot and open it. Added a test GetSnapshotLink in
      db_test.
      
      Reviewers: sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D28713
      6c1b040c
  34. 05 11月, 2014 1 次提交
  35. 30 10月, 2014 2 次提交
    • Igor Canadi's avatar
      WalManager · 63590548
      Igor Canadi 创作于
      Summary: Decoupling code that deals with archived log files outside of DBImpl. That will make this code easier to reason about and test. It will also make the code easier to improve, because an improver doesn't have to understand DBImpl code in entirety.
      
      Test Plan: added test
      
      Reviewers: ljin, yhchiang, rven, sdong
      
      Reviewed By: sdong
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D27873
      63590548
    • Yueh-Hsuan Chiang's avatar
      Apply InfoLogLevel to the logs in db/db_filesnapshot.cc · 7b3a618f
      Yueh-Hsuan Chiang 创作于
      Summary: Apply InfoLogLevel to the logs in db/db_filesnapshot.cc
      
      Test Plan: make
      
      Reviewers: ljin, sdong, rven, igor
      
      Reviewed By: igor
      
      Subscribers: dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D27813
      7b3a618f