1. Nov 15, 2014
    • Igor Canadi's avatar
      Explicitly clean JobContext · 5c04acda
      Igor Canadi authored
      Summary: This way we can gurantee that old MemTables get destructed before DBImpl gets destructed, which might be useful if we want to make them depend on state from DBImpl.
      Test Plan: make check with asserts in JobContext's destructor
      Reviewers: ljin, sdong, yhchiang, rven, jonahcohen
      Reviewed By: jonahcohen
      Subscribers: dhruba, leveldb
      Differential Revision: https://reviews.facebook.net/D28959
    • Venkatesh Radhakrishnan's avatar
      Provide openable snapshots · 6c1b040c
      Venkatesh Radhakrishnan authored
      Summary: Store links to live files in directory on same disk
      Test Plan:
      Take snapshot and open it. Added a test GetSnapshotLink in
      Reviewers: sdong
      Reviewed By: sdong
      Subscribers: dhruba, leveldb
      Differential Revision: https://reviews.facebook.net/D28713
  2. Nov 05, 2014
  3. Oct 30, 2014
    • Igor Canadi's avatar
      WalManager · 63590548
      Igor Canadi authored
      Summary: Decoupling code that deals with archived log files outside of DBImpl. That will make this code easier to reason about and test. It will also make the code easier to improve, because an improver doesn't have to understand DBImpl code in entirety.
      Test Plan: added test
      Reviewers: ljin, yhchiang, rven, sdong
      Reviewed By: sdong
      Subscribers: dhruba, leveldb
      Differential Revision: https://reviews.facebook.net/D27873
    • Yueh-Hsuan Chiang's avatar
      Apply InfoLogLevel to the logs in db/db_filesnapshot.cc · 7b3a618f
      Yueh-Hsuan Chiang authored
      Summary: Apply InfoLogLevel to the logs in db/db_filesnapshot.cc
      Test Plan: make
      Reviewers: ljin, sdong, rven, igor
      Reviewed By: igor
      Subscribers: dhruba, leveldb
      Differential Revision: https://reviews.facebook.net/D27813
  4. Oct 29, 2014
    • Igor Canadi's avatar
      FlushProcess · a39e931e
      Igor Canadi authored
      Abstract out FlushProcess and take it out of DBImpl.
      This also includes taking DeletionState outside of DBImpl.
      Currently this diff is only doing the refactoring. Future work includes:
      1. Decoupling flush_process.cc, make it depend on less state
      2. Write flush_process_test, which will mock out everything that FlushProcess depends on and test it in isolation
      Test Plan: make check
      Reviewers: rven, yhchiang, sdong, ljin
      Reviewed By: ljin
      Subscribers: dhruba, leveldb
      Differential Revision: https://reviews.facebook.net/D27561
  5. Sep 06, 2014
  6. Sep 05, 2014
  7. Aug 27, 2014
  8. Jul 03, 2014
  9. May 21, 2014
  10. Apr 16, 2014
    • Igor Canadi's avatar
      RocksDBLite · 588bca20
      Igor Canadi authored
      Introducing RocksDBLite! Removes all the non-essential features and reduces the binary size. This effort should help our adoption on mobile.
      Binary size when compiling for IOS (`TARGET_OS=IOS m static_lib`) is down to 9MB from 15MB (without stripping)
      Test Plan: compiles :)
      Reviewers: dhruba, haobo, ljin, sdong, yhchiang
      Reviewed By: yhchiang
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D17835
  11. Apr 08, 2014
  12. Apr 04, 2014
  13. Apr 03, 2014
    • Haobo Xu's avatar
      [RocksDB] Fix a race condition in GetSortedWalFiles · 48bc0c6a
      Haobo Xu authored
      Summary: This patch fixed a race condition where a log file is moved to archived dir in the middle of GetSortedWalFiles. Without the fix, the log file would be missed in the result, which leads to transaction log iterator gap. A test utility SyncPoint is added to help reproducing the race condition.
      Test Plan: TransactionLogIteratorRace; make check
      Reviewers: dhruba, ljin
      Reviewed By: dhruba
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D17121
  14. Feb 26, 2014
    • Igor Canadi's avatar
      [CF] Adaptation of GetLiveFiles for CF · dc277f0a
      Igor Canadi authored
      Summary: Even if user flushes the memtables before getting live files, we still can't guarantee that new data didn't come in (to already-flushed memtables). If we want backups to provide consistent view of the database, we still need to get WAL files.
      Test Plan: backupable_db_test
      Reviewers: dhruba
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D16299
  15. Feb 15, 2014
    • Igor Canadi's avatar
      [CF] DB test to run on non-default column family · c67d48c8
      Igor Canadi authored
      This is a huge diff and it was hectic, but the idea is actually quite simple. Every operation (Put, Get, etc.) done on default column family in DBTest is now forwarded to non-default ("pikachu"). The good news is that we had zero test failures! Column families look stable so far.
      One interesting test that I adapted for column families is MultiThreadedTest. I replaced every Put() with a WriteBatch writing to all column families concurrently. Every Put in the write batch contains unique_id. Instead of Get() I do a multiget across all column families with the same key. If atomicity holds, I expect to see the same unique_id in all column families.
      Test Plan: This is a test!
      Reviewers: dhruba, haobo, kailiu, sdong
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D16149
  16. Feb 13, 2014
    • Igor Canadi's avatar
      [CF] Rethinking ColumnFamilyHandle and fix to dropping column families · b06840aa
      Igor Canadi authored
      The change to the public behavior:
      * When opening a DB or creating new column family client gets a ColumnFamilyHandle.
      * As long as column family handle is alive, client can do whatever he wants with it, even drop it
      * Dropped column family can still be read from (using the column family handle)
      * Added a new call CloseColumnFamily(). Client has to close all column families that he has opened before deleting the DB
      * As soon as column family is closed, any calls to DB using that column family handle will fail (also any outstanding calls)
      * Ref-counting ColumnFamilyData
      * New thread-safety for ColumnFamilySet
      * Dropped column families are now completely dropped and their memory cleaned-up
      Test Plan: added some tests to column_family_test
      Reviewers: dhruba, haobo, kailiu, sdong
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D16101
  17. Jan 30, 2014
  18. Jan 28, 2014
  19. Jan 16, 2014
    • Igor Canadi's avatar
      Move functions from VersionSet to Version · 2f4eda78
      Igor Canadi authored
      There were some functions in VersionSet that had no reason to be there instead of Version. Moving them to Version will make column families implementation easier.
      The functions moved are:
      * NumLevelBytes
      * LevelSummary
      * LevelFileSummary
      * MaxNextLevelOverlappingBytes
      * AddLiveFiles (previously AddLiveFilesCurrentVersion())
      * NeedSlowdownForNumLevel0Files
      The diff continues on (and depends on) D15171
      Test Plan: make check
      Reviewers: dhruba, haobo, kailiu, sdong, emayanke
      Reviewed By: sdong
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D15183
  20. Jan 02, 2014
    • Igor Canadi's avatar
      Support multi-threaded DisableFileDeletions() and EnableFileDeletions() · b60c14f6
      Igor Canadi authored
      We don't want two threads to clash if they concurrently call DisableFileDeletions() and EnableFileDeletions(). I'm adding a counter that will enable file deletions only after all DisableFileDeletions() calls have been negated with EnableFileDeletions().
      However, we also don't want to break the old behavior, so I added a parameter force to EnableFileDeletions(). If force is true, we will still enable file deletions after every call to EnableFileDeletions(), which is what is happening now.
      Test Plan: make check
      Reviewers: dhruba, haobo, sanketh
      Reviewed By: dhruba
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D14781
  21. Nov 09, 2013
    • Igor Canadi's avatar
      Speed up FindObsoleteFiles · 1510339e
      Igor Canadi authored
      Here's one solution we discussed on speeding up FindObsoleteFiles. Keep a set of all files in DBImpl and update the set every time we create a file. I probably missed few other spots where we create a file.
      It might speed things up a bit, but makes code uglier. I don't really like it.
      Much better approach would be to abstract all file handling to a separate class. Think of it as layer between DBImpl and Env. Having a separate class deal with file namings and deletion would benefit both code cleanliness (especially with huge DBImpl) and speed things up. It will take a huge effort to do this, though.
      Let's discuss offline today.
      Test Plan: Ran ./db_stress, verified that files are getting deleted
      Reviewers: dhruba, haobo, kailiu, emayanke
      Reviewed By: dhruba
      Differential Revision: https://reviews.facebook.net/D13827
  22. Oct 25, 2013
    • Mayank Agarwal's avatar
      Unify DeleteFile and DeleteWalFiles · 56305221
      Mayank Agarwal authored
      This is to simplify rocksdb public APIs and improve the code quality.
      Created an additional parameter to ParseFileName for log sub type and improved the code for deleting a wal file.
      Wrote exhaustive unit-tests in delete_file_test
      Unification of other redundant APIs can be taken up in a separate diff
      Test Plan: Expanded delete_file test
      Reviewers: dhruba, haobo, kailiu, sdong
      Reviewed By: dhruba
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D13647
  23. Oct 17, 2013
  24. Oct 09, 2013
    • Naman Gupta's avatar
      Add option for storing transaction logs in a separate dir · cbf4a064
      Naman Gupta authored
      Summary: In some cases, you might not want to store the data log (write ahead log) files in the same dir as the sst files. An example use case is leaf, which stores sst files in tmpfs. And would like to save the log files in a separate dir (disk) to save memory.
      Test Plan: make all. Ran db_test test. A few test failing. P2785018. If you guys don't see an obvious problem with the code, maybe somebody from the rocksdb team could help me debug the issue here. Running this on leaf worked well. I could see logs stored on disk, and deleted appropriately after compactions. Obviously this is only one set of options. The unit tests cover different options. Seems like I'm missing some edge cases.
      Reviewers: dhruba, haobo, leveldb
      CC: xinyaohu, sumeet
      Differential Revision: https://reviews.facebook.net/D13239
  25. Oct 05, 2013
    • Dhruba Borthakur's avatar
      Change namespace from leveldb to rocksdb · a143ef9b
      Dhruba Borthakur authored
      Change namespace from leveldb to rocksdb. This allows a single
      application to link in open-source leveldb code as well as
      rocksdb code into the same process.
      Test Plan: compile rocksdb
      Reviewers: emayanke
      Reviewed By: emayanke
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D13287
    • Mayank Agarwal's avatar
      Add backward compatible option in GetLiveFiles to choose whether to not Flush first · 854d2363
      Mayank Agarwal authored
      As explained in comments in GetLiveFiles in db.h, this option will cause flush to be skipped in GetLiveFiles because some use-cases use GetSortedWalFiles after GetLiveFiles to generate more complete snapshots.
      Using GetSortedWalFiles after GetLiveFiles allows us to not Flush in GetLiveFiles first because wals have everything.
      Note: file deletions will be disabled before calling GLF or GSWF so live logs will not move to archive logs or get delted.
      Note: Manifest file is truncated to a proper value in GLF, so it will always reply from the proper wal files on a restart
      Test Plan: make
      Reviewers: dhruba, haobo
      Reviewed By: dhruba
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D13257
  26. Sep 05, 2013
  27. Aug 24, 2013
  28. Aug 20, 2013
  29. Jun 06, 2013
  30. Nov 29, 2012
    • Abhishek Kona's avatar
      Fix all the lint errors. · d29f1819
      Abhishek Kona authored
      Scripted and removed all trailing spaces and converted all tabs to
      Also fixed other lint errors.
      All lint errors from this point of time should be taken seriously.
      Test Plan: make all check
      Reviewers: dhruba
      Reviewed By: dhruba
      CC: leveldb
      Differential Revision: https://reviews.facebook.net/D7059
  31. Sep 25, 2012
    • Dhruba Borthakur's avatar
      The BackupAPI should also list the length of the manifest file. · ae36e509
      Dhruba Borthakur authored
      The GetLiveFiles() api lists the set of sst files and the current
      MANIFEST file. But the database continues to append new data to the
      MANIFEST file even when the application is backing it up to the
      backup location. This means that the database-version that is
      stored in the MANIFEST FILE in the backup location
      does not correspond to the sst files returned by GetLiveFiles.
      This API adds a new parameter to GetLiveFiles. This new parmeter
      returns the current size of the MANIFEST file.
      Test Plan: Unit test attached.
      Reviewers: heyongqiang
      Reviewed By: heyongqiang
      Differential Revision: https://reviews.facebook.net/D5631
  32. Sep 18, 2012