1. 01 May, 2018 1 commit
    • Andrew Kryczka's avatar
      Avoid directory renames in BackupEngine · 747c8532
      Andrew Kryczka authored
      Summary:
      We used to name private directories like "1.tmp" while BackupEngine populated them, and then rename without the ".tmp" suffix (i.e., rename "1.tmp" to "1") after all files were copied. On glusterfs, directory renames like this require operations across many hosts, and partial failures have caused operational problems.
      
      Fortunately we don't need to rename private directories. We already have a meta-file that uses the tempfile-rename pattern to commit a backup atomically after all its files have been successfully copied. So we can copy private files directly to their final location, so now there's no directory rename.
      Closes https://github.com/facebook/rocksdb/pull/3749
      
      Differential Revision: D7705610
      
      Pulled By: ajkr
      
      fbshipit-source-id: fd724a28dd2bf993ce323a5f2cb7e7d6980cc346
      747c8532
  2. 22 Feb, 2018 1 commit
    • Andrew Kryczka's avatar
      BackupEngine gluster-friendly file naming convention · b0929776
      Andrew Kryczka authored
      Summary:
      Use the rsync tempfile naming convention in our `BackupEngine`. The temp file follows the format, `.<filename>.<suffix>`, which is later renamed to `<filename>`. We fix `tmp` as the `<suffix>` as we don't need to use random bytes for now. The benefit is gluster treats this tempfile naming convention specially and applies hashing only to `<filename>`, so the file won't need to be linked or moved when it's renamed. Our gluster team suggested this will make things operationally easier.
      Closes https://github.com/facebook/rocksdb/pull/3463
      
      Differential Revision: D6893333
      
      Pulled By: ajkr
      
      fbshipit-source-id: fd7622978f4b2487fce33cde40dd3124f16bcaa8
      b0929776
  3. 16 Dec, 2017 1 commit
    • Andrew Kryczka's avatar
      fix backup meta-file buffer overrun · a79c7c05
      Andrew Kryczka authored
      Summary:
      - check most times after calling snprintf that the buffer didn't fill up. Previously we'd proceed and use `buf_size - len` as the length in subsequent calls, which underflowed as those are unsigned size_t.
      - replace some memcpys with snprintf for consistency
      Closes https://github.com/facebook/rocksdb/pull/3255
      
      Differential Revision: D6541464
      
      Pulled By: ajkr
      
      fbshipit-source-id: 8610ea6a24f38e0a37c6d17bc65b7c712da6d932
      a79c7c05
  4. 29 Nov, 2017 1 commit
    • Prashant D's avatar
      utilities/backupable : Fix coverity issues · 81cf262f
      Prashant D authored
      Summary:
      1. Class BackupMeta
      ```
      52      : timestamp_(0), size_(0), meta_filename_(meta_filename),
      
      CID 1168103 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
      2. uninit_member: Non-static class member sequence_number_ is not initialized in this constructor nor in any functions that it calls.
      153        file_infos_(file_infos), env_(env) {}
      ```
      2. class BackupEngineImpl
      ```
      513  }
              7. uninit_member: Non-static class member latest_backup_id_ is not initialized in this constructor nor in any functions that it calls.
      
      CID 1322803 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
      9. uninit_member: Non-static class member latest_valid_backup_id_ is not initialized in this constructor nor in any functions that it calls.
      514}
      ```
      3. struct BackupAfterCopyOrCreateWorkItem
      ```
      368  struct BackupAfterCopyOrCreateWorkItem {
      369    std::future<CopyOrCreateResult> result;
              1. member_decl: Class member declaration for shared.
      370    bool shared;
              3. member_decl: Class member declaration for needed_to_copy.
      371    bool needed_to_copy;
              5. member_decl: Class member declaration for backup_env.
      372    Env* backup_env;
      373    std::string dst_path_tmp;
      374    std::string dst_path;
      375    std::string dst_relative;
              2. uninit_member: Non-static class member shared is not initialized in this constructor nor in any functions that it calls.
              4. uninit_member: Non-static class member needed_to_copy is not initialized in this constructor nor in any functions that it calls.
      
      CID 1396122 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
      6. uninit_member: Non-static class member backup_env is not initialized in this constructor nor in any functions that it calls.
      376    BackupAfterCopyOrCreateWorkItem() {}
      ```
      4. struct CopyOrCreateWorkItem
      ```
      318  struct CopyOrCreateWorkItem {
      319    std::string src_path;
      320    std::string dst_path;
      321    std::string contents;
              1. member_decl: Class member declaration for src_env.
      322    Env* src_env;
              3. member_decl: Class member declaration for dst_env.
      323    Env* dst_env;
              5. member_decl: Class member declaration for sync.
      324    bool sync;
              7. member_decl: Class member declaration for rate_limiter.
      325    RateLimiter* rate_limiter;
              9. member_decl: Class member declaration for size_limit.
      326    uint64_t size_limit;
      327    std::promise<CopyOrCreateResult> result;
      328    std::function<void()> progress_callback;
      329
              2. uninit_member: Non-static class member src_env is not initialized in this constructor nor in any functions that it calls.
              4. uninit_member: Non-static class member dst_env is not initialized in this constructor nor in any functions that it calls.
              6. uninit_member: Non-static class member sync is not initialized in this constructor nor in any functions that it calls.
              8. uninit_member: Non-static class member rate_limiter is not initialized in this constructor nor in any functions that it calls.
      
      CID 1396123 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
      10. uninit_member: Non-static class member size_limit is not initialized in this constructor nor in any functions that it calls.
      330    CopyOrCreateWorkItem() {}
      ```
      5. struct RestoreAfterCopyOrCreateWorkItem
      ```
      struct RestoreAfterCopyOrCreateWorkItem {
      410    std::future<CopyOrCreateResult> result;
              1. member_decl: Class member declaration for checksum_value.
      411    uint32_t checksum_value;
      
      CID 1396153 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
      2. uninit_member: Non-static class member checksum_value is not initialized in this constructor nor in any functions that it calls.
      412    RestoreAfterCopyOrCreateWorkItem() {}
      ```
      Closes https://github.com/facebook/rocksdb/pull/3131
      
      Differential Revision: D6428556
      
      Pulled By: sagar0
      
      fbshipit-source-id: a86675444543eff028e3cae6942197a143a112c4
      81cf262f
  5. 13 Sep, 2017 1 commit
    • Andrew Kryczka's avatar
      support opening zero backups during engine init · f5148ade
      Andrew Kryczka authored
      Summary:
      There are internal users who open BackupEngine for writing new backups only, and they don't care whether old backups can be read or not. The condition `BackupableDBOptions::max_valid_backups_to_open == 0` should be supported (previously in df74b775 I made the mistake of choosing 0 as a special value to disable the limit).
      Closes https://github.com/facebook/rocksdb/pull/2819
      
      Differential Revision: D5751599
      
      Pulled By: ajkr
      
      fbshipit-source-id: e73ac19eb5d756d6b68601eae8e43407ee4f2752
      f5148ade
  6. 01 Sep, 2017 1 commit
    • Andrew Kryczka's avatar
      fix backup engine when latest backup corrupt · b97685ae
      Andrew Kryczka authored
      Summary:
      Backup engine is intentionally openable even when some backups are corrupt. Previously the engine could write new backups as long as the most recent backup wasn't corrupt. This PR makes the backup engine able to create new backups even when the most recent one is corrupt.
      
      We now maintain two ID instance variables:
      
      - `latest_backup_id_` is used when creating backup to choose the new ID
      - `latest_valid_backup_id_` is used when restoring latest backup since we want most recent valid one
      Closes https://github.com/facebook/rocksdb/pull/2804
      
      Differential Revision: D5734148
      
      Pulled By: ajkr
      
      fbshipit-source-id: db440707b31df2c7b084188aa5f6368449e10bcf
      b97685ae
  7. 22 Jul, 2017 2 commits
  8. 16 Jul, 2017 1 commit
  9. 14 Jun, 2017 1 commit
  10. 23 May, 2017 1 commit
  11. 28 Apr, 2017 1 commit
  12. 25 Apr, 2017 1 commit
    • Andrew Kryczka's avatar
      Reunite checkpoint and backup core logic · e5e545a0
      Andrew Kryczka authored
      Summary:
      These code paths forked when checkpoint was introduced by copy/pasting the core backup logic. Over time they diverged and bug fixes were sometimes applied to one but not the other (like fix to include all relevant WALs for 2PC), or it required extra effort to fix both (like fix to forge CURRENT file). This diff reunites the code paths by extracting the core logic into a function, CreateCustomCheckpoint(), that is customizable via callbacks to implement both checkpoint and backup.
      
      Related changes:
      
      - flush_before_backup is now forcibly enabled when 2PC is enabled
      - Extracted CheckpointImpl class definition into a header file. This is so the function, CreateCustomCheckpoint(), can be called by internal rocksdb code but not exposed to users.
      - Implemented more functions in DummyDB/DummyLogFile (in backupable_db_test.cc) that are used by CreateCustomCheckpoint().
      Closes https://github.com/facebook/rocksdb/pull/1932
      
      Differential Revision: D4622986
      
      Pulled By: ajkr
      
      fbshipit-source-id: 157723884236ee3999a682673b64f7457a7a0d87
      e5e545a0
  13. 20 Apr, 2017 1 commit
    • Andrew Kryczka's avatar
      Limit backups opened · df74b775
      Andrew Kryczka authored
      Summary:
      This was requested by a customer who wants to proactively monitor whether any valid backups are available. The existing performance was poor because Open() serially reads every small meta-file (one per backup), which was slow on HDFS.
      
      Now we only read the minimum number of meta-files to find `max_valid_backups_to_open` valid backups. The customer mentioned above can just set it to one.
      Closes https://github.com/facebook/rocksdb/pull/2151
      
      Differential Revision: D4882564
      
      Pulled By: ajkr
      
      fbshipit-source-id: cb0edf9e8ac693e4d5f24902e725a011ed8c0c2f
      df74b775
  14. 04 Apr, 2017 1 commit
  15. 31 Mar, 2017 1 commit
  16. 30 Mar, 2017 1 commit
  17. 16 Mar, 2017 1 commit
    • Islam AbdelRahman's avatar
      Add macros to include file name and line number during Logging · e1916368
      Islam AbdelRahman authored
      Summary:
      current logging
      ```
      2017/03/14-14:20:30.393432 7fedde9f5700 (Original Log Time 2017/03/14-14:20:30.393414) [default] Level summary: base level 1 max bytes base 268435456 files[1 0 0 0 0 0 0] max score 0.25
      2017/03/14-14:20:30.393438 7fedde9f5700 [JOB 2] Try to delete WAL files size 61417909, prev total WAL file size 73820858, number of live WAL files 2.
      2017/03/14-14:20:30.393464 7fedde9f5700 [DEBUG] [JOB 2] Delete /dev/shm/old_logging//MANIFEST-000001 type=3 #1 -- OK
      2017/03/14-14:20:30.393472 7fedde9f5700 [DEBUG] [JOB 2] Delete /dev/shm/old_logging//000003.log type=0 #3 -- OK
      2017/03/14-14:20:31.427103 7fedd49f1700 [default] New memtable created with log file: #9. Immutable memtables: 0.
      2017/03/14-14:20:31.427179 7fedde9f5700 [JOB 3] Syncing log #6
      2017/03/14-14:20:31.427190 7fedde9f5700 (Original Log Time 2017/03/14-14:20:31.427170) Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
      2017/03/14-14:20:31.
      Closes https://github.com/facebook/rocksdb/pull/1990
      
      Differential Revision: D4708695
      
      Pulled By: IslamAbdelRahman
      
      fbshipit-source-id: cb8968f
      e1916368
  18. 03 Mar, 2017 1 commit
    • Andrew Kryczka's avatar
      Statistic for how often rate limiter is drained · 7c80a6d7
      Andrew Kryczka authored
      Summary:
      This is the metric I plan to use for adaptive rate limiting. The statistics are updated only if the rate limiter is drained by flush or compaction. I believe (but am not certain) that this is the normal case.
      
      The Statistics object is passed in RateLimiter::Request() to avoid requiring changes to client code, which would've been necessary if we passed it in the RateLimiter constructor.
      Closes https://github.com/facebook/rocksdb/pull/1946
      
      Differential Revision: D4646489
      
      Pulled By: ajkr
      
      fbshipit-source-id: d8e0161
      7c80a6d7
  19. 24 Feb, 2017 1 commit
  20. 23 Feb, 2017 1 commit
    • Andrew Kryczka's avatar
      Gracefully handle previous backup interrupted · 5040414e
      Andrew Kryczka authored
      Summary:
      As the last step in backup creation, the .tmp directory is renamed omitting the .tmp suffix. In case the process terminates before this, the .tmp directory will be left behind. Even if this happens, we want future backups to succeed, so I added some checks/cleanup for this case.
      Closes https://github.com/facebook/rocksdb/pull/1896
      
      Differential Revision: D4597323
      
      Pulled By: ajkr
      
      fbshipit-source-id: 48900d8
      5040414e
  21. 07 Feb, 2017 1 commit
    • Dmitri Smirnov's avatar
      Windows thread · 0a4cdde5
      Dmitri Smirnov authored
      Summary:
      introduce new methods into a public threadpool interface,
      - allow submission of std::functions as they allow greater flexibility.
      - add Joining methods to the implementation to join scheduled and submitted jobs with
        an option to cancel jobs that did not start executing.
      - Remove ugly `#ifdefs` between pthread and std implementation, make it uniform.
      - introduce pimpl for a drop in replacement of the implementation
      - Introduce rocksdb::port::Thread typedef which is a replacement for std::thread.  On Posix Thread defaults as before std::thread.
      - Implement WindowsThread that allocates memory in a more controllable manner than windows std::thread with a replaceable implementation.
      - should be no functionality changes.
      Closes https://github.com/facebook/rocksdb/pull/1823
      
      Differential Revision: D4492902
      
      Pulled By: siying
      
      fbshipit-source-id: c74cb11
      0a4cdde5
  22. 01 Jan, 2017 1 commit
  23. 23 Dec, 2016 1 commit
    • Aaron Gao's avatar
      direct io write support · 972f96b3
      Aaron Gao authored
      Summary:
      rocksdb direct io support
      
      ```
      [gzh@dev11575.prn2 ~/rocksdb] ./db_bench -benchmarks=fillseq --num=1000000
      Initializing RocksDB Options from the specified file
      Initializing RocksDB Options from command-line flags
      RocksDB:    version 5.0
      Date:       Wed Nov 23 13:17:43 2016
      CPU:        40 * Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
      CPUCache:   25600 KB
      Keys:       16 bytes each
      Values:     100 bytes each (50 bytes after compression)
      Entries:    1000000
      Prefix:    0 bytes
      Keys per prefix:    0
      RawSize:    110.6 MB (estimated)
      FileSize:   62.9 MB (estimated)
      Write rate: 0 bytes/second
      Compression: Snappy
      Memtablerep: skip_list
      Perf Level: 1
      WARNING: Assertions are enabled; benchmarks unnecessarily slow
      ------------------------------------------------
      Initializing RocksDB Options from the specified file
      Initializing RocksDB Options from command-line flags
      DB path: [/tmp/rocksdbtest-112628/dbbench]
      fillseq      :       4.393 micros/op 227639 ops/sec;   25.2 MB/s
      
      [gzh@dev11575.prn2 ~/roc
      Closes https://github.com/facebook/rocksdb/pull/1564
      
      Differential Revision: D4241093
      
      Pulled By: lightmark
      
      fbshipit-source-id: 98c29e3
      972f96b3
  24. 17 Dec, 2016 1 commit
    • Daniel Black's avatar
      gcc-7 requires include <functional> for std::function · 816c1e30
      Daniel Black authored
      Summary:
      Fixes compile error:
      
      In file included from ./util/statistics.h:17:0,
                       from ./util/stop_watch.h:8,
                       from ./util/perf_step_timer.h:9,
                       from ./util/iostats_context_imp.h:8,
                       from ./util/posix_logger.h:27,
                       from ./port/util_logger.h:18,
                       from ./db/auto_roll_logger.h:15,
                       from db/auto_roll_logger.cc:6:
      ./util/thread_local.h:65:16: error: 'function' in namespace 'std' does not name a template type
         typedef std::function<void(void*, void*)> FoldFunc;
      Closes https://github.com/facebook/rocksdb/pull/1656
      
      Differential Revision: D4318702
      
      Pulled By: yiwu-arbug
      
      fbshipit-source-id: 8c5d17a
      816c1e30
  25. 15 Dec, 2016 1 commit
    • Andrew Kryczka's avatar
      Fail BackupEngine::Open upon meta-file read error · 83f9a6fd
      Andrew Kryczka authored
      Summary:
      We used to treat any failure to read a backup's meta-file as if the backup were corrupted; however, we should distinguish corruption errors from errors in the backup Env. This fixes an issue where callers would get inconsistent results from GetBackupInfo() if they called it on an engine that encountered Env error during initialization. Now we fail Initialize() in this case so callers cannot invoke GetBackupInfo() on such engines.
      Closes https://github.com/facebook/rocksdb/pull/1654
      
      Differential Revision: D4318573
      
      Pulled By: ajkr
      
      fbshipit-source-id: f7a7c54
      83f9a6fd
  26. 13 Dec, 2016 1 commit
  27. 17 Nov, 2016 1 commit
  28. 07 Jul, 2016 1 commit
  29. 02 Jul, 2016 2 commits
  30. 30 Jun, 2016 1 commit
  31. 10 Jun, 2016 1 commit
    • Wanning Jiang's avatar
      Backup Options · 56887f6c
      Wanning Jiang authored
      Summary: Backup options file to private directory
      
      Test Plan:
      backupable_db_test.cc, BackupOptions
      	   Modify DB options by calling OpenDB for 3 times. Check the latest options file is in the right place. Also check no redundent files are backuped.
      
      Reviewers: andrewkr
      
      Reviewed By: andrewkr
      
      Subscribers: leveldb, dhruba, andrewkr
      
      Differential Revision: https://reviews.facebook.net/D59373
      56887f6c
  32. 04 Jun, 2016 1 commit
    • Uddipta Maity's avatar
      Adding support for sharing throttler between multiple backup and restores · 1147e5b0
      Uddipta Maity authored
      Summary:
      Rocksdb backup and restore rate limiting is currently done per backup/restore.
      So, it is difficult to control rate across multiple backup/restores. With this
      change, a throttler can be provided. If a throttler is provided, it is used.
      Otherwise, a new throttler is created based on the actual rate limits specified
      in the options.
      
      Test Plan: Added unit tests
      
      Reviewers: ldemailly, andrewkr, sdong
      
      Reviewed By: andrewkr
      
      Subscribers: igor, yiwu, andrewkr, dhruba
      
      Differential Revision: https://reviews.facebook.net/D56265
      1147e5b0
  33. 21 May, 2016 1 commit
  34. 27 Apr, 2016 1 commit
    • Andrew Kryczka's avatar
      Retrieve file size from proper Env · 1995e34d
      Andrew Kryczka authored
      Summary:
      When db_env_ != backup_env_, InsertPathnameToSizeBytes() would
      use the wrong Env during backup creation. This happened because this function
      used backup_env_ instead of db_env_ to get WAL/data file sizes.
      
      This diff adds an argument to InsertPathnameToSizeBytes() indicating which Env
      to use.
      
      Test Plan: ran @anirbanb's BackupTestTool
      
      Reviewers: sdong
      
      Reviewed By: sdong
      
      Subscribers: andrewkr, dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D57159
      1995e34d
  35. 19 Apr, 2016 1 commit
    • Andrew Kryczka's avatar
      Delete deprecated *BackupableDB interface for backups · 40b840f2
      Andrew Kryczka authored
      Summary:
      This interface is redundant and has been deprecated for a while.
      It's also unused internally. Let's delete it.
      
      I moved the comments to the corresponding functions in BackupEngine/
      BackupEngineReadOnly. This caused the diff tool to not work cleanly.
      
      Test Plan:
      unit tests
      
        $ ./backupable_db_test
      
      Reviewers: yhchiang, sdong
      
      Reviewed By: sdong
      
      Subscribers: andrewkr, dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D56331
      40b840f2
  36. 16 Apr, 2016 1 commit
    • sdong's avatar
      Fix backupable_db_test test cases that can't run by itself · cea8ed97
      sdong authored
      Summary:
      Several of backupable_db_test fails if running standalone, because of directory missing. Fix it by:
      (1) garbage collector skips shared directory if it doesn't exit
      (2) BackupableDBTest.Issue921Test to create the parent directory of the backup directory fist.
      
      Test Plan: Run the tests individually and make sure they pass
      
      Subscribers: leveldb, andrewkr, dhruba
      
      Differential Revision: https://reviews.facebook.net/D56829
      cea8ed97
  37. 09 Apr, 2016 1 commit
    • Andrew Kryczka's avatar
      Fix build errors for windows · 114a1b87
      Andrew Kryczka authored
      Summary:
      - Need to use unsigned long long for 64-bit literals on windows
      - Need size_t for backup meta-file length since clang doesn't let us assign size_t to int
      
      Test Plan: backupable_db_test and options_test
      
      Reviewers: IslamAbdelRahman, yhchiang, sdong
      
      Reviewed By: sdong
      
      Subscribers: andrewkr, dhruba, leveldb
      
      Differential Revision: https://reviews.facebook.net/D56391
      114a1b87
  38. 02 Apr, 2016 1 commit
    • Uddipta Maity's avatar
      Rocksdb backup can store optional application specific metadata · b55e2165
      Uddipta Maity authored
      Summary:
      Rocksdb backup engine maintains metadata about backups in separate files. But,
      there was no way to add extra application specific data to it. Adding support
      for that.
      In some use cases, applications decide to restore a backup based on some
      metadata. This will help those cases to cheaply decide whether to restore or
      not.
      
      Test Plan:
      Added a unit test. Existing ones are passing
      
      Sample meta file for BinaryMetadata test-
      
      ```
      
      1459454043
      0
      metadata 6162630A64656600676869
      2
      private/1/MANIFEST-000001 crc32 1184723444
      private/1/CURRENT crc32 3505765120
      
      ```
      
      Reviewers: sdong, ldemailly, andrewkr
      
      Reviewed By: andrewkr
      
      Subscribers: andrewkr, dhruba, ldemailly
      
      Differential Revision: https://reviews.facebook.net/D56007
      b55e2165