• Andrew Kryczka's avatar
    Handle concurrent manifest update and backup creation · 69c471bd
    Andrew Kryczka authored
    Fixed two related race conditions in backup creation.
    (1) CreateNewBackup() uses DB::DisableFileDeletions() to prevent table files
    from being deleted while it is copying; however, the MANIFEST file could still
    rotate during this time. The fix is to stop deleting the old manifest in the
    rotation logic. It will be deleted safely later when PurgeObsoleteFiles() runs
    (can only happen when file deletions are enabled).
    (2) CreateNewBackup() did not account for the CURRENT file being mutable.
    This is significant because the files returned by GetLiveFiles() contain a
    particular manifest filename, but the manifest to which CURRENT refers can
    change at any time. This causes problems when CURRENT changes between the call
    to GetLiveFiles() and when it's copied to the backup directory. To workaround this, I
    manually forge a CURRENT file referring to the manifest filename returned in
    (2) also applies to the checkpointing code, so let me know if this approach is
    good and I'll make the same change there.
    Test Plan:
    new test for roll manifest during backup creation.
    running the test before this change:
      $ ./backupable_db_test --gtest_filter=BackupableDBTest.ChangeManifestDuringBackupCreation
      IO error: /tmp/rocksdbtest-9383/backupable_db/MANIFEST-000001: No such file or directory
    running the test after this change:
      $ ./backupable_db_test --gtest_filter=BackupableDBTest.ChangeManifestDuringBackupCreation
      [ RUN      ] BackupableDBTest.ChangeManifestDuringBackupCreation
      [       OK ] BackupableDBTest.ChangeManifestDuringBackupCreation (2836 ms)
    Reviewers: IslamAbdelRahman, anthony, sdong
    Reviewed By: sdong
    Subscribers: dhruba, leveldb
    Differential Revision: https://reviews.facebook.net/D54711