Skip to content
  • Islam AbdelRahman's avatar
    Support IngestExternalFile (remove AddFile restrictions) · 869ae5d7
    Islam AbdelRahman authored
    Summary:
    Changes in the diff
    
    API changes:
    - Introduce IngestExternalFile to replace AddFile (I think this make the API more clear)
    - Introduce IngestExternalFileOptions (This struct will encapsulate the options for ingesting the external file)
    - Deprecate AddFile() API
    
    Logic changes:
    - If our file overlap with the memtable we will flush the memtable
    - We will find the first level in the LSM tree that our file key range overlap with the keys in it
    - We will find the lowest level in the LSM tree above the the level we found in step 2 that our file can fit in and ingest our file in it
    - We will assign a global sequence number to our new file
    - Remove AddFile restrictions by using global sequence numbers
    
    Other changes:
    - Refactor all AddFile logic to be encapsulated in ExternalSstFileIngestionJob
    
    Test Plan:
    unit tests (still need to add more)
    addfile_stress (https://reviews.facebook.net/D65037)
    
    Reviewers: yiwu, andrewkr, lightmark, yhchiang, sdong
    
    Reviewed By: sdong
    
    Subscribers: jkedgar, hcz, andrewkr, dhruba
    
    Differential Revision: https://reviews.facebook.net/D65061
    869ae5d7