Package com.linkedin.davinci.ingestion

Class IsolatedIngestionBackend

  • All Implemented Interfaces:
    IngestionBackend, java.io.Closeable, java.lang.AutoCloseable
    public class IsolatedIngestionBackend
extends DefaultIngestionBackend
implements IngestionBackend
    This class is the implementation of ingestion backend designed for ingestion isolation. It contains references to local ingestion components - including storage metadata service, storage service and store ingestion service that serves local ingestion, as well as ingestion request client that sends commands to isolated ingestion service process and ingestion listener that listens to ingestion reports from child process. Since RocksDB storage can only be owned by a single process, we have decided to keep metadata partition storage opened in child process and in the main process, we rely on MainIngestionStorageMetadataService to serve as the in-memory metadata cache and persist the metadata updates from main process to metadata partition in child process. Topic partition ingestion requests will first be sent to child process and after COMPLETED is reported, they will be re-subscribed in main process to serve read traffics for user application and receive future updates. The implementation of APIs in this class should consider the states in both main process and child process, as we need to make sure we send the command to the correct process which holds the target storage engine.