Package com.linkedin.davinci.repository
Class NativeMetadataRepository
- java.lang.Object
-
- com.linkedin.davinci.repository.NativeMetadataRepository
-
- All Implemented Interfaces:
ClusterInfoProvider,ReadOnlySchemaRepository,ReadOnlyStoreRepository,SubscriptionBasedReadOnlyStoreRepository,VeniceResource
- Direct Known Subclasses:
ThinClientMetaStoreBasedRepository
public abstract class NativeMetadataRepository extends java.lang.Object implements SubscriptionBasedReadOnlyStoreRepository, ReadOnlySchemaRepository, ClusterInfoProvider
Venice in-house implementation of a read only metadata repository where callers can retrieve various metadata such as Store objects and their corresponding schemas. The implementers of this abstract class all relies on some flavors of Venice system store to carry the metadata from Venice internal components (source) to external consumers such as a DaVinci client (destination). This abstract class includes the implementation of an in-memory cache for all subscribed stores' metadata. Callers are served by the cache and the cache is refreshed periodically by updating it with methods provided by the implementers.
-
-
Field Summary
Fields Modifier and Type Field Description protected ClientConfigclientConfigprotected java.util.Map<java.lang.String,Store>subscribedStoreMap
-
Constructor Summary
Constructors Modifier Constructor Description protectedNativeMetadataRepository(ClientConfig clientConfig, VeniceProperties backendConfig)protectedNativeMetadataRepository(ClientConfig clientConfig, VeniceProperties backendConfig, java.time.Clock clock)
-
Method Summary
All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description voidclear()TODO: we may need to rename this function to be 'close' since this resource should not used any more after calling this function.java.util.List<Store>getAllStores()Get all stores in the current repositoryprotected SchemaDatagetAndCacheSchemaDataFromSystemStore(java.lang.String storeName)protected voidgetAndSetStoreConfigFromSystemStore(java.lang.String storeName)Get the store cluster config from system store and update the local cache with it.intgetBatchGetLimit(java.lang.String storeName)Get batch-get limit for the specified storeDerivedSchemaEntrygetDerivedSchema(java.lang.String storeName, int valueSchemaId, int derivedSchemaId)GeneratedSchemaIDgetDerivedSchemaId(java.lang.String storeName, java.lang.String derivedSchemaStr)Look up derived schema id and its corresponding value schema id by given store name and derived schema.java.util.Collection<DerivedSchemaEntry>getDerivedSchemas(java.lang.String storeName)static NativeMetadataRepositorygetInstance(ClientConfig clientConfig, VeniceProperties backendConfig)static NativeMetadataRepositorygetInstance(ClientConfig clientConfig, VeniceProperties backendConfig, ICProvider icProvider)SchemaEntrygetKeySchema(java.lang.String storeName)This function is used to retrieve key schema for the given store.DerivedSchemaEntrygetLatestDerivedSchema(java.lang.String storeName, int valueSchemaId)Get the most recent derived schema added to the given store and value schema idRmdSchemaEntrygetReplicationMetadataSchema(java.lang.String storeName, int valueSchemaId, int replicationMetadataVersionId)java.util.Collection<RmdSchemaEntry>getReplicationMetadataSchemas(java.lang.String storeName)protected SchemaDatagetSchemaDataFromMetaSystemStore(java.lang.String storeName)protected abstract SchemaDatagetSchemaDataFromSystemStore(java.lang.String storeName)StoregetStore(java.lang.String storeName)Get one store by given name from repository.protected StoreConfiggetStoreConfigFromMetaSystemStore(java.lang.String storeName)protected abstract StoreConfiggetStoreConfigFromSystemStore(java.lang.String storeName)protected abstract StoregetStoreFromSystemStore(java.lang.String storeName, java.lang.String clusterName)protected abstract StoreMetaValuegetStoreMetaValue(java.lang.String storeName, StoreMetaKey key)StoregetStoreOrThrow(java.lang.String storeName)SchemaEntrygetSupersetOrLatestValueSchema(java.lang.String storeName)Get the most recent value schema or superset value schema if one exists.SchemaEntrygetSupersetSchema(java.lang.String storeName)Get the superset value schema for a given store.longgetTotalStoreReadQuota()Get total read quota of all stores.SchemaEntrygetValueSchema(java.lang.String storeName, int id)This function is used to retrieve the value schema for the given store and value schema id.intgetValueSchemaId(java.lang.String storeName, java.lang.String valueSchemaStr)This function is used to retrieve value schema id for the given store and schema.protected SchemaEntrygetValueSchemaInternally(java.lang.String storeName, int id)java.util.Collection<SchemaEntry>getValueSchemas(java.lang.String storeName)This function is used to retrieve all the value schemas for the given store.java.lang.StringgetVeniceCluster(java.lang.String storeName)Get the associated Venice cluster name given a Venice store name.booleanhasStore(java.lang.String storeName)Whether the store exists or not.booleanhasValueSchema(java.lang.String storeName, int id)This function is used to check whether the value schema id is valid in the given store.booleanisReadComputationEnabled(java.lang.String storeName)Whether computation is enabled for the specified store.protected voidnotifyStoreChanged(Store store)protected voidnotifyStoreCreated(Store store)protected voidnotifyStoreDeleted(Store store)protected StoreputStore(Store newStore)voidrefresh()This method will be triggered periodically to keep the store/schema information up-to-date.StorerefreshOneStore(java.lang.String storeName)Selective refresh operation which fetches one store from ZKvoidregisterStoreDataChangedListener(StoreDataChangedListener listener)Register store data change listener.protected StoreremoveStore(java.lang.String storeName)protected voidremoveStoreSchema(java.lang.String storeName)This function is used to remove schema entry for the given store from local cache, and related listeners as well.voidstart()voidsubscribe(java.lang.String storeName)voidunregisterStoreDataChangedListener(StoreDataChangedListener listener)Unregister store data change listener.voidunsubscribe(java.lang.String storeName)-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface com.linkedin.venice.meta.ReadOnlySchemaRepository
getLatestDerivedSchema
-
Methods inherited from interface com.linkedin.venice.meta.ReadOnlyStoreRepository
waitVersion, waitVersion
-
-
-
-
Field Detail
-
clientConfig
protected final ClientConfig clientConfig
-
subscribedStoreMap
protected final java.util.Map<java.lang.String,Store> subscribedStoreMap
-
-
Constructor Detail
-
NativeMetadataRepository
protected NativeMetadataRepository(ClientConfig clientConfig, VeniceProperties backendConfig)
-
NativeMetadataRepository
protected NativeMetadataRepository(ClientConfig clientConfig, VeniceProperties backendConfig, java.time.Clock clock)
-
-
Method Detail
-
start
public void start()
-
getInstance
public static NativeMetadataRepository getInstance(ClientConfig clientConfig, VeniceProperties backendConfig)
-
getInstance
public static NativeMetadataRepository getInstance(ClientConfig clientConfig, VeniceProperties backendConfig, ICProvider icProvider)
-
subscribe
public void subscribe(java.lang.String storeName) throws java.lang.InterruptedException- Specified by:
subscribein interfaceSubscriptionBasedReadOnlyStoreRepository- Throws:
java.lang.InterruptedException
-
unsubscribe
public void unsubscribe(java.lang.String storeName)
- Specified by:
unsubscribein interfaceSubscriptionBasedReadOnlyStoreRepository
-
getStore
public Store getStore(java.lang.String storeName)
Description copied from interface:ReadOnlyStoreRepositoryGet one store by given name from repository.- Specified by:
getStorein interfaceReadOnlyStoreRepository- Parameters:
storeName- name of wanted store.- Returns:
- Store for given name.
-
getStoreOrThrow
public Store getStoreOrThrow(java.lang.String storeName) throws VeniceNoStoreException
- Specified by:
getStoreOrThrowin interfaceReadOnlyStoreRepository- Throws:
VeniceNoStoreException
-
hasStore
public boolean hasStore(java.lang.String storeName)
Description copied from interface:ReadOnlyStoreRepositoryWhether the store exists or not.- Specified by:
hasStorein interfaceReadOnlyStoreRepository- Parameters:
storeName- store name- Returns:
-
refreshOneStore
public Store refreshOneStore(java.lang.String storeName)
Description copied from interface:ReadOnlyStoreRepositorySelective refresh operation which fetches one store from ZK- Specified by:
refreshOneStorein interfaceReadOnlyStoreRepository- Parameters:
storeName- store name- Returns:
- the newly refreshed store
-
getAllStores
public java.util.List<Store> getAllStores()
Description copied from interface:ReadOnlyStoreRepositoryGet all stores in the current repository- Specified by:
getAllStoresin interfaceReadOnlyStoreRepository- Returns:
-
getTotalStoreReadQuota
public long getTotalStoreReadQuota()
Description copied from interface:ReadOnlyStoreRepositoryGet total read quota of all stores.- Specified by:
getTotalStoreReadQuotain interfaceReadOnlyStoreRepository
-
registerStoreDataChangedListener
public void registerStoreDataChangedListener(StoreDataChangedListener listener)
Description copied from interface:ReadOnlyStoreRepositoryRegister store data change listener.- Specified by:
registerStoreDataChangedListenerin interfaceReadOnlyStoreRepository
-
unregisterStoreDataChangedListener
public void unregisterStoreDataChangedListener(StoreDataChangedListener listener)
Description copied from interface:ReadOnlyStoreRepositoryUnregister store data change listener.- Specified by:
unregisterStoreDataChangedListenerin interfaceReadOnlyStoreRepository
-
getBatchGetLimit
public int getBatchGetLimit(java.lang.String storeName)
Description copied from interface:ReadOnlyStoreRepositoryGet batch-get limit for the specified store- Specified by:
getBatchGetLimitin interfaceReadOnlyStoreRepository- Returns:
-
isReadComputationEnabled
public boolean isReadComputationEnabled(java.lang.String storeName)
Description copied from interface:ReadOnlyStoreRepositoryWhether computation is enabled for the specified store.- Specified by:
isReadComputationEnabledin interfaceReadOnlyStoreRepository- Parameters:
storeName- store name- Returns:
-
getKeySchema
public SchemaEntry getKeySchema(java.lang.String storeName)
This function is used to retrieve key schema for the given store. If key schema for the given store doesn't exist, will return null; Otherwise, it will return the key schema;- Specified by:
getKeySchemain interfaceReadOnlySchemaRepository- Returns:
- null, if key schema for the given store doesn't exist; key schema entry, otherwise;
-
getValueSchema
public SchemaEntry getValueSchema(java.lang.String storeName, int id)
This function is used to retrieve the value schema for the given store and value schema id. Caller shouldn't modify the returned SchemeEntry- Specified by:
getValueSchemain interfaceReadOnlySchemaRepository- Returns:
- null, if the schema doesn't exist; value schema entry, otherwise;
-
hasValueSchema
public boolean hasValueSchema(java.lang.String storeName, int id)This function is used to check whether the value schema id is valid in the given store.- Specified by:
hasValueSchemain interfaceReadOnlySchemaRepository
-
getValueSchemaId
public int getValueSchemaId(java.lang.String storeName, java.lang.String valueSchemaStr)This function is used to retrieve value schema id for the given store and schema.- Specified by:
getValueSchemaIdin interfaceReadOnlySchemaRepository- Returns:
SchemaData.INVALID_VALUE_SCHEMA_ID, if the schema doesn't exist in the given store; schema id (int), if the schema exists in the given store
-
getValueSchemas
public java.util.Collection<SchemaEntry> getValueSchemas(java.lang.String storeName)
This function is used to retrieve all the value schemas for the given store.- Specified by:
getValueSchemasin interfaceReadOnlySchemaRepository
-
getSupersetOrLatestValueSchema
public SchemaEntry getSupersetOrLatestValueSchema(java.lang.String storeName)
Description copied from interface:ReadOnlySchemaRepositoryGet the most recent value schema or superset value schema if one exists.- Specified by:
getSupersetOrLatestValueSchemain interfaceReadOnlySchemaRepository
-
getSupersetSchema
public SchemaEntry getSupersetSchema(java.lang.String storeName)
Description copied from interface:ReadOnlySchemaRepositoryGet the superset value schema for a given store. Each store has at most one active superset schema. Specifically a store must have some features enabled (e.g. read compute, write compute) to have a superset value schema which evolves as new value schemas are added.- Specified by:
getSupersetSchemain interfaceReadOnlySchemaRepository- Returns:
- Superset value schema or
nullif store {@param storeName} does not have any superset value schema.
-
getDerivedSchemaId
public GeneratedSchemaID getDerivedSchemaId(java.lang.String storeName, java.lang.String derivedSchemaStr)
Description copied from interface:ReadOnlySchemaRepositoryLook up derived schema id and its corresponding value schema id by given store name and derived schema. This is likely used by clients that write to Venice- Specified by:
getDerivedSchemaIdin interfaceReadOnlySchemaRepository- Returns:
- a pair where the first value is value schema id and the second value is derived schema id
-
getDerivedSchema
public DerivedSchemaEntry getDerivedSchema(java.lang.String storeName, int valueSchemaId, int derivedSchemaId)
- Specified by:
getDerivedSchemain interfaceReadOnlySchemaRepository
-
getDerivedSchemas
public java.util.Collection<DerivedSchemaEntry> getDerivedSchemas(java.lang.String storeName)
- Specified by:
getDerivedSchemasin interfaceReadOnlySchemaRepository
-
getLatestDerivedSchema
public DerivedSchemaEntry getLatestDerivedSchema(java.lang.String storeName, int valueSchemaId)
Description copied from interface:ReadOnlySchemaRepositoryGet the most recent derived schema added to the given store and value schema id- Specified by:
getLatestDerivedSchemain interfaceReadOnlySchemaRepository
-
getReplicationMetadataSchema
public RmdSchemaEntry getReplicationMetadataSchema(java.lang.String storeName, int valueSchemaId, int replicationMetadataVersionId)
- Specified by:
getReplicationMetadataSchemain interfaceReadOnlySchemaRepository
-
getReplicationMetadataSchemas
public java.util.Collection<RmdSchemaEntry> getReplicationMetadataSchemas(java.lang.String storeName)
- Specified by:
getReplicationMetadataSchemasin interfaceReadOnlySchemaRepository
-
refresh
public void refresh()
This method will be triggered periodically to keep the store/schema information up-to-date.- Specified by:
refreshin interfaceVeniceResource
-
clear
public void clear()
TODO: we may need to rename this function to be 'close' since this resource should not used any more after calling this function.- Specified by:
clearin interfaceVeniceResource
-
getAndSetStoreConfigFromSystemStore
protected void getAndSetStoreConfigFromSystemStore(java.lang.String storeName)
Get the store cluster config from system store and update the local cache with it. Different implementation will get the data differently but should all populate the store cluster config map.
-
getStoreConfigFromSystemStore
protected abstract StoreConfig getStoreConfigFromSystemStore(java.lang.String storeName)
-
getStoreFromSystemStore
protected abstract Store getStoreFromSystemStore(java.lang.String storeName, java.lang.String clusterName)
-
getStoreMetaValue
protected abstract StoreMetaValue getStoreMetaValue(java.lang.String storeName, StoreMetaKey key)
-
getStoreConfigFromMetaSystemStore
protected StoreConfig getStoreConfigFromMetaSystemStore(java.lang.String storeName)
-
getSchemaDataFromMetaSystemStore
protected SchemaData getSchemaDataFromMetaSystemStore(java.lang.String storeName)
-
removeStore
protected Store removeStore(java.lang.String storeName)
-
notifyStoreCreated
protected void notifyStoreCreated(Store store)
-
notifyStoreDeleted
protected void notifyStoreDeleted(Store store)
-
notifyStoreChanged
protected void notifyStoreChanged(Store store)
-
getAndCacheSchemaDataFromSystemStore
protected SchemaData getAndCacheSchemaDataFromSystemStore(java.lang.String storeName)
-
getValueSchemaInternally
protected SchemaEntry getValueSchemaInternally(java.lang.String storeName, int id)
-
getSchemaDataFromSystemStore
protected abstract SchemaData getSchemaDataFromSystemStore(java.lang.String storeName)
-
removeStoreSchema
protected void removeStoreSchema(java.lang.String storeName)
This function is used to remove schema entry for the given store from local cache, and related listeners as well.
-
getVeniceCluster
public java.lang.String getVeniceCluster(java.lang.String storeName)
Description copied from interface:ClusterInfoProviderGet the associated Venice cluster name given a Venice store name.- Specified by:
getVeniceClusterin interfaceClusterInfoProvider- Returns:
- the cluster name that the store belongs to or null if such information cannot be provided.
-
-