Package com.linkedin.venice.hadoop

Interface InputDataInfoProvider

All Superinterfaces:

java.lang.AutoCloseable, java.io.Closeable

All Known Implementing Classes:

DefaultInputDataInfoProvider, KafkaInputDataInfoProvider

public interface InputDataInfoProvider
extends java.io.Closeable

This interface lets users get input data information

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	InputDataInfoProvider.InputDataInfo	A POJO that contains input data information (schema information and input data file size)

Method Summary

Modifier and Type	Method	Description
org.apache.avro.Schema	extractAvroSubSchema(org.apache.avro.Schema origin, java.lang.String fieldName)
long	getInputLastModificationTime(java.lang.String inputUri)
void	initZstdConfig(int numFiles)
static void	loadZstdTrainingSamples(VeniceRecordIterator recordIterator, PushJobZstdConfig pushJobZstdConfig)	This function loads training samples from recordReader abstraction for building the Zstd dictionary.
byte[]	trainZstdDictionary()
InputDataInfoProvider.InputDataInfo	validateInputAndGetInfo(java.lang.String inputUri)

Methods inherited from interface java.io.Closeable

close

Method Detail

validateInputAndGetInfo

InputDataInfoProvider.InputDataInfo validateInputAndGetInfo(java.lang.String inputUri)
                                                     throws java.lang.Exception

Throws:

java.lang.Exception

initZstdConfig

void initZstdConfig(int numFiles)

loadZstdTrainingSamples

static void loadZstdTrainingSamples(VeniceRecordIterator recordIterator,
                                    PushJobZstdConfig pushJobZstdConfig)

This function loads training samples from recordReader abstraction for building the Zstd dictionary.

Parameters:

recordIterator - The data accessor of input records.

trainZstdDictionary

byte[] trainZstdDictionary()

extractAvroSubSchema

org.apache.avro.Schema extractAvroSubSchema(org.apache.avro.Schema origin,
                                            java.lang.String fieldName)

getInputLastModificationTime

long getInputLastModificationTime(java.lang.String inputUri)
                           throws java.io.IOException

Throws:

java.io.IOException