org.apache.gobblin.service.modules.dataset

Interface DatasetDescriptor

All Known Implementing Classes:

BaseDatasetDescriptor, FSDatasetDescriptor, HiveDatasetDescriptor, HttpDatasetDescriptor, SqlDatasetDescriptor

@Alpha
public interface DatasetDescriptor

The interface for dataset descriptors. Each dataset is described in terms of the following attributes:

platform (e.g. HDFS, ADLS, JDBC).

path, which describes the fully qualified name of the dataset.

a format descriptor, which encapsulates its representation (e.g. avro, csv), codec (e.g. gzip, deflate), and encryption config (e.g. aes_rotating, gpg).

Method Summary

Modifier and Type	Method and Description
boolean	contains(DatasetDescriptor other)
String	getDescription()
FormatConfig	getFormatConfig()
String	getPath() Returns the fully qualified name of a dataset.
String	getPlatform()
com.typesafe.config.Config	getRawConfig()
boolean	isRetentionApplied()

Method Detail

getPlatform

String getPlatform()

Returns:

the dataset platform i.e. the storage system backing the dataset (e.g. HDFS, ADLS, JDBC etc.)

getPath

String getPath()

Returns the fully qualified name of a dataset. The fully qualified name is the absolute directory path of a dataset when the dataset is backed by a FileSystem. In the case of a database table, it is dbName.tableName.

Returns:

dataset path.

getFormatConfig

FormatConfig getFormatConfig()

Returns:

storage format of the dataset.

isRetentionApplied

boolean isRetentionApplied()

Returns:

true if retention has been applied to the dataset.

getDescription

String getDescription()

Returns:

a human-readable description of the dataset.

contains

boolean contains(DatasetDescriptor other)

Returns:

true if this DatasetDescriptor contains the other DatasetDescriptor i.e. the datasets described by the other DatasetDescriptor is a subset of this DatasetDescriptor. This operation is non-commutative.

getRawConfig

com.typesafe.config.Config getRawConfig()

Returns:

the raw config.