com.datastax.oss.dsbulk.connectors.commons

Class AbstractFileBasedConnector

java.lang.Object

com.datastax.oss.dsbulk.connectors.commons.AbstractFileBasedConnector

All Implemented Interfaces:

Connector, AutoCloseable

public abstract class AbstractFileBasedConnector
extends Object
implements Connector

A parent class for connectors that read from and write to text-based files.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static interface	AbstractFileBasedConnector.RecordReader A reader for Records.
protected static interface	AbstractFileBasedConnector.RecordWriter A writer for Records.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	compression
protected static String	COMPRESSION
protected Charset	encoding
protected static String	ENCODING
protected static String	FILE_NAME_FORMAT
protected static String	FILE_NAME_PATTERN
protected AtomicInteger	fileCounter
protected String	fileNameFormat
protected List<URL>	files
protected static String	MAX_CONCURRENT_FILES
protected static String	MAX_RECORDS
protected int	maxConcurrentFiles
protected long	maxRecords
protected AtomicInteger	nextWriterIndex
protected String	pattern
protected boolean	read
protected boolean	recursive
protected static String	RECURSIVE
protected int	resourceCount
protected boolean	retainRecordSources
protected List<Path>	roots
protected AbstractFileBasedConnector.RecordWriter	singleWriter
protected static String	SKIP_RECORDS
protected long	skipRecords
protected static String	URL
protected static String	URLFILE
protected List<URL>	urls
protected Deque<AbstractFileBasedConnector.RecordWriter>	writers
protected List<AbstractFileBasedConnector.RecordWriter>	writersToClose

Constructor Summary

Constructors

Constructor and Description
AbstractFileBasedConnector()

Method Summary

Modifier and Type	Method and Description
protected reactor.core.publisher.Flux<Record>	applyPerFileLimits(reactor.core.publisher.Flux<Record> records) Applies per-file limits to a stream of records coming from readSingleFile(java.net.URL, URI).
void	close()
void	configure(com.typesafe.config.Config settings, boolean read, boolean retainRecordSources)
protected abstract String	getConnectorName() Returns the connector name, e.g.
protected URL	getOrCreateDestinationURL() Returns the URL that the connector should write to.
void	init()
protected boolean	isDataSizeSamplingAvailable() Checks whether it is safe to perform data size sampling on this connector's data source.
protected List<URL>	loadURLs(com.typesafe.config.Config settings) Validates and computes the list of URLs to be read or written.
protected abstract AbstractFileBasedConnector.RecordReader	newSingleFileReader(URL url, URI resource) Returns a new AbstractFileBasedConnector.RecordReader instance; cannot be null.
protected abstract AbstractFileBasedConnector.RecordWriter	newSingleFileWriter() Returns a new AbstractFileBasedConnector.RecordWriter instance; cannot be null.
protected void	processURLsForRead() Inspects the list or URLs as loaded by loadURLs(Config) and determines exactly what files and folders need to be read.
protected void	processURLsForWrite() Inspects the list or URLs as loaded by loadURLs(Config) and determines if the connector should write to a single file, or to a directory of files.
org.reactivestreams.Publisher<Resource>	read()
int	readConcurrency()
protected reactor.core.publisher.Flux<Record>	readSingleFile(URL url, URI resource) Reads a single text file accessible through the given URL.
protected reactor.core.publisher.Flux<URL>	scanRootDirectory(Path root) Scans a directory for readable files and returns the files found as a stream.
Function<org.reactivestreams.Publisher<Record>,org.reactivestreams.Publisher<Record>>	write()
int	writeConcurrency()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.datastax.oss.dsbulk.connectors.api.Connector

getRecordMetadata, supports

Field Detail

URL

protected static final String URL

See Also:

Constant Field Values

URLFILE

protected static final String URLFILE

See Also:

Constant Field Values

COMPRESSION

protected static final String COMPRESSION

See Also:

Constant Field Values

ENCODING

protected static final String ENCODING

See Also:

Constant Field Values

FILE_NAME_PATTERN

protected static final String FILE_NAME_PATTERN

See Also:

Constant Field Values

SKIP_RECORDS

protected static final String SKIP_RECORDS

See Also:

Constant Field Values

MAX_RECORDS

protected static final String MAX_RECORDS

See Also:

Constant Field Values

MAX_CONCURRENT_FILES

protected static final String MAX_CONCURRENT_FILES

See Also:

Constant Field Values

RECURSIVE

protected static final String RECURSIVE

See Also:

Constant Field Values

FILE_NAME_FORMAT

protected static final String FILE_NAME_FORMAT

See Also:

Constant Field Values

read

protected boolean read

retainRecordSources

protected boolean retainRecordSources

urls

protected List<URL> urls

roots

protected List<Path> roots

files

protected List<URL> files

encoding

protected Charset encoding

compression

protected String compression

fileNameFormat

protected String fileNameFormat

recursive

protected boolean recursive

pattern

protected String pattern

skipRecords

protected long skipRecords

maxRecords

protected long maxRecords

resourceCount

protected int resourceCount

maxConcurrentFiles

protected int maxConcurrentFiles

writers

protected Deque<AbstractFileBasedConnector.RecordWriter> writers

singleWriter

protected AbstractFileBasedConnector.RecordWriter singleWriter

writersToClose

protected List<AbstractFileBasedConnector.RecordWriter> writersToClose

fileCounter

protected AtomicInteger fileCounter

nextWriterIndex

protected AtomicInteger nextWriterIndex

Constructor Detail

AbstractFileBasedConnector

public AbstractFileBasedConnector()

Method Detail

readConcurrency

public int readConcurrency()

Specified by:

readConcurrency in interface Connector

writeConcurrency

public int writeConcurrency()

Specified by:

writeConcurrency in interface Connector

configure

public void configure(@NonNull
                      com.typesafe.config.Config settings,
                      boolean read,
                      boolean retainRecordSources)

Specified by:

configure in interface Connector

init

public void init()
          throws URISyntaxException,
                 IOException

Specified by:

init in interface Connector

Throws:

URISyntaxException

IOException

read

@NonNull
public org.reactivestreams.Publisher<Resource> read()

Specified by:

read in interface Connector

write

@NonNull
public Function<org.reactivestreams.Publisher<Record>,org.reactivestreams.Publisher<Record>> write()

Specified by:

write in interface Connector

close

public void close()

Specified by:

close in interface Connector

Specified by:

close in interface AutoCloseable

getConnectorName

@NonNull
protected abstract String getConnectorName()

Returns the connector name, e.g. "csv" for the CSV connector. This should match the programmatic connector name and also its configuration namespace; that is, if this method returns "foo", then the connector configuration namespace is expected to be connector.foo.

readSingleFile

@NonNull
protected reactor.core.publisher.Flux<Record> readSingleFile(@NonNull
                                                                      URL url,
                                                                      @NonNull
                                                                      URI resource)

Reads a single text file accessible through the given URL. Used during the data reading phase.

Implementors should not care about maxRecords and skipRecords, these will be applied later on, see applyPerFileLimits(Flux).

Parameters:

url - The URL to read; must not be null; must be accessible and readable (but not necessarily hosted on the local filesystem).

resource - The resource URI.

Returns:

A stream of Records; never null but may be empty.

newSingleFileReader

@NonNull
protected abstract AbstractFileBasedConnector.RecordReader newSingleFileReader(@NonNull
                                                                                        URL url,
                                                                                        URI resource)
                                                                                 throws IOException

Returns a new AbstractFileBasedConnector.RecordReader instance; cannot be null. Only used when reading. Each invocation of this method is expected to return a newly-allocated instance. The reader is expected to be initialized already, and ready to emit its first record. It is possible to throw IOException if the reader cannot be initialized.

Throws:

IOException

newSingleFileWriter

@NonNull
protected abstract AbstractFileBasedConnector.RecordWriter newSingleFileWriter()

Returns a new AbstractFileBasedConnector.RecordWriter instance; cannot be null. Only used when writing. Each invocation of this method is expected to return a newly-allocated instance.

loadURLs

@NonNull
protected List<URL> loadURLs(@NonNull
                                      com.typesafe.config.Config settings)

Validates and computes the list of URLs to be read or written. This method honors both the URLFILE and URL configuration options.

Expected to be called during the configuration phase.

processURLsForRead

protected void processURLsForRead()
                           throws URISyntaxException,
                                  IOException

Inspects the list or URLs as loaded by loadURLs(Config) and determines exactly what files and folders need to be read.

Should be called at the beginning of the initialization process, but only when reading, never when writing.

This method expects that loadURLs(Config) has been previously called.

Throws:

URISyntaxException

IOException

processURLsForWrite

protected void processURLsForWrite()
                            throws URISyntaxException,
                                   IOException

Inspects the list or URLs as loaded by loadURLs(Config) and determines if the connector should write to a single file, or to a directory of files.

Should be called at the beginning of the initialization process, but only when writing, never when reading.

This method expects that loadURLs(Config) has been previously called, and also expects exactly one URL to be present, which can be either a directory or a file.

Throws:

URISyntaxException

IOException

scanRootDirectory

@NonNull
protected reactor.core.publisher.Flux<URL> scanRootDirectory(@NonNull
                                                                      Path root)

Scans a directory for readable files and returns the files found as a stream. Only used when reading, never when writing. Normally used as part of the actual data reading phase.

applyPerFileLimits

@NonNull
protected reactor.core.publisher.Flux<Record> applyPerFileLimits(@NonNull
                                                                          reactor.core.publisher.Flux<Record> records)

Applies per-file limits to a stream of records coming from readSingleFile(java.net.URL, URI).

getOrCreateDestinationURL

@NonNull
protected URL getOrCreateDestinationURL()

Returns the URL that the connector should write to. Not used for reads.

This can be either a single file or a directory of files. If the former, each invocation of this method will return the same URL; if the latter, each invocation of this method will generate a new URL inside the directory, with a unique file name.

isDataSizeSamplingAvailable

protected boolean isDataSizeSamplingAvailable()

Checks whether it is safe to perform data size sampling on this connector's data source. Data size sampling is usually not safe if the data can only be streamed once.

This implementation simply checks that none of the urls to read is reading from standard input, since standard input is not rewindable. Any other URL protocol is considered safe.

Returns:

true if it is safe to perform data size sampling, false otherwise.