de.gerdiproject.harvest.etls.loaders

Class AbstractURLLoader<S extends de.gerdiproject.harvest.IDocument>

java.lang.Object

de.gerdiproject.harvest.etls.loaders.AbstractIteratorLoader<S>

de.gerdiproject.harvest.etls.loaders.AbstractURLLoader<S>

Type Parameters:

S - The type of the sent documents

All Implemented Interfaces:

ILoader<Iterator<S>>

Direct Known Subclasses:

ElasticSearchLoader

public abstract class AbstractURLLoader<S extends de.gerdiproject.harvest.IDocument>
extends AbstractIteratorLoader<S>

This abstract class offers a basis for sending documents to a search index via a REST request.

Author:

Robin Weiss

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,de.gerdiproject.harvest.IDocument>	batchMap
protected Charset	charset
protected HashGenerator	hashGenerator
protected org.slf4j.Logger	logger
protected IntegerParameter	maxBatchSizeParam
protected StringParameter	urlParam

Fields inherited from class de.gerdiproject.harvest.etls.loaders.AbstractIteratorLoader

dedicatedEtl, hasLoadedDocuments

Constructor Summary

Constructors

Constructor and Description
AbstractURLLoader() Constructor that initializes the Logger.

Method Summary

Modifier and Type	Method and Description
protected String	checkPreconditionErrors() Checks if the loader can start
void	clear() Closes potentially open readers/writers and finishes the Load process if it is still ongoing.
protected String	getCredentials() Retrieves the credentials that may be necessary for to authenticate the loader with the URL.
protected abstract int	getSizeOfDocument(String documentId, de.gerdiproject.harvest.IDocument document) Calculates the size of a single document within the batch in bytes.
protected String	getUrl() Returns the loader target URL as a string.
void	init(AbstractETL<?,?> etl) Initializes the loader for a new harvest.
void	load(Iterator<S> documents) Loads the document to a targeted search index.
protected abstract void	loadBatch(Map<String,de.gerdiproject.harvest.IDocument> documents) The core of the loading function which is to be implemented by subclasses.
void	loadElement(S document) Loads a single element of the Iterator.
protected void	tryLoadingBatch() Sends documents to an external place.
void	unregisterParameters() If the Loader created parameters in the Configuration, this method should unregister all of them.

Methods inherited from class de.gerdiproject.harvest.etls.loaders.AbstractIteratorLoader

loadElementAndIncrement

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected final org.slf4j.Logger logger

batchMap

protected final Map<String,de.gerdiproject.harvest.IDocument> batchMap

maxBatchSizeParam

protected final IntegerParameter maxBatchSizeParam

urlParam

protected final StringParameter urlParam

hashGenerator

protected final HashGenerator hashGenerator

charset

protected volatile Charset charset

Constructor Detail

AbstractURLLoader

public AbstractURLLoader()

Constructor that initializes the Logger.

Method Detail

unregisterParameters

public void unregisterParameters()

Description copied from interface: ILoader

If the Loader created parameters in the Configuration, this method should unregister all of them. It's called when a different Loader is used.

init

public void init(AbstractETL<?,?> etl)

Description copied from interface: ILoader

Initializes the loader for a new harvest.

Specified by:

init in interface ILoader<Iterator<S extends de.gerdiproject.harvest.IDocument>>

Overrides:

init in class AbstractIteratorLoader<S extends de.gerdiproject.harvest.IDocument>

Parameters:

etl - the AbstractETL to which the loader belongs

load

public void load(Iterator<S> documents)
          throws LoaderException

Description copied from interface: ILoader

Loads the document to a targeted search index.

Specified by:

load in interface ILoader<Iterator<S extends de.gerdiproject.harvest.IDocument>>

Overrides:

load in class AbstractIteratorLoader<S extends de.gerdiproject.harvest.IDocument>

Parameters:

documents - the document that is to be loaded

Throws:

LoaderException - when the load did not work properly

loadElement

public void loadElement(S document)
                 throws LoaderException

Description copied from class: AbstractIteratorLoader

Loads a single element of the Iterator.

Specified by:

loadElement in class AbstractIteratorLoader<S extends de.gerdiproject.harvest.IDocument>

Parameters:

document - a document that is to be loaded

Throws:

LoaderException - when the load failed

clear

public void clear()

Description copied from interface: ILoader

Closes potentially open readers/writers and finishes the Load process if it is still ongoing.

getSizeOfDocument

protected abstract int getSizeOfDocument(String documentId,
                                         de.gerdiproject.harvest.IDocument document)

Calculates the size of a single document within the batch in bytes.

Parameters:

documentId - the unique identifier of the document

document - the document of which the size is measured

Returns:

the size of the document in bytes

loadBatch

protected abstract void loadBatch(Map<String,de.gerdiproject.harvest.IDocument> documents)

The core of the loading function which is to be implemented by subclasses. This method should load each document of the map to the destination URL via a REST request.

Parameters:

documents - a map of documentIDs to documents that are to be loaded, the values may also be null, in which case the document is to be removed from the index

checkPreconditionErrors

protected String checkPreconditionErrors()

Checks if the loader can start

Returns:

a message explaining if the pre-conditions failed

tryLoadingBatch

protected void tryLoadingBatch()
                        throws LoaderException

Sends documents to an external place.

Throws:

LoaderException - when the batch could not be loaded

getCredentials

protected String getCredentials()

Retrieves the credentials that may be necessary for to authenticate the loader with the URL.

Returns:

the credentials that may be necessary to authenticate the loader with the URL

getUrl

protected String getUrl()

Returns the loader target URL as a string.

Returns:

the loader target URL as a string