AbstractETL (Restful Harvester Library 8.1.0 API)

java.lang.Object
- de.gerdiproject.harvest.etls.AbstractETL<T,S>

Type Parameters:

T - the type of the extracted source data

S - the type of the transformed documents

All Implemented Interfaces:

IEventListener

Direct Known Subclasses:

AbstractIteratorETL
```
public abstract class AbstractETL<T,S>
extends Object
implements IEventListener
```
This class offers a skeleton for harvesting a repository to retrieve all of its metadata. The metadata can subsequently be sent to a search index via an ILoader. This most basic ETL class offers functions that can be controlled via REST requests from the ETLRestResource, as well as some utility objects that are required by all harvests. Subclasses must implement the concrete harvesting process.

Author:

Robin Weiss

Field Summary

Fields
Modifier and Type	Field and Description
`protected BooleanParameter`	`enabledParameter`
`protected IExtractor<T>`	`extractor`
`protected String`	`hash`
`protected TimestampedList<ETLHealth>`	`healthHistory`
`protected ILoader<S>`	`loader`
`protected org.slf4j.Logger`	`logger`
`protected String`	`name`
`protected TimestampedList<ETLState>`	`stateHistory`
`protected ITransformer<T,S>`	`transformer`

Constructor Summary

Constructors
Constructor and Description
`AbstractETL()` Constructor that initializes helper classes and fields.
`AbstractETL(String name)` Constructor that initializes helper classes and fields.

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`abortHarvest()` Aborts the harvesting process, allowing a new harvest to be started.
`void`	`addEventListeners()` Adds event listeners.
`void`	`cancelHarvest()` Cancels the ETL if it is queued to harvest, cleaning up readers and writers if necessary.
`protected abstract IExtractor<T>`	`createExtractor()` Creates an IExtractor for retrieving elements from the harvested repository to be used by the ITransformer.
`protected ILoader<S>`	`createLoader()` Creates an ILoader for sending the harvested documents to a search index.
`protected abstract ITransformer<T,S>`	`createTransformer()` Creates an ITransformer for transforming source elements to documents that can be used by the ILoader.
`protected void`	`finishHarvestExceptionally(Throwable reason)` This method is called after an ongoing harvest failed due to an exception.
`ETLJson`	`getAsJson()` Returns a simplified JSON representation of the ETL.
`Charset`	`getCharset()` Returns the charset of the harvested data.
`abstract int`	`getHarvestedCount()` Retrieves the number of documents that have been loaded.
`String`	`getHash()` Returns the checksum hash of the documents which are to be harvested.
`ETLHealth`	`getHealth()` Returns an enum that represents the health status of the ETL.
`int`	`getMaxNumberOfDocuments()` Returns the total number of documents that can possibly be harvested.
`String`	`getName()` Returns the name of the ETL.
`ETLState`	`getState()` Returns an enum that represents what the ETL is currently doing.
`void`	`harvest()` Starts the harvest.
`void`	`init(String moduleName)` Initializes the ETL, calculating the hash and maximum number of harvestable documents.
`protected String`	`initHash()` Computes a hash value of the files that are to be harvested, which is used for checking if the files have changed.
`protected int`	`initMaxNumberOfDocuments()` Calculates the total number of harvested documents.
`boolean`	`isEnabled()` Returns true if this ETL is enabled and initialized.
`void`	`loadFromJson(ETLJson json)` Loads the state from a JSON representation of this ETL.
`protected void`	`onContextDestroyed(ContextDestroyedEvent event)` The implementation of the context destroyed callback.
`protected void`	`onHarvestFinished(HarvestFinishedEvent event)` The implementation of the harvest finished callback.
`protected void`	`onParameterChanged(ParameterChangedEvent event)` The implementation of the parameter changed callback.
`protected void`	`onResetContext(ResetContextEvent event)` The implementation of the service reset callback.
`void`	`prepareHarvest()` Checks pre-conditions required for starting a harvest and updates the data that is to be extracted.
`protected void`	`registerParameters()` Registers all configurable parameters.
`void`	`removeEventListeners()` Removes all event listeners of this class.
`void`	`setHealth(ETLHealth health)` Changes the health status.
`void`	`setName(String name)` Changes the name of the ETL.
`void`	`setStatus(ETLState state)` Changes the status that represents what the ETL is currently doing.
`protected void`	`skipHarvest()` Marks the harvest as done.
`String`	`toString()`
`void`	`update()` Updates the harvested source documents, calculating the hash and maximum number of harvestable documents.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - extractor
```
protected IExtractor<T> extractor
```
  - transformer
```
protected ITransformer<T,S> transformer
```
  - loader
```
protected ILoader<S> loader
```
  - enabledParameter
```
protected volatile BooleanParameter enabledParameter
```
  - logger
```
protected final org.slf4j.Logger logger
```
  - name
```
protected String name
```
  - hash
```
protected volatile String hash
```
  - healthHistory
```
protected final TimestampedList<ETLHealth> healthHistory
```
  - stateHistory
```
protected final TimestampedList<ETLState> stateHistory
```
- Constructor Detail
  - AbstractETL
```
public AbstractETL()
```
    Constructor that initializes helper classes and fields. And uses the class name as ETL name.
  - AbstractETL
```
public AbstractETL(String name)
```
    Constructor that initializes helper classes and fields.
    
    Parameters:
    
    name - the name of this ETL
- Method Detail
  - createExtractor
```
protected abstract IExtractor<T> createExtractor()
```
    Creates an IExtractor for retrieving elements from the harvested repository to be used by the ITransformer.
    
    Returns:
    
    an IExtractor for retrieving elements from the harvested repository
  - createTransformer
```
protected abstract ITransformer<T,S> createTransformer()
```
    Creates an ITransformer for transforming source elements to documents that can be used by the ILoader.
    
    Returns:
    
    ITransformer for transforming source elements
  - createLoader
```
protected ILoader<S> createLoader()
```
    Creates an ILoader for sending the harvested documents to a search index.
    
    Returns:
    
    an ILoader for sending the harvested documents
  - registerParameters
```
protected void registerParameters()
```
    Registers all configurable parameters.
  - addEventListeners
```
public void addEventListeners()
```
    Description copied from interface: IEventListener
    
    Adds event listeners.
    
    Specified by:
    
    addEventListeners in interface IEventListener
  - removeEventListeners
```
public void removeEventListeners()
```
    Description copied from interface: IEventListener
    
    Removes all event listeners of this class.
    
    Specified by:
    
    removeEventListeners in interface IEventListener
  - loadFromJson
```
public void loadFromJson(ETLJson json)
```
    Loads the state from a JSON representation of this ETL.
    
    Parameters:
    
    json - a simplified JSON representation of the ETL
  - getAsJson
```
public ETLJson getAsJson()
```
    Returns a simplified JSON representation of the ETL.
    
    Returns:
    
    a simplified JSON representation of the ETL
  - abortHarvest
```
public void abortHarvest()
                  throws IllegalStateException
```
    Aborts the harvesting process, allowing a new harvest to be started.
    
    Throws:
    
    IllegalStateException - if the current process cannot be aborted
  - init
```
public void init(String moduleName)
          throws IllegalStateException
```
    Initializes the ETL, calculating the hash and maximum number of harvestable documents.
    
    Parameters:
    
    moduleName - the name of the harvester service
    
    Throws:
    
    IllegalStateException - thrown if init is called after the ETL is initialized
  - initMaxNumberOfDocuments
```
protected int initMaxNumberOfDocuments()
```
    Calculates the total number of harvested documents.
    
    Returns:
    
    the total number of documents that are to be harvested
  - initHash
```
protected String initHash()
```
    Computes a hash value of the files that are to be harvested, which is used for checking if the files have changed.
    
    Returns:
    
    a hash as a checksum of the data which is to be harvested, or null if no hash could be retrieved
  - update
```
public void update()
            throws ETLPreconditionException
```
    Updates the harvested source documents, calculating the hash and maximum number of harvestable documents.
    
    Throws:
    
    ETLPreconditionException
  - getMaxNumberOfDocuments
```
public int getMaxNumberOfDocuments()
```
    Returns the total number of documents that can possibly be harvested.
    
    Returns:
    
    the total number of documents that can possibly be harvested
  - prepareHarvest
```
public void prepareHarvest()
                    throws ETLPreconditionException
```
    Checks pre-conditions required for starting a harvest and updates the data that is to be extracted.
    
    Throws:
    
    ETLPreconditionException - thrown if the harvest cannot start
  - cancelHarvest
```
public void cancelHarvest()
```
    Cancels the ETL if it is queued to harvest, cleaning up readers and writers if necessary.
  - skipHarvest
```
protected void skipHarvest()
```
    Marks the harvest as done.
  - harvest
```
public final void harvest()
```
    Starts the harvest.
  - finishHarvestExceptionally
```
protected void finishHarvestExceptionally(Throwable reason)
```
    This method is called after an ongoing harvest failed due to an exception.
    
    Parameters:
    
    reason - the exception that caused the harvest to fail
  - getHash
```
public String getHash()
```
    Returns the checksum hash of the documents which are to be harvested.
    
    Returns:
    
    the checksum hash of the documents which are to be harvested
  - getState
```
public ETLState getState()
```
    Returns an enum that represents what the ETL is currently doing.
    
    Returns:
    
    an enum that represents the state of the ETL
  - setStatus
```
public void setStatus(ETLState state)
```
    Changes the status that represents what the ETL is currently doing.
    
    Parameters:
    
    state - a new status
  - getHealth
```
public ETLHealth getHealth()
```
    Returns an enum that represents the health status of the ETL.
    
    Returns:
    
    an enum that represents the health status of the ETL
  - setHealth
```
public void setHealth(ETLHealth health)
```
    Changes the health status.
    
    Parameters:
    
    health - the new health status value
  - getHarvestedCount
```
public abstract int getHarvestedCount()
```
    Retrieves the number of documents that have been loaded.
    
    Returns:
    
    the number of documents that have been loaded
  - getName
```
public final String getName()
```
    Returns the name of the ETL.
    
    Returns:
    
    the name of the ETL
  - setName
```
public final void setName(String name)
```
    Changes the name of the ETL. This method should not be called manually. It is required by the ETLManager.
    
    Parameters:
    
    name - the new name of the ETL
  - getCharset
```
public Charset getCharset()
```
    Returns the charset of the harvested data.
    
    Returns:
    
    the charset of the harvested data
  - isEnabled
```
public boolean isEnabled()
```
    Returns true if this ETL is enabled and initialized.
    
    Returns:
    
    true if this ETL is enabled and initialized
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - onResetContext
```
protected void onResetContext(ResetContextEvent event)
```
    The implementation of the service reset callback. Attempts to cancel ongoing harvests.
    
    Parameters:
    
    event - the event that triggered the callback
  - onHarvestFinished
```
protected void onHarvestFinished(HarvestFinishedEvent event)
```
    The implementation of the harvest finished callback. Sets the status to IDLE if it is DONE.
    
    Parameters:
    
    event - the event that triggered the callback
  - onParameterChanged
```
protected void onParameterChanged(ParameterChangedEvent event)
```
    The implementation of the parameter changed callback.
    
    Parameters:
    
    event - the event that triggered the callback
  - onContextDestroyed
```
protected void onContextDestroyed(ContextDestroyedEvent event)
```
    The implementation of the context destroyed callback.
    
    Parameters:
    
    event - the event that triggered the callback

Class AbstractETL<T,S>

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

extractor

transformer

loader

enabledParameter

logger

name

hash

healthHistory

stateHistory

Constructor Detail

AbstractETL

AbstractETL

Method Detail

createExtractor

createTransformer

createLoader

registerParameters

addEventListeners

removeEventListeners

loadFromJson

getAsJson

abortHarvest

init

initMaxNumberOfDocuments

initHash

update

getMaxNumberOfDocuments

prepareHarvest

cancelHarvest

skipHarvest

harvest

finishHarvestExceptionally

getHash

getState

setStatus

getHealth

setHealth

getHarvestedCount

getName

setName

getCharset

isEnabled

toString

onResetContext

onHarvestFinished

onParameterChanged

onContextDestroyed