de.jungblut.crawl

Class FetchResultPersister<T extends FetchResult>

java.lang.Object

de.jungblut.crawl.FetchResultPersister<T>

All Implemented Interfaces:

Runnable

public final class FetchResultPersister<T extends FetchResult>
extends Object
implements Runnable

Asynchronous persister thread, taking a resultwriter and handles the logic behind asynchronous writing to disk or an arbitrary sink implemented by the ResultWriter. The lifecycle is the following, you can submit an instance of this class as a thread and if you want to stop it you can call the stop() method. Then it will make the backed queue read-only and flush all items in it to the result writer.

Author:

thomas.jungblut

Constructor Summary

Constructors

Constructor and Description
FetchResultPersister(ResultWriter<T> resWriter)
FetchResultPersister(ResultWriter<T> resWriter, org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method and Description
void	add(T result) Add a crawled result to the back queue.
void	run() Run logic of this Runnable.
void	stop() Stop this persister and make the queue read-only.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FetchResultPersister

public FetchResultPersister(ResultWriter<T> resWriter)
                     throws IOException

Throws:

IOException

FetchResultPersister

public FetchResultPersister(ResultWriter<T> resWriter,
                            org.apache.hadoop.conf.Configuration conf)
                     throws IOException

Throws:

IOException

Method Detail

add

public final void add(T result)

Add a crawled result to the back queue. Notice that after stop() has been called, this queue will not accept any items anymore.

stop

public void stop()

Stop this persister and make the queue read-only.

run

public final void run()

Run logic of this Runnable. Basically if we haven't stopped, we will poll the queue, if there is no item anymore we will wait for 10 seconds. If we have received a stop() command, the queue will be freezed and all exisiting items will be flushed to disk. ResultWriter close is guaranteed to run in case of failure (guarded by finally).

Specified by:

run in interface Runnable