de.jungblut.crawl

Class SequenceFileResultWriter<T extends FetchResult>

java.lang.Object

de.jungblut.crawl.SequenceFileResultWriter<T>

All Implemented Interfaces:

ResultWriter<T>, AutoCloseable

public class SequenceFileResultWriter<T extends FetchResult>
extends Object
implements ResultWriter<T>

Writes the result into a sequencefile "files/crawl/result.seq". It tab separates the outlinks in the sequencefile value, the key is the origin url.

Author:

thomas.jungblut

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.io.SequenceFile.Writer	writer

Constructor Summary

Constructors

Constructor and Description
SequenceFileResultWriter()

Method Summary

Modifier and Type	Method and Description
void	close()
org.apache.hadoop.fs.Path	getOutputPath()
void	open(org.apache.hadoop.conf.Configuration conf) Opens the given result writer with a configuration.
void	write(FetchResult result) Writes a single item to the output.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

writer

protected org.apache.hadoop.io.SequenceFile.Writer writer

Constructor Detail

SequenceFileResultWriter

public SequenceFileResultWriter()

Method Detail

open

public void open(org.apache.hadoop.conf.Configuration conf)
          throws IOException

Description copied from interface: ResultWriter

Opens the given result writer with a configuration.

Specified by:

open in interface ResultWriter<T extends FetchResult>

Throws:

IOException

write

public void write(FetchResult result)
           throws IOException

Description copied from interface: ResultWriter

Writes a single item to the output.

Specified by:

write in interface ResultWriter<T extends FetchResult>

Throws:

IOException

getOutputPath

public org.apache.hadoop.fs.Path getOutputPath()

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception