de.l3s.icrawl.crawler.frontier

Class FileBasedFrontier

java.lang.Object

de.l3s.icrawl.crawler.frontier.BaseFrontier

de.l3s.icrawl.crawler.frontier.FileBasedFrontier

All Implemented Interfaces:

Frontier, Closeable, AutoCloseable

public class FileBasedFrontier
extends BaseFrontier
implements Frontier

Field Summary

Fields inherited from class de.l3s.icrawl.crawler.frontier.BaseFrontier

emptyQueue, incoming, lock, outgoing, seenUrls, totalIncoming

Constructor Summary

Constructors

Constructor and Description
FileBasedFrontier(File queueDirectory, com.codahale.metrics.MetricRegistry metrics, int numQueues, boolean persist)

Method Summary

Modifier and Type	Method and Description
void	close()
protected Optional<CrawlUrl>	popInternal()
protected void	pushInternal(CrawlUrl url)

Methods inherited from class de.l3s.icrawl.crawler.frontier.BaseFrontier

pop, push

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface de.l3s.icrawl.crawler.frontier.Frontier

pop, push

Constructor Detail

FileBasedFrontier

public FileBasedFrontier(File queueDirectory,
                         com.codahale.metrics.MetricRegistry metrics,
                         int numQueues,
                         boolean persist)
                  throws IOException

Throws:

IOException

Method Detail

pushInternal

protected void pushInternal(CrawlUrl url)

Specified by:

pushInternal in class BaseFrontier

popInternal

protected Optional<CrawlUrl> popInternal()

Specified by:

popInternal in class BaseFrontier

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class BaseFrontier

Throws:

IOException