de.l3s.icrawl.crawler.frontier

Class BaseFrontier

java.lang.Object

de.l3s.icrawl.crawler.frontier.BaseFrontier

All Implemented Interfaces:

Frontier, Closeable, AutoCloseable

Direct Known Subclasses:

FileBasedFrontier, InMemoryFrontier

public abstract class BaseFrontier
extends Object
implements Frontier, Closeable

Field Summary

Fields

Modifier and Type	Field and Description
protected com.codahale.metrics.Meter	emptyQueue
protected com.codahale.metrics.Meter	incoming
protected Object	lock
protected com.codahale.metrics.Meter	outgoing
protected com.google.common.hash.BloomFilter<CharSequence>	seenUrls
protected long	totalIncoming

Constructor Summary

Constructors

Constructor and Description
BaseFrontier(com.codahale.metrics.MetricRegistry metrics)

Method Summary

Modifier and Type	Method and Description
void	close()
Optional<CrawlUrl>	pop()
protected abstract Optional<CrawlUrl>	popInternal()
void	push(Collection<CrawlUrl> urls)
protected abstract void	pushInternal(CrawlUrl url)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

seenUrls

protected final com.google.common.hash.BloomFilter<CharSequence> seenUrls

incoming

protected final com.codahale.metrics.Meter incoming

outgoing

protected final com.codahale.metrics.Meter outgoing

emptyQueue

protected com.codahale.metrics.Meter emptyQueue

totalIncoming

protected long totalIncoming

lock

protected final Object lock

Constructor Detail

BaseFrontier

public BaseFrontier(com.codahale.metrics.MetricRegistry metrics)

Method Detail

push

public void push(Collection<CrawlUrl> urls)

Specified by:

push in interface Frontier

pop

public Optional<CrawlUrl> pop()

Specified by:

pop in interface Frontier

popInternal

protected abstract Optional<CrawlUrl> popInternal()

pushInternal

protected abstract void pushInternal(CrawlUrl url)

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException