de.jungblut.crawl

Class MultithreadedCrawler<T extends FetchResult>

java.lang.Object

de.jungblut.crawl.MultithreadedCrawler<T>

All Implemented Interfaces:

Crawler<T>

public final class MultithreadedCrawler<T extends FetchResult>
extends Object
implements Crawler<T>

Fast multithreaded crawler, will start a fixed threadpool of 32 threads each will be fed by 10 urls at once. Majorly designed for speed and to use all the available bandwidth. Based on other internet bandwidths, you may retune the parameters of threadpool sizes and how many items should be batched. For my 6k ADSL it works fine by 32 threads batched on 10 urls. You may scale this linearly up, since this class has almost no contention and small sequential code. It is also backed by a bloom filter to check if a URL was visited, so the memory footprint stays low.

Author:

thomas.jungblut

Constructor Summary

Constructors

Constructor and Description
MultithreadedCrawler(int fetches, Extractor<T> extractor, ResultWriter<T> writer) Constructs a new Multithreaded Crawler with 32 threads working on 10 url batches at each time.
MultithreadedCrawler(int threadPoolSize, int batchSize, int fetches, Extractor<T> extractor, ResultWriter<T> writer) Constructs a new Multithreaded Crawler.

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)
void	process(String... seedUrls) Starts the crawler, starting by the seedURL.
void	setup(int fetches, Extractor<T> extractor, ResultWriter<T> writer) Setups this crawler.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultithreadedCrawler

public MultithreadedCrawler(int threadPoolSize,
                            int batchSize,
                            int fetches,
                            Extractor<T> extractor,
                            ResultWriter<T> writer)
                     throws IOException

Constructs a new Multithreaded Crawler.

Parameters:

threadPoolSize - the number of threads to use.

batchSize - the number of URLs a batch for a thread should contain.

fetches - the number of urls to fetch.

extractor - the extraction logic.

writer - the writer.

Throws:

IOException

MultithreadedCrawler

public MultithreadedCrawler(int fetches,
                            Extractor<T> extractor,
                            ResultWriter<T> writer)
                     throws IOException

Constructs a new Multithreaded Crawler with 32 threads working on 10 url batches at each time.

Parameters:

fetches - the number of urls to fetch.

extractor - the extraction logic.

writer - the writer.

Throws:

IOException

Method Detail

setup

public final void setup(int fetches,
                        Extractor<T> extractor,
                        ResultWriter<T> writer)
                 throws IOException

Description copied from interface: Crawler

Setups this crawler.

Specified by:

setup in interface Crawler<T extends FetchResult>

Parameters:

fetches - how many maximum fetches it should do.

extractor - the given Extractor to extract a FetchResult.

writer - the ResultWriter to write the result to a sink.

Throws:

IOException

process

public final void process(String... seedUrls)
                   throws InterruptedException,
                          ExecutionException

Description copied from interface: Crawler

Starts the crawler, starting by the seedURL. The real logic is implemented by the crawler itself.

Specified by:

process in interface Crawler<T extends FetchResult>

Throws:

InterruptedException

ExecutionException

main

public static void main(String[] args)
                 throws InterruptedException,
                        ExecutionException,
                        IOException

Throws:

InterruptedException

ExecutionException

IOException