de.jungblut.crawl.extraction

Class OutlinkExtractor

java.lang.Object

de.jungblut.crawl.extraction.OutlinkExtractor

All Implemented Interfaces:

Extractor<FetchResult>

public final class OutlinkExtractor
extends Object
implements Extractor<FetchResult>

Outlink extractor, parses a page just for its outlinks.

Author:

thomas.jungblut

Constructor Summary

Constructors

Constructor and Description
OutlinkExtractor()

Method Summary

Modifier and Type	Method and Description
static String	consumeStream(InputStream stream) Consumes a given InputStream and returns a string consisting of the html code of the site.
FetchResult	extract(String realUrl) Extracts from a given URL all the content needed and return it.
static String	extractBaseUrl(String url) Extracts a base url from the given url (to make relative outlinks to absolute ones).
static HashSet<String>	extractOutlinks(String html, String url) Extracts outlinks of the given HTML doc in string.
static HashSet<String>	filter(HashSet<String> set, Pattern matcher) Filters outlinks from a parsed page that NOT matches the given matcher.
static InputStream	getConnection(String realUrl)
static boolean	isValid(String s) Checks if the site does not end with unparsable suffixes likes PDF and if its a valid url by extracting a base url at at index 0.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OutlinkExtractor

public OutlinkExtractor()

Method Detail

extract

public FetchResult extract(String realUrl)

Description copied from interface: Extractor

Extracts from a given URL all the content needed and return it. Null if nothing should be returned or could be parsed.

Specified by:

extract in interface Extractor<FetchResult>

getConnection

public static InputStream getConnection(String realUrl)
                                 throws IOException

Returns:

an opened stream.

Throws:

IOException

filter

public static HashSet<String> filter(HashSet<String> set,
                                     Pattern matcher)

Filters outlinks from a parsed page that NOT matches the given matcher.

extractOutlinks

public static HashSet<String> extractOutlinks(String html,
                                              String url)
                                       throws org.htmlparser.util.ParserException

Extracts outlinks of the given HTML doc in string.

Parameters:

html - the html to extract the outlinkts from.

url - the url where we found the current document.

Returns:

a set of outlinks.

Throws:

org.htmlparser.util.ParserException

consumeStream

public static String consumeStream(InputStream stream)
                            throws IOException

Consumes a given InputStream and returns a string consisting of the html code of the site.

Throws:

IOException

extractBaseUrl

public static String extractBaseUrl(String url)

Extracts a base url from the given url (to make relative outlinks to absolute ones).

Returns:

a base url or null if none was found.

isValid

public static boolean isValid(String s)

Checks if the site does not end with unparsable suffixes likes PDF and if its a valid url by extracting a base url at at index 0.