de.gerdiproject.harvest.utils.data

Class WebDataRetriever

java.lang.Object

de.gerdiproject.harvest.utils.data.WebDataRetriever

All Implemented Interfaces:

IDataRetriever

public class WebDataRetriever
extends Object
implements IDataRetriever

This class provides methods for reading files from the web.

Author:

Robin Weiss

Constructor Summary

Constructors

Constructor and Description
WebDataRetriever(com.google.gson.Gson gson, Charset charset) Constructor that sets the GSON (de-)serializer for reading and writing JSON objects, as well as the charset.
WebDataRetriever(com.google.gson.Gson gson, Charset charset, int timeout) Constructor that sets the GSON (de-)serializer for reading and writing JSON objects, as well as the charset and timeout.
WebDataRetriever(WebDataRetriever other) Constructor that copies settings from another WebDataRetriever.

Method Summary

Modifier and Type	Method and Description
org.jsoup.nodes.Document	getHtml(String url) Tries to parse the content from a specified path as a HTML document.
InputStream	getInputStream(HttpURLConnection connection) Returns the correct InputStream based on the Content-Encoding header of a connection.
<T> T	getObject(String url, Class<T> targetClass) Tries to parse the content from a specified path as a JSON object.
<T> T	getObject(String url, Type targetType) Tries to parse the content from a specified path as a JSON object.
Map<String,List<String>>	getRestHeader(RestRequestType method, String url, String body, String authorization, String contentType) Sends an authorized REST request with a specified body and returns the header fields.
String	getRestResponse(RestRequestType method, String url, String body, String authorization, String contentType) Sends an authorized REST request with a specified body and returns the response as a string.
String	getString(String url) Tries to parse the content from a specified path as a string.
HttpURLConnection	sendWebRequest(RestRequestType method, String urlString, String body, String authorization, String contentType, int retries) Sends a REST request with a specified body and returns the connection.
void	setCharset(Charset charset) Changes the charset used for (de-)serializing requests.
void	setTimeout(int timeout)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WebDataRetriever

public WebDataRetriever(com.google.gson.Gson gson,
                        Charset charset,
                        int timeout)

Constructor that sets the GSON (de-)serializer for reading and writing JSON objects, as well as the charset and timeout.

Parameters:

gson - the GSON (de-)serializer for reading and writing JSON objects

charset - the charset of the files to be read and written

timeout - the web request timeout in milliseconds

WebDataRetriever

public WebDataRetriever(com.google.gson.Gson gson,
                        Charset charset)

Constructor that sets the GSON (de-)serializer for reading and writing JSON objects, as well as the charset.

Parameters:

gson - the GSON (de-)serializer for reading and writing JSON objects

charset - the charset of the files to be read and written

WebDataRetriever

public WebDataRetriever(WebDataRetriever other)

Constructor that copies settings from another WebDataRetriever.

Parameters:

other - the WebDataRetriever of which the settings are copied

Method Detail

getString

public String getString(String url)

Description copied from interface: IDataRetriever

Tries to parse the content from a specified path as a string.

Specified by:

getString in interface IDataRetriever

Parameters:

url - the path to a HTML file

Returns:

a text, or null if the file could not be parsed

getObject

public <T> T getObject(String url,
                       Class<T> targetClass)

Description copied from interface: IDataRetriever

Tries to parse the content from a specified path as a JSON object.

Specified by:

getObject in interface IDataRetriever

Type Parameters:

T - the type of the object that is to be read

Parameters:

url - the path to a JSON file

targetClass - the class of the object that is read from disc

Returns:

an object, or null if the file could not be parsed

getObject

public <T> T getObject(String url,
                       Type targetType)

Description copied from interface: IDataRetriever

Tries to parse the content from a specified path as a JSON object.

Specified by:

getObject in interface IDataRetriever

Type Parameters:

T - the type of the object that is to be read

Parameters:

url - the path to a JSON file

targetType - the type of the object that is read from disc

Returns:

an object, or null if the file could not be parsed

getHtml

public org.jsoup.nodes.Document getHtml(String url)

Description copied from interface: IDataRetriever

Tries to parse the content from a specified path as a HTML document.

Specified by:

getHtml in interface IDataRetriever

Parameters:

url - the path to a HTML file

Returns:

a HTML document, or null if the file could not be parsed

getRestResponse

public String getRestResponse(RestRequestType method,
                              String url,
                              String body,
                              String authorization,
                              String contentType)
                       throws HTTPException,
                              IOException

Sends an authorized REST request with a specified body and returns the response as a string.

Parameters:

method - the request method that is being sent

url - the URL to which the request is being sent

body - the body of the request, or null if no body is to be sent

authorization - the base-64-encoded username and password, or null if no authorization is required

contentType - the contentType of the body

Returns:

the HTTP response as plain text

Throws:

HTTPException - thrown if the response code is not 2xx

IOException - thrown if the response output stream could not be created

getRestHeader

public Map<String,List<String>> getRestHeader(RestRequestType method,
                                              String url,
                                              String body,
                                              String authorization,
                                              String contentType)
                                       throws HTTPException,
                                              IOException

Sends an authorized REST request with a specified body and returns the header fields.

Parameters:

method - the request method that is being sent

url - the URL to which the request is being sent

body - the body of the request, or null if no body is to be sent

authorization - the base-64-encoded username and password, or null if no authorization is required

contentType - the contentType of the body

Returns:

the response header fields, or null if the response could not be parsed

Throws:

HTTPException - thrown if the response code is not 2xx

IOException - thrown if the response output stream could not be created

sendWebRequest

public HttpURLConnection sendWebRequest(RestRequestType method,
                                        String urlString,
                                        String body,
                                        String authorization,
                                        String contentType,
                                        int retries)
                                 throws IOException,
                                        HTTPException

Sends a REST request with a specified body and returns the connection.

Parameters:

method - the request method that is being sent

urlString - the URL to which the request is being sent

body - the body of the request, or null if no body is to be sent

authorization - the base-64-encoded username and password, or null if no authorization is required

contentType - the contentType of the body

retries - the number of retries if the request fails with a response code 5xx

Returns:

the connection to the host

Throws:

HTTPException - thrown if the response code is not 2xx

IOException - thrown if the response output stream could not be created

getInputStream

public InputStream getInputStream(HttpURLConnection connection)
                           throws IOException

Returns the correct InputStream based on the Content-Encoding header of a connection. Necessary to support compression.

Parameters:

connection - the connection to be checked

Returns:

an InputStream subclass

Throws:

IOException - thrown if InputStream is corrupted

setTimeout

public void setTimeout(int timeout)

setCharset

public void setCharset(Charset charset)

Description copied from interface: IDataRetriever

Changes the charset used for (de-)serializing requests.

Specified by:

setCharset in interface IDataRetriever

Parameters:

charset - the new charset