cn.spark2fire.jscrapy.downloader

类 PhantomJSDownloader

java.lang.Object

cn.spark2fire.jscrapy.downloader.AbstractDownloader

cn.spark2fire.jscrapy.downloader.PhantomJSDownloader

所有已实现的接口:

Downloader

public class PhantomJSDownloader
extends AbstractDownloader

this downloader is used to download pages which need to render the javascript

版本:

0.5.3

作者:

dolphineor@gmail.com

字段概要

字段

限定符和类型	字段和说明
private static String	crawlJsPath
private static org.slf4j.Logger	logger
private static String	phantomJsCommand
private int	retryNum
private int	threadNum

构造器概要

构造器

构造器和说明
PhantomJSDownloader()
PhantomJSDownloader(String phantomJsCommand) 添加新的构造函数，支持phantomjs自定义命令 example: phantomjs.exe 支持windows环境 phantomjs --ignore-ssl-errors=yes 忽略抓取地址是https时的一些错误 /usr/local/bin/phantomjs 命令的绝对路径，避免因系统环境变量引起的IOException
PhantomJSDownloader(String phantomJsCommand, String crawlJsPath) 新增构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js crawl.js start -- var system = require('system'); var url = system.args[1]; var page = require('webpage').create(); page.settings.loadImages = false; page.settings.resourceTimeout = 5000; page.open(url, function (status) { if (status !

方法概要

限定符和类型	方法和说明
Page	download(Request request, Task task)
protected String	getPage(Request request)
int	getRetryNum()
private void	initPhantomjsCrawlPath()
PhantomJSDownloader	setRetryNum(int retryNum)
void	setThread(int threadNum)

从类继承的方法 cn.spark2fire.jscrapy.downloader.AbstractDownloader

download, download, onError, onSuccess

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

logger

private static org.slf4j.Logger logger

crawlJsPath

private static String crawlJsPath

phantomJsCommand

private static String phantomJsCommand

retryNum

private int retryNum

threadNum

private int threadNum

构造器详细资料

PhantomJSDownloader

public PhantomJSDownloader()

PhantomJSDownloader

public PhantomJSDownloader(String phantomJsCommand)

添加新的构造函数，支持phantomjs自定义命令 example: phantomjs.exe 支持windows环境 phantomjs --ignore-ssl-errors=yes 忽略抓取地址是https时的一些错误 /usr/local/bin/phantomjs 命令的绝对路径，避免因系统环境变量引起的IOException

参数:

phantomJsCommand - phantomJsCommand

PhantomJSDownloader

public PhantomJSDownloader(String phantomJsCommand,
                           String crawlJsPath)

新增构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js

 crawl.js start --
 
   var system = require('system');
   var url = system.args[1];
   
   var page = require('webpage').create();
   page.settings.loadImages = false;
   page.settings.resourceTimeout = 5000;
   
   page.open(url, function (status) {
       if (status != 'success') {
           console.log("HTTP request failed!");
       } else {
           console.log(page.content);
       }
   
       page.close();
       phantom.exit();
   });
   
 -- crawl.js end
 

具体项目时可以将以上js代码复制下来使用 example: new PhantomJSDownloader("/your/path/phantomjs", "/your/path/crawl.js");

参数:

phantomJsCommand - phantomJsCommand

crawlJsPath - crawlJsPath

方法详细资料

initPhantomjsCrawlPath

private void initPhantomjsCrawlPath()

download

public Page download(Request request,
                     Task task)

setThread

public void setThread(int threadNum)

getPage

protected String getPage(Request request)

getRetryNum

public int getRetryNum()

setRetryNum

public PhantomJSDownloader setRetryNum(int retryNum)