cn.spark2fire.jscrapy.scheduler

类 RedisScheduler

java.lang.Object

cn.spark2fire.jscrapy.scheduler.DuplicateRemovedScheduler

cn.spark2fire.jscrapy.scheduler.RedisScheduler

所有已实现的接口:

DuplicateRemover, MonitorableScheduler, Scheduler

直接已知子类:

RedisPriorityScheduler

public class RedisScheduler
extends DuplicateRemovedScheduler
implements MonitorableScheduler, DuplicateRemover

Use Redis as url scheduler for distributed crawlers.

从以下版本开始:

0.2.0

作者:

code4crafter@gmail.com

字段概要

字段

限定符和类型	字段和说明
private static String	ITEM_PREFIX
protected redis.clients.jedis.JedisPool	pool
private static String	QUEUE_PREFIX
private static String	SET_PREFIX

从类继承的字段 cn.spark2fire.jscrapy.scheduler.DuplicateRemovedScheduler

logger

构造器概要

构造器

构造器和说明
RedisScheduler(redis.clients.jedis.JedisPool pool)
RedisScheduler(String host)

方法概要

限定符和类型	方法和说明
private boolean	checkForAdditionalInfo(Request request)
protected String	getItemKey(Task task)
int	getLeftRequestsCount(Task task)
protected String	getQueueKey(Task task)
protected String	getSetKey(Task task)
int	getTotalRequestsCount(Task task)
boolean	isDuplicate(Request request, Task task)
Request	poll(Task task)
protected void	pushWhenNoDuplicate(Request request, Task task)
void	resetDuplicateCheck(Task task)

从类继承的方法 cn.spark2fire.jscrapy.scheduler.DuplicateRemovedScheduler

getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 cn.spark2fire.jscrapy.scheduler.Scheduler

push

字段详细资料

pool

protected redis.clients.jedis.JedisPool pool

QUEUE_PREFIX

private static final String QUEUE_PREFIX

另请参阅:

常量字段值

SET_PREFIX

private static final String SET_PREFIX

另请参阅:

常量字段值

ITEM_PREFIX

private static final String ITEM_PREFIX

另请参阅:

常量字段值

构造器详细资料

RedisScheduler

public RedisScheduler(String host)

RedisScheduler

public RedisScheduler(redis.clients.jedis.JedisPool pool)

方法详细资料

resetDuplicateCheck

public void resetDuplicateCheck(Task task)

指定者:

resetDuplicateCheck 在接口中 DuplicateRemover

isDuplicate

public boolean isDuplicate(Request request,
                           Task task)

指定者:

isDuplicate 在接口中 DuplicateRemover

pushWhenNoDuplicate

protected void pushWhenNoDuplicate(Request request,
                                   Task task)

覆盖:

pushWhenNoDuplicate 在类中 DuplicateRemovedScheduler

checkForAdditionalInfo

private boolean checkForAdditionalInfo(Request request)

poll

public Request poll(Task task)

指定者:

poll 在接口中 Scheduler

getSetKey

protected String getSetKey(Task task)

getQueueKey

protected String getQueueKey(Task task)

getItemKey

protected String getItemKey(Task task)

getLeftRequestsCount

public int getLeftRequestsCount(Task task)

指定者:

getLeftRequestsCount 在接口中 MonitorableScheduler

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

指定者:

getTotalRequestsCount 在接口中 DuplicateRemover

指定者:

getTotalRequestsCount 在接口中 MonitorableScheduler