cn.spark2fire.jscrapy.scheduler

类 FileCacheQueueScheduler

java.lang.Object

cn.spark2fire.jscrapy.scheduler.DuplicateRemovedScheduler

cn.spark2fire.jscrapy.scheduler.FileCacheQueueScheduler

所有已实现的接口:

MonitorableScheduler, Scheduler, Closeable, AutoCloseable

public class FileCacheQueueScheduler
extends DuplicateRemovedScheduler
implements MonitorableScheduler, Closeable

Store urls and cursor in files so that a Spider can resume the status when shutdown.

从以下版本开始:

0.2.0

作者:

code4crafter@gmail.com

字段概要

字段

限定符和类型	字段和说明
private AtomicInteger	cursor
private String	fileCursor
private PrintWriter	fileCursorWriter
private String	filePath
private String	fileUrlAllName
private PrintWriter	fileUrlWriter
private ScheduledExecutorService	flushThreadPool
private AtomicBoolean	inited
private BlockingQueue<Request>	queue
private Task	task
private Set<String>	urls

从类继承的字段 cn.spark2fire.jscrapy.scheduler.DuplicateRemovedScheduler

logger

构造器概要

构造器

构造器和说明
FileCacheQueueScheduler(String filePath)

方法概要

限定符和类型	方法和说明
void	close()
protected Request	deserializeRequest(String line)
private void	flush()
private String	getFileName(String filename)
int	getLeftRequestsCount(Task task)
int	getTotalRequestsCount(Task task)
private void	init(Task task)
private void	initDuplicateRemover()
private void	initFlushThread()
private void	initWriter()
Request	poll(Task task)
protected void	pushWhenNoDuplicate(Request request, Task task)
private void	readCursorFile()
private void	readFile()
private void	readUrlFile()
protected String	serializeRequest(Request request)

从类继承的方法 cn.spark2fire.jscrapy.scheduler.DuplicateRemovedScheduler

getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 cn.spark2fire.jscrapy.scheduler.Scheduler

push

字段详细资料

filePath

private String filePath

fileUrlAllName

private String fileUrlAllName

task

private Task task

fileCursor

private String fileCursor

fileUrlWriter

private PrintWriter fileUrlWriter

fileCursorWriter

private PrintWriter fileCursorWriter

cursor

private AtomicInteger cursor

inited

private AtomicBoolean inited

queue

private BlockingQueue<Request> queue

urls

private Set<String> urls

flushThreadPool

private ScheduledExecutorService flushThreadPool

构造器详细资料

FileCacheQueueScheduler

public FileCacheQueueScheduler(String filePath)

方法详细资料

flush

private void flush()

init

private void init(Task task)

initDuplicateRemover

private void initDuplicateRemover()

initFlushThread

private void initFlushThread()

initWriter

private void initWriter()

readFile

private void readFile()

readUrlFile

private void readUrlFile()
                  throws IOException

抛出:

IOException

readCursorFile

private void readCursorFile()
                     throws IOException

抛出:

IOException

close

public void close()
           throws IOException

指定者:

close 在接口中 Closeable

指定者:

close 在接口中 AutoCloseable

抛出:

IOException

getFileName

private String getFileName(String filename)

pushWhenNoDuplicate

protected void pushWhenNoDuplicate(Request request,
                                   Task task)

覆盖:

pushWhenNoDuplicate 在类中 DuplicateRemovedScheduler

poll

public Request poll(Task task)

指定者:

poll 在接口中 Scheduler

getLeftRequestsCount

public int getLeftRequestsCount(Task task)

指定者:

getLeftRequestsCount 在接口中 MonitorableScheduler

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

指定者:

getTotalRequestsCount 在接口中 MonitorableScheduler

serializeRequest

protected String serializeRequest(Request request)

deserializeRequest

protected Request deserializeRequest(String line)