cn.spark2fire.jscrapy.scheduler

类 BloomFilterDuplicateRemover

java.lang.Object

cn.spark2fire.jscrapy.scheduler.BloomFilterDuplicateRemover

所有已实现的接口:

DuplicateRemover

public class BloomFilterDuplicateRemover
extends Object
implements DuplicateRemover

BloomFilterDuplicateRemover for huge number of urls.

从以下版本开始:

0.5.1

作者:

code4crafer@gmail.com

字段概要

字段

限定符和类型	字段和说明
private com.google.common.hash.BloomFilter<CharSequence>	bloomFilter
private AtomicInteger	counter
private int	expectedInsertions
private double	fpp

构造器概要

构造器

构造器和说明
BloomFilterDuplicateRemover(int expectedInsertions)
BloomFilterDuplicateRemover(int expectedInsertions, double fpp)

方法概要

限定符和类型	方法和说明
int	getTotalRequestsCount(Task task)
protected String	getUrl(Request request)
boolean	isDuplicate(Request request, Task task)
protected com.google.common.hash.BloomFilter<CharSequence>	rebuildBloomFilter()
void	resetDuplicateCheck(Task task)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

expectedInsertions

private int expectedInsertions

fpp

private double fpp

counter

private AtomicInteger counter

bloomFilter

private final com.google.common.hash.BloomFilter<CharSequence> bloomFilter

构造器详细资料

BloomFilterDuplicateRemover

public BloomFilterDuplicateRemover(int expectedInsertions)

BloomFilterDuplicateRemover

public BloomFilterDuplicateRemover(int expectedInsertions,
                                   double fpp)

参数:

expectedInsertions - the number of expected insertions to the constructed

fpp - the desired false positive probability (must be positive and less than 1.0)

方法详细资料

rebuildBloomFilter

protected com.google.common.hash.BloomFilter<CharSequence> rebuildBloomFilter()

isDuplicate

public boolean isDuplicate(Request request,
                           Task task)

指定者:

isDuplicate 在接口中 DuplicateRemover

getUrl

protected String getUrl(Request request)

resetDuplicateCheck

public void resetDuplicateCheck(Task task)

指定者:

resetDuplicateCheck 在接口中 DuplicateRemover

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

指定者:

getTotalRequestsCount 在接口中 DuplicateRemover