| 程序包 | 说明 |
|---|---|
| cn.wanghaomiao.seimi.core | |
| cn.wanghaomiao.seimi.def | |
| cn.wanghaomiao.seimi.http.hc | |
| cn.wanghaomiao.seimi.http.okhttp | |
| cn.wanghaomiao.seimi.struct | |
| cn.wanghaomiao.seimi.utils |
| 限定符和类型 | 方法和说明 |
|---|---|
Request |
SeimiQueue.bPop(String crawlerName)
阻塞式出队一个请求
|
| 限定符和类型 | 方法和说明 |
|---|---|
List<Request> |
SeimiCrawler.startRequests()
起始的Request,可以应对更复杂的情况,当
String[] startUrls();无法满足需求的情况下推荐使用 |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
SeimiQueue.addProcessed(Request req)
记录一个处理过的请求
|
void |
SeimiCrawler.handleErrorRequest(Request request)
当一个请求处理异常次数超过开发者所设置或是默认设置的最大重新处理次数时会调用该方法记录异常请求
|
boolean |
SeimiQueue.isProcessed(Request req)
判断一个URL是否处理过了
|
Response |
SeimiDownloader.process(Request request)
处理抓取请求生成response
|
boolean |
SeimiQueue.push(Request req)
入队一个请求
|
| 限定符和类型 | 方法和说明 |
|---|---|
Request |
DefaultRedisQueue.bPop(String crawlerName) |
Request |
DefaultLocalQueue.bPop(String crawlerName) |
| 限定符和类型 | 方法和说明 |
|---|---|
LinkedBlockingQueue<Request> |
DefaultLocalQueue.getQueue(String crawlerName) |
List<Request> |
BaseSeimiCrawler.startRequests() |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
DefaultRedisQueue.addProcessed(Request req) |
void |
DefaultLocalQueue.addProcessed(Request req) |
void |
BaseSeimiCrawler.handleErrorRequest(Request request) |
boolean |
DefaultRedisQueue.isProcessed(Request req) |
boolean |
DefaultLocalQueue.isProcessed(Request req) |
boolean |
DefaultRedisQueue.push(Request req) |
boolean |
DefaultLocalQueue.push(Request req) |
protected void |
BaseSeimiCrawler.push(Request request) |
| 限定符和类型 | 方法和说明 |
|---|---|
static org.apache.http.client.methods.RequestBuilder |
HcRequestGenerator.getHttpRequestBuilder(Request request,
CrawlerModel crawlerModel) |
Response |
HcDownloader.process(Request request) |
| 限定符和类型 | 方法和说明 |
|---|---|
static okhttp3.Request.Builder |
OkHttpRequestGenerator.getOkHttpRequesBuilder(Request seimiReq,
CrawlerModel crawlerModel) |
Response |
OkHttpDownloader.process(Request request) |
| 限定符和类型 | 方法和说明 |
|---|---|
static Request |
Request.build(String url,
String callBack) |
static Request |
Request.build(String url,
String callBack,
HttpMethod httpMethod,
Map<String,String> params,
Map<String,String> meta) |
static Request |
Request.build(String url,
String callBack,
HttpMethod httpMethod,
Map<String,String> params,
Map<String,String> meta,
int maxReqcount) |
static Request |
Request.build(String url,
String callBack,
int maxReqCount) |
Request |
Response.getRequest() |
Request |
Request.setCallBack(String callBack) |
Request |
Request.setCrawlerName(String crawlerName) |
Request |
Request.setCurrentReqCount(int currentReqCount) |
Request |
Request.setHeader(Map<String,String> header) |
Request |
Request.setHttpMethod(HttpMethod httpMethod) |
Request |
Request.setMaxReqCount(int maxReqCount) |
Request |
Request.setMeta(Map<String,String> meta) |
Request |
Request.setParams(Map<String,String> params) |
Request |
Request.setSeimiAgentContentType(SeimiAgentContentType seimiAgentContentType) |
Request |
Request.setSeimiAgentRenderTime(long seimiAgentRenderTime) |
Request |
Request.setSeimiAgentScript(String seimiAgentScript) |
Request |
Request.setSeimiAgentUseCookie(Boolean seimiAgentUseCookie) |
Request |
Request.setSkipDuplicateFilter(boolean skipDuplicateFilter) |
Request |
Request.setStop(boolean stop) |
Request |
Request.setUrl(String url) |
Request |
Request.useSeimiAgent() |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
Response.setRequest(Request request) |
| 限定符和类型 | 方法和说明 |
|---|---|
static String |
GenericUtils.signRequest(Request request) |
Copyright © 2017. All Rights Reserved.