cn.spark2fire.jscrapy.model

类 OOSpider<T>

java.lang.Object

cn.spark2fire.jscrapy.Spider

cn.spark2fire.jscrapy.model.OOSpider<T>

所有已实现的接口:

Task, Runnable

public class OOSpider<T>
extends Spider

The spider for page model extractor.
In webmagic, we call a POJO containing extract result as "page model".
You can customize a crawler by write a page model with annotations.
Such as:

  @TargetUrl("http://my.oschina.net/flashsword/blog/\\d+")
  public class OschinaBlog{

       @ExtractBy("//title")
      private String title;

       @ExtractBy(value = "div.BlogContent",type = ExtractBy.Type.Css)
      private String content;

       @ExtractBy(value = "//div[@class='BlogTags']/a/text()", multi = true)
      private List<String> tags;
 }
 

And start the spider by:

   OOSpider.create(Site.me().addStartUrl("http://my.oschina.net/flashsword/blog")
        ,new JsonFilePageModelPipeline(), OschinaBlog.class).run();
 }
 

从以下版本开始:

0.2.0

作者:

code4crafter@gmail.com

嵌套类概要

从类继承的嵌套类/接口 cn.spark2fire.jscrapy.Spider

Spider.Status

字段概要

字段

限定符和类型	字段和说明
private ModelPageProcessor	modelPageProcessor
private ModelPipeline	modelPipeline
private List<Class>	pageModelClasses
private PageModelPipeline	pageModelPipeline

从类继承的字段 cn.spark2fire.jscrapy.Spider

destroyWhenExit, downloader, executorService, exitWhenComplete, logger, pageProcessor, pipelines, scheduler, site, spawnUrl, startRequests, stat, STAT_INIT, STAT_RUNNING, STAT_STOPPED, threadNum, threadPool, uuid

构造器概要

构造器

限定符	构造器和说明
protected	OOSpider(ModelPageProcessor modelPageProcessor)
	OOSpider(PageProcessor pageProcessor)
	OOSpider(Site site, PageModelPipeline pageModelPipeline, Class... pageModels) create a spider

方法概要

限定符和类型	方法和说明
OOSpider	addPageModel(PageModelPipeline pageModelPipeline, Class... pageModels)
static OOSpider	create(Site site, Class... pageModels)
static OOSpider	create(Site site, PageModelPipeline pageModelPipeline, Class... pageModels)
protected CollectorPipeline	getCollectorPipeline()
OOSpider	setIsExtractLinks(boolean isExtractLinks)

从类继承的方法 cn.spark2fire.jscrapy.Spider

addPipeline, addRequest, addUrl, checkIfRunning, clearPipeline, close, create, downloader, extractAndAddRequests, get, getAll, getPageCount, getScheduler, getSite, getSpiderListeners, getStartTime, getStatus, getThreadAlive, getUUID, initComponent, isExitWhenComplete, isSpawnUrl, onError, onSuccess, pipeline, run, runAsync, scheduler, setDownloader, setEmptySleepTime, setExecutorService, setExitWhenComplete, setPipelines, setScheduler, setSpawnUrl, setSpiderListeners, setUUID, sleep, start, startRequest, startUrls, stop, test, thread, thread

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

modelPageProcessor

private ModelPageProcessor modelPageProcessor

modelPipeline

private ModelPipeline modelPipeline

pageModelPipeline

private PageModelPipeline pageModelPipeline

pageModelClasses

private List<Class> pageModelClasses

构造器详细资料

OOSpider

protected OOSpider(ModelPageProcessor modelPageProcessor)

OOSpider

public OOSpider(PageProcessor pageProcessor)

OOSpider

public OOSpider(Site site,
                PageModelPipeline pageModelPipeline,
                Class... pageModels)

create a spider

参数:

site - site

pageModelPipeline - pageModelPipeline

pageModels - pageModels

方法详细资料

getCollectorPipeline

protected CollectorPipeline getCollectorPipeline()

覆盖:

getCollectorPipeline 在类中 Spider

create

public static OOSpider create(Site site,
                              Class... pageModels)

create

public static OOSpider create(Site site,
                              PageModelPipeline pageModelPipeline,
                              Class... pageModels)

addPageModel

public OOSpider addPageModel(PageModelPipeline pageModelPipeline,
                             Class... pageModels)

setIsExtractLinks

public OOSpider setIsExtractLinks(boolean isExtractLinks)