org.apache.crunch.impl.spark
Class SparkPipeline

java.lang.Object
  org.apache.crunch.impl.dist.DistributedPipeline
      org.apache.crunch.impl.spark.SparkPipeline

All Implemented Interfaces:

Pipeline

public class SparkPipeline
extends DistributedPipeline

Constructor Summary

SparkPipeline(org.apache.spark.api.java.JavaSparkContext sparkContext, String appName)

SparkPipeline(String sparkConnect, String appName)

Method Summary

<T> void	cache(PCollection<T> pcollection, CachingOptions options)
PipelineResult	done()
<T> Iterable<T>	materialize(PCollection<T> pcollection)
PipelineResult	run()
PipelineExecution	runAsync()

Methods inherited from class org.apache.crunch.impl.dist.DistributedPipeline

cleanup, createIntermediateOutput, createTempPath, enableDebug, getConfiguration, getFactory, getMaterializeSourceTarget, getName, getNextAnonymousStageId, read, read, readTextFile, setConfiguration, write, write, writeTextFile

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkPipeline

public SparkPipeline(String sparkConnect,
                     String appName)

SparkPipeline

public SparkPipeline(org.apache.spark.api.java.JavaSparkContext sparkContext,
                     String appName)

Method Detail

materialize

public <T> Iterable<T> materialize(PCollection<T> pcollection)

cache

public <T> void cache(PCollection<T> pcollection,
                      CachingOptions options)

run

public PipelineResult run()

runAsync

public PipelineExecution runAsync()

done

public PipelineResult done()

Specified by:

done in interface Pipeline

Overrides:

done in class DistributedPipeline