public class ComputeStreamingResponse
extends java.lang.Object
NOTE:
- NOT using Elasticsearch in practice - proved to be some speed issues with ES and Spark that appear to be ES-Spark specific - leave code in anticipation that the ES-Spark issues resolve...
- Even if rdd.count() calls are embedded in logger.debug statements, they are computed by Spark. Thus, they are commented out in the code below - uncomment for rdd.count() debug
| Constructor and Description |
|---|
ComputeStreamingResponse(org.apache.hadoop.fs.FileSystem fileSys) |
| Modifier and Type | Method and Description |
|---|---|
void |
performQuery()
Method to read in data from an allowed input source/format and perform the query
|
void |
performQuery(org.apache.spark.streaming.api.java.JavaDStream<org.apache.hadoop.io.MapWritable> input)
Method to perform the query given an input JavaDStream of JSON
|
org.apache.spark.streaming.api.java.JavaDStream<org.apache.hadoop.io.MapWritable> |
readData()
Method to read in the data from an allowed input format, filter, and return a RDD of MapWritable data elements
|
org.apache.spark.streaming.api.java.JavaDStream<org.apache.hadoop.io.MapWritable> |
readDataES()
Method to read in the data from elasticsearch, filter, and return a RDD of MapWritable data elements
|
void |
start()
Method to start the computation
|
void |
teardown()
Method to tear down necessary elements when app is complete
|
public ComputeStreamingResponse(org.apache.hadoop.fs.FileSystem fileSys)
throws PIRException
PIRExceptionpublic void start()
public void teardown()
public void performQuery()
throws java.io.IOException,
PIRException
java.io.IOExceptionPIRExceptionpublic org.apache.spark.streaming.api.java.JavaDStream<org.apache.hadoop.io.MapWritable> readData()
throws java.io.IOException,
PIRException
java.io.IOExceptionPIRExceptionpublic org.apache.spark.streaming.api.java.JavaDStream<org.apache.hadoop.io.MapWritable> readDataES()
throws java.io.IOException
java.io.IOExceptionpublic void performQuery(org.apache.spark.streaming.api.java.JavaDStream<org.apache.hadoop.io.MapWritable> input)