org.apache.hudi.examples.common

Class RandomJsonSource

java.lang.Object

org.apache.hudi.utilities.sources.Source<org.apache.spark.api.java.JavaRDD<String>>

org.apache.hudi.utilities.sources.JsonSource

org.apache.hudi.examples.common.RandomJsonSource

All Implemented Interfaces:

Serializable, SourceCommitCallback

public class RandomJsonSource
extends JsonSource

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.utilities.sources.Source

Source.SourceType

Field Summary

Fields inherited from class org.apache.hudi.utilities.sources.Source

persistRdd, props, sourceProfileSupplier, sparkContext, sparkSession, writeTableVersion

Constructor Summary

Constructors

Constructor and Description
RandomJsonSource(org.apache.hudi.common.config.TypedProperties props, org.apache.spark.api.java.JavaSparkContext sparkContext, org.apache.spark.sql.SparkSession sparkSession, SchemaProvider schemaProvider)

Method Summary

Modifier and Type	Method and Description
protected InputBatch<org.apache.spark.api.java.JavaRDD<String>>	readFromCheckpoint(org.apache.hudi.common.util.Option<org.apache.hudi.common.table.checkpoint.Checkpoint> lastCkptStr, long sourceLimit)

Methods inherited from class org.apache.hudi.utilities.sources.JsonSource

fetchNewData

Methods inherited from class org.apache.hudi.utilities.sources.Source

allowSourcePersist, assertCheckpointVersion, fetchNext, getSourceType, getSparkSession, releaseResources, translateCheckpoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.utilities.callback.SourceCommitCallback

onCommit

Constructor Detail

RandomJsonSource

public RandomJsonSource(org.apache.hudi.common.config.TypedProperties props,
                        org.apache.spark.api.java.JavaSparkContext sparkContext,
                        org.apache.spark.sql.SparkSession sparkSession,
                        SchemaProvider schemaProvider)

Method Detail

readFromCheckpoint

protected InputBatch<org.apache.spark.api.java.JavaRDD<String>> readFromCheckpoint(org.apache.hudi.common.util.Option<org.apache.hudi.common.table.checkpoint.Checkpoint> lastCkptStr,
                                                                                   long sourceLimit)

Overrides:

readFromCheckpoint in class Source<org.apache.spark.api.java.JavaRDD<String>>