FlinkStreamerConfig (hudi-flink 0.12.1-rc2 API)

java.lang.Object
- org.apache.flink.api.common.ExecutionConfig.GlobalJobParameters
- - org.apache.flink.configuration.Configuration
  - - org.apache.hudi.streamer.FlinkStreamerConfig

All Implemented Interfaces:

Serializable, Cloneable, org.apache.flink.configuration.ReadableConfig, org.apache.flink.configuration.WritableConfig, org.apache.flink.core.io.IOReadableWritable
```
public class FlinkStreamerConfig
extends org.apache.flink.configuration.Configuration
```
Configurations for Hoodie Flink streamer.

See Also:

Serialized Form

Field Summary

Fields
Modifier and Type	Field and Description
`Integer`	`archiveMaxCommits`
`Integer`	`archiveMinCommits`
`Integer`	`bucketAssignNum`
`Long`	`checkpointInterval` Flink checkpoint interval.
`Boolean`	`cleanAsyncEnabled`
`String`	`cleanPolicy`
`Integer`	`cleanRetainCommits`
`Integer`	`cleanRetainFileVersions`
`Boolean`	`commitOnErrors`
`Boolean`	`compactionAsyncEnabled`
`Integer`	`compactionDeltaCommits`
`Integer`	`compactionDeltaSeconds`
`Integer`	`compactionMaxMemory`
`Long`	`compactionTargetIo`
`Integer`	`compactionTasks`
`String`	`compactionTriggerStrategy`
`List<String>`	`configs`
`String`	`flinkCheckPointPath`
`Boolean`	`help`
`Boolean`	`hiveStylePartitioning`
`Boolean`	`hiveSyncAssumeDatePartition`
`Boolean`	`hiveSyncAutoCreateDb`
`String`	`hiveSyncDb`
`Boolean`	`hiveSyncEnabled`
`String`	`hiveSyncFileFormat`
`Boolean`	`hiveSyncIgnoreExceptions`
`String`	`hiveSyncJdbcUrl`
`String`	`hiveSyncMetastoreUri`
`String`	`hiveSyncMode`
`String`	`hiveSyncPartitionExtractorClass`
`String`	`hiveSyncPartitionFields`
`String`	`hiveSyncPassword`
`Boolean`	`hiveSyncSkipRoSuffix`
`Boolean`	`hiveSyncSupportTimestamp`
`String`	`hiveSyncTable`
`Boolean`	`hiveSyncUseJdbc`
`String`	`hiveSyncUsername`
`Boolean`	`indexBootstrapEnabled`
`Integer`	`indexBootstrapNum`
`Boolean`	`indexGlobalEnabled`
`String`	`indexPartitionRegex`
`Double`	`indexStateTtl`
`Boolean`	`insertCluster`
`String`	`instantRetryInterval`
`String`	`instantRetryTimes`
`String`	`kafkaBootstrapServers`
`String`	`kafkaGroupId`
`String`	`kafkaTopic`
`String`	`keygenClass`
`String`	`keygenType`
`Integer`	`metadataCompactionDeltaCommits`
`Boolean`	`metadataEnabled`
`WriteOperationType`	`operation`
`Integer`	`parquetPageSize`
`String`	`partitionDefaultName`
`String`	`partitionPathField`
`String`	`payloadClassName`
`Boolean`	`preCombine`
`String`	`propsFilePath`
`String`	`recordKeyField`
`String`	`sourceAvroSchema`
`String`	`sourceAvroSchemaPath`
`String`	`sourceOrderingField`
`org.apache.flink.runtime.state.StateBackend`	`stateBackend`
`String`	`tableType`
`String`	`targetBasePath`
`String`	`targetTableName`
`List<String>`	`transformerClassNames`
`Boolean`	`utcTimezone`
`Double`	`writeBatchSize`
`Integer`	`writeLogBlockSize`
`Integer`	`writeLogMaxSize`
`Integer`	`writeMergeMaxMemory`
`Integer`	`writeParquetBlockSize`
`Integer`	`writeParquetMaxFileSize`
`String`	`writePartitionFormat`
`Boolean`	`writePartitionUrlEncode`
`Long`	`writeRateLimit`
`Double`	`writeTaskMaxSize`
`Integer`	`writeTaskNum`

Fields inherited from class org.apache.flink.configuration.Configuration
confData

Constructor Summary

Constructors
Constructor and Description

FlinkStreamerConfig()

Constructors
Constructor and Description
`FlinkStreamerConfig()`

Method Summary

All Methods Static Methods Concrete Methods
Modifier and Type	Method and Description
`static org.apache.flink.configuration.Configuration`	`toFlinkConfig(FlinkStreamerConfig config)` Transforms a `HoodieFlinkStreamer.Config` into `Configuration`.

Methods inherited from class org.apache.flink.configuration.Configuration
addAll, addAll, addAllToProperties, clone, contains, containsKey, equals, fromMap, get, getBoolean, getBoolean, getBoolean, getBytes, getClass, getDouble, getDouble, getDouble, getEnum, getFloat, getFloat, getFloat, getInteger, getInteger, getInteger, getLong, getLong, getLong, getOptional, getString, getString, getString, getValue, hashCode, keySet, read, removeConfig, set, setBoolean, setBoolean, setBytes, setClass, setDouble, setDouble, setFloat, setFloat, setInteger, setInteger, setLong, setLong, setString, setString, toMap, toString, write

Methods inherited from class java.lang.Object
finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

kafkaTopic
```
public String kafkaTopic
```

kafkaGroupId
```
public String kafkaGroupId
```

kafkaBootstrapServers
```
public String kafkaBootstrapServers
```

flinkCheckPointPath
```
public String flinkCheckPointPath
```

stateBackend

public org.apache.flink.runtime.state.StateBackend stateBackend

instantRetryTimes
```
public String instantRetryTimes
```

instantRetryInterval
```
public String instantRetryInterval
```

targetBasePath
```
public String targetBasePath
```

targetTableName
```
public String targetTableName
```

tableType
```
public String tableType
```

insertCluster
```
public Boolean insertCluster
```

propsFilePath
```
public String propsFilePath
```

configs
```
public List<String> configs
```

recordKeyField
```
public String recordKeyField
```

partitionPathField
```
public String partitionPathField
```

keygenClass
```
public String keygenClass
```

keygenType
```
public String keygenType
```

sourceOrderingField
```
public String sourceOrderingField
```

payloadClassName
```
public String payloadClassName
```

operation
```
public WriteOperationType operation
```

preCombine
```
public Boolean preCombine
```

commitOnErrors
```
public Boolean commitOnErrors
```

transformerClassNames

public List<String> transformerClassNames

metadataEnabled
```
public Boolean metadataEnabled
```

metadataCompactionDeltaCommits

public Integer metadataCompactionDeltaCommits

writePartitionFormat
```
public String writePartitionFormat
```

writeRateLimit
```
public Long writeRateLimit
```

writeParquetBlockSize
```
public Integer writeParquetBlockSize
```

writeParquetMaxFileSize
```
public Integer writeParquetMaxFileSize
```

parquetPageSize
```
public Integer parquetPageSize
```

checkpointInterval
```
public Long checkpointInterval
```
Flink checkpoint interval.

help
```
public Boolean help
```

indexBootstrapNum
```
public Integer indexBootstrapNum
```

bucketAssignNum
```
public Integer bucketAssignNum
```

writeTaskNum
```
public Integer writeTaskNum
```

partitionDefaultName
```
public String partitionDefaultName
```

indexBootstrapEnabled
```
public Boolean indexBootstrapEnabled
```

indexStateTtl
```
public Double indexStateTtl
```

indexGlobalEnabled
```
public Boolean indexGlobalEnabled
```

indexPartitionRegex
```
public String indexPartitionRegex
```

sourceAvroSchemaPath
```
public String sourceAvroSchemaPath
```

sourceAvroSchema
```
public String sourceAvroSchema
```

utcTimezone
```
public Boolean utcTimezone
```

writePartitionUrlEncode
```
public Boolean writePartitionUrlEncode
```

hiveStylePartitioning
```
public Boolean hiveStylePartitioning
```

writeTaskMaxSize
```
public Double writeTaskMaxSize
```

writeBatchSize
```
public Double writeBatchSize
```

writeLogBlockSize
```
public Integer writeLogBlockSize
```

writeLogMaxSize
```
public Integer writeLogMaxSize
```

writeMergeMaxMemory
```
public Integer writeMergeMaxMemory
```

compactionAsyncEnabled
```
public Boolean compactionAsyncEnabled
```

compactionTasks
```
public Integer compactionTasks
```

compactionTriggerStrategy

public String compactionTriggerStrategy

compactionDeltaCommits
```
public Integer compactionDeltaCommits
```

compactionDeltaSeconds
```
public Integer compactionDeltaSeconds
```

compactionMaxMemory
```
public Integer compactionMaxMemory
```

compactionTargetIo
```
public Long compactionTargetIo
```

cleanAsyncEnabled
```
public Boolean cleanAsyncEnabled
```

cleanPolicy
```
public String cleanPolicy
```

cleanRetainCommits
```
public Integer cleanRetainCommits
```

cleanRetainFileVersions
```
public Integer cleanRetainFileVersions
```

archiveMaxCommits
```
public Integer archiveMaxCommits
```

archiveMinCommits
```
public Integer archiveMinCommits
```

hiveSyncEnabled
```
public Boolean hiveSyncEnabled
```

hiveSyncDb
```
public String hiveSyncDb
```

hiveSyncTable
```
public String hiveSyncTable
```

hiveSyncFileFormat
```
public String hiveSyncFileFormat
```

hiveSyncMode
```
public String hiveSyncMode
```

hiveSyncUsername
```
public String hiveSyncUsername
```

hiveSyncPassword
```
public String hiveSyncPassword
```

hiveSyncJdbcUrl
```
public String hiveSyncJdbcUrl
```

hiveSyncMetastoreUri
```
public String hiveSyncMetastoreUri
```

hiveSyncPartitionFields
```
public String hiveSyncPartitionFields
```

hiveSyncPartitionExtractorClass

public String hiveSyncPartitionExtractorClass

hiveSyncAssumeDatePartition

public Boolean hiveSyncAssumeDatePartition

hiveSyncUseJdbc
```
public Boolean hiveSyncUseJdbc
```

hiveSyncAutoCreateDb
```
public Boolean hiveSyncAutoCreateDb
```

hiveSyncIgnoreExceptions

public Boolean hiveSyncIgnoreExceptions

hiveSyncSkipRoSuffix
```
public Boolean hiveSyncSkipRoSuffix
```

hiveSyncSupportTimestamp

public Boolean hiveSyncSupportTimestamp

Constructor Detail
- FlinkStreamerConfig
```
public FlinkStreamerConfig()
```

Method Detail
- toFlinkConfig
```
public static org.apache.flink.configuration.Configuration toFlinkConfig(FlinkStreamerConfig config)
```
  Transforms a HoodieFlinkStreamer.Config into Configuration. The latter is more suitable for the table APIs. It reads all the properties in the properties file (set by `--props` option) and cmd line options (set by `--hoodie-conf` option).

Class FlinkStreamerConfig

Field Summary

Fields inherited from class org.apache.flink.configuration.Configuration

Constructor Summary

Method Summary

Methods inherited from class org.apache.flink.configuration.Configuration

Methods inherited from class java.lang.Object

Field Detail

kafkaTopic

kafkaGroupId

kafkaBootstrapServers

flinkCheckPointPath

stateBackend

instantRetryTimes

instantRetryInterval

targetBasePath

targetTableName

tableType

insertCluster

propsFilePath

configs

recordKeyField

partitionPathField

keygenClass

keygenType

sourceOrderingField

payloadClassName

operation

preCombine

commitOnErrors

transformerClassNames

metadataEnabled

metadataCompactionDeltaCommits

writePartitionFormat

writeRateLimit

writeParquetBlockSize

writeParquetMaxFileSize

parquetPageSize

checkpointInterval

help

indexBootstrapNum

bucketAssignNum

writeTaskNum

partitionDefaultName

indexBootstrapEnabled

indexStateTtl

indexGlobalEnabled

indexPartitionRegex

sourceAvroSchemaPath

sourceAvroSchema

utcTimezone

writePartitionUrlEncode

hiveStylePartitioning

writeTaskMaxSize

writeBatchSize

writeLogBlockSize

writeLogMaxSize

writeMergeMaxMemory

compactionAsyncEnabled

compactionTasks

compactionTriggerStrategy

compactionDeltaCommits

compactionDeltaSeconds

compactionMaxMemory

compactionTargetIo

cleanAsyncEnabled

cleanPolicy

cleanRetainCommits

cleanRetainFileVersions

archiveMaxCommits

archiveMinCommits

hiveSyncEnabled

hiveSyncDb

hiveSyncTable

hiveSyncFileFormat

hiveSyncMode

hiveSyncUsername

hiveSyncPassword

hiveSyncJdbcUrl

hiveSyncMetastoreUri