org.apache.hudi.io

Class IOUtils

java.lang.Object

org.apache.hudi.io.IOUtils

public class IOUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
IOUtils()

Method Summary

Modifier and Type	Method and Description
static long	getMaxMemoryAllowedForMerge(TaskContextSupplier context, String maxMemoryFraction) Dynamic calculation of max memory to use for spillable map.
static long	getMaxMemoryPerCompaction(TaskContextSupplier context, HoodieConfig hoodieConfig)
static long	getMaxMemoryPerPartitionMerge(TaskContextSupplier context, HoodieConfig hoodieConfig)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IOUtils

public IOUtils()

Method Detail

getMaxMemoryAllowedForMerge

public static long getMaxMemoryAllowedForMerge(TaskContextSupplier context,
                                               String maxMemoryFraction)

Dynamic calculation of max memory to use for spillable map. There is always more than one task running on an executor and each task maintains a spillable map. user.available.memory = executor.memory * (1 - memory.fraction) spillable.available.memory = user.available.memory * hoodie.memory.fraction / executor.cores. Anytime the engine memory fractions/total memory is changed, the memory used for spillable map changes accordingly.

getMaxMemoryPerPartitionMerge

public static long getMaxMemoryPerPartitionMerge(TaskContextSupplier context,
                                                 HoodieConfig hoodieConfig)

getMaxMemoryPerCompaction

public static long getMaxMemoryPerCompaction(TaskContextSupplier context,
                                             HoodieConfig hoodieConfig)