org.apache.flink.table.runtime.operators.aggregate

类 BytesHashMap

java.lang.Object

org.apache.flink.table.runtime.operators.aggregate.BytesHashMap

public class BytesHashMap
extends Object

Bytes based hash map. It can be used for performing aggregations where the aggregated values are fixed-width, because the data is stored in continuous memory, AggBuffer of variable length cannot be applied to this HashMap. The KeyValue form in hash map is designed to reduce the cost of key fetching in lookup. The memory is divided into two areas:

- Bucket area: this contains: pointer + hashcode. Bytes 0 to 8: a pointer to the record in the record area Bytes 8 to 16: a holds key's full 32-bit hashcode

- Record area: this contains the actual data in linked list records. A BytesHashMap's record has four parts: Bytes 0 to 4: len(k) Bytes 4 to 4 + len(k): key data Bytes 4 + len(k) to 8 + len(k): len(v) Bytes 8 + len(k) to 8 + len(k) + len(v): value data

BytesHashMap are influenced by Apache Spark BytesToBytesMap.

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	BytesHashMap.Entry BytesHashMap Entry contains key and value field.
static class	BytesHashMap.LookupInfo Handle returned by lookup(BinaryRowData) function.

字段概要

字段

限定符和类型	字段和说明
static int	BUCKET_SIZE
static int	RECORD_EXTRA_LENGTH

构造器概要

构造器

构造器和说明
BytesHashMap(Object owner, org.apache.flink.runtime.memory.MemoryManager memoryManager, long memorySize, org.apache.flink.table.types.logical.LogicalType[] keyTypes, org.apache.flink.table.types.logical.LogicalType[] valueTypes)
BytesHashMap(Object owner, org.apache.flink.runtime.memory.MemoryManager memoryManager, long memorySize, org.apache.flink.table.types.logical.LogicalType[] keyTypes, org.apache.flink.table.types.logical.LogicalType[] valueTypes, boolean inferBucketMemory)

方法概要

限定符和类型	方法和说明
org.apache.flink.table.data.binary.BinaryRowData	append(BytesHashMap.LookupInfo info, org.apache.flink.table.data.binary.BinaryRowData value) Append an value into the hash map's record area.
void	free() release the map's record and bucket area's memory segments.
void	free(boolean reservedRecordMemory)
List<org.apache.flink.core.memory.MemorySegment>	getBucketAreaMemorySegments()
org.apache.flink.util.MutableObjectIterator<BytesHashMap.Entry>	getEntryIterator() Returns a destructive iterator for iterating over the entries of this map.
long	getNumElements()
long	getNumSpillFiles()
ArrayList<org.apache.flink.core.memory.MemorySegment>	getRecordAreaMemorySegments()
long	getSpillInBytes()
long	getUsedMemoryInBytes()
BytesHashMap.LookupInfo	lookup(org.apache.flink.table.data.binary.BinaryRowData key)
void	reset() reset the map's record and bucket area's memory segments for reusing.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

BUCKET_SIZE

public static final int BUCKET_SIZE

另请参阅:

常量字段值

RECORD_EXTRA_LENGTH

public static final int RECORD_EXTRA_LENGTH

另请参阅:

常量字段值

构造器详细资料

BytesHashMap

public BytesHashMap(Object owner,
                    org.apache.flink.runtime.memory.MemoryManager memoryManager,
                    long memorySize,
                    org.apache.flink.table.types.logical.LogicalType[] keyTypes,
                    org.apache.flink.table.types.logical.LogicalType[] valueTypes)

BytesHashMap

public BytesHashMap(Object owner,
                    org.apache.flink.runtime.memory.MemoryManager memoryManager,
                    long memorySize,
                    org.apache.flink.table.types.logical.LogicalType[] keyTypes,
                    org.apache.flink.table.types.logical.LogicalType[] valueTypes,
                    boolean inferBucketMemory)

方法详细资料

lookup

public BytesHashMap.LookupInfo lookup(org.apache.flink.table.data.binary.BinaryRowData key)

参数:

key - by which looking up the value in the hash map. Only support the key in the BinaryRowData form who has only one MemorySegment.

返回:

BytesHashMap.LookupInfo

append

public org.apache.flink.table.data.binary.BinaryRowData append(BytesHashMap.LookupInfo info,
                                                               org.apache.flink.table.data.binary.BinaryRowData value)
                                                        throws IOException

Append an value into the hash map's record area.

返回:

An BinaryRowData mapping to the memory segments in the map's record area belonging to the newly appended value.

抛出:

EOFException - if the map can't allocate much more memory.

IOException

getNumSpillFiles

public long getNumSpillFiles()

getUsedMemoryInBytes

public long getUsedMemoryInBytes()

getSpillInBytes

public long getSpillInBytes()

getNumElements

public long getNumElements()

getEntryIterator

public org.apache.flink.util.MutableObjectIterator<BytesHashMap.Entry> getEntryIterator()

Returns a destructive iterator for iterating over the entries of this map. It frees each page as it moves onto next one. Notice: it is illegal to call any method on the map after `destructiveIterator()` has been called.

返回:

an entry iterator for iterating the value appended in the hash map.

getRecordAreaMemorySegments

public ArrayList<org.apache.flink.core.memory.MemorySegment> getRecordAreaMemorySegments()

返回:

the underlying memory segments of the hash map's record area

getBucketAreaMemorySegments

public List<org.apache.flink.core.memory.MemorySegment> getBucketAreaMemorySegments()

free

public void free()

release the map's record and bucket area's memory segments.

free

public void free(boolean reservedRecordMemory)

参数:

reservedRecordMemory - reserved fixed memory or not.

reset

public void reset()

reset the map's record and bucket area's memory segments for reusing.