Class SerializedPartition<K>

  • Type Parameters:
    K - the key type of its partitions.
    All Implemented Interfaces:
    Partition<byte[],​K>

    public final class SerializedPartition<K>
    extends java.lang.Object
    implements Partition<byte[],​K>
    A collection of data elements. The data is stored as an array of bytes. This is a unit of read / write towards Blocks. Releasing the memory(either off-heap or on-heap) occurs on block deletion. TODO #396: Refactoring SerializedPartition into multiple classes
    • Constructor Detail

      • SerializedPartition

        public SerializedPartition​(K key,
                                   Serializer serializer,
                                   MemoryPoolAssigner memoryPoolAssigner)
                            throws java.io.IOException,
                                   MemoryAllocationException
        Creates a serialized Partition without actual data. Data can be written to this partition until it is committed.
        Parameters:
        key - the key of this partition.
        serializer - the serializer to be used to serialize data.
        memoryPoolAssigner - the memory pool assigner for memory allocation.
        Throws:
        java.io.IOException - if fail to chain the output stream.
        MemoryAllocationException - if fail to allocate memory.
      • SerializedPartition

        public SerializedPartition​(K key,
                                   byte[] serializedData,
                                   int length,
                                   MemoryPoolAssigner memoryPoolAssigner)
        Creates a serialized Partition with actual data residing in on-heap region. Data cannot be written to this partition after the construction.
        Parameters:
        key - the key.
        serializedData - the serialized data.
        length - the length of the actual serialized data. (It can be different with serializedData.length)
        memoryPoolAssigner - the memory pool assigner.
      • SerializedPartition

        public SerializedPartition​(K key,
                                   java.util.List<MemoryChunk> serializedChunkList,
                                   int length,
                                   MemoryPoolAssigner memoryPoolAssigner)
        Creates a serialized Partition with actual data residing in off-heap region. Data cannot be written to this partition after the construction.
        Parameters:
        key - the key.
        serializedChunkList - the serialized data in list list of MemoryChunks.
        length - the length of the actual serialized data.(It can be different with serializedData.length)
        memoryPoolAssigner - the memory pool assigner.
    • Method Detail

      • write

        public void write​(java.lang.Object element)
                   throws java.io.IOException
        Writes an element to non-committed partition.
        Specified by:
        write in interface Partition<byte[],​K>
        Parameters:
        element - element to write.
        Throws:
        java.io.IOException - if the partition is already committed.
      • commit

        public void commit()
                    throws java.io.IOException
        Commits a partition to prevent further data write.
        Specified by:
        commit in interface Partition<byte[],​K>
        Throws:
        java.io.IOException - if fail to commit partition.
      • getKey

        public K getKey()
        Specified by:
        getKey in interface Partition<byte[],​K>
        Returns:
        the key value.
      • isSerialized

        public boolean isSerialized()
        Specified by:
        isSerialized in interface Partition<byte[],​K>
        Returns:
        whether the data in this Partition is serialized or not.
      • getData

        public byte[] getData()
                       throws java.io.IOException
        This method should only be used when this partition is residing in on-heap region.
        Specified by:
        getData in interface Partition<byte[],​K>
        Returns:
        the serialized data.
        Throws:
        java.io.IOException - if the partition is not committed yet.
      • getDirectBufferList

        public java.util.List<java.nio.ByteBuffer> getDirectBufferList()
                                                                throws java.io.IOException
        This method is used to emit the output as SerializedPartition.
        Returns:
        the serialized data in list of ByteBuffers
        Throws:
        java.io.IOException - if the partition is not committed yet.
      • getLength

        public int getLength()
                      throws java.io.IOException
        Returns:
        the length of the actual data.
        Throws:
        java.io.IOException - if the partition is not committed yet.
      • isOffheap

        public boolean isOffheap()
        Returns:
        whether this SerializedPartition is residing in off-heap region.
      • release

        public void release()
        Releases the off-heap memory that this SerializedPartition holds. TODO #403: Remove 'transient' uses of SerializedPartition.