Package org.apache.nemo.runtime.executor.data.block

Interface Block<K extends java.io.Serializable>

  • Type Parameters:
    K - the key type of its partitions.
    All Known Implementing Classes:
    FileBlock, NonSerializedMemoryBlock, SerializedMemoryBlock
    public interface Block<K extends java.io.Serializable>
    This interface represents a block, which is the output of a specific task.

    • Method Detail

      • write

        void write​(K key,
           java.lang.Object element)
        Writes an element to non-committed block. Invariant: This should not be invoked after this block is committed. Invariant: This method does not support concurrent write.
        Parameters:
        key - the key.
        element - the element to write. Classes implementing this interface may throw org.apache.nemo.common.exception.BlockWriteException for any error occurred while trying to write a block. (This exception will be thrown to the scheduler through Executor and have to be handled by the scheduler with fault tolerance mechanism.)

      • writePartitions

        void writePartitions​(java.lang.Iterable<NonSerializedPartition<K>> partitions)
        Stores NonSerializedPartitions to this block. Invariant: This should not be invoked after this block is committed. Invariant: This method does not support concurrent write.
        Parameters:
        partitions - the NonSerializedPartitions to store. Classes implementing this interface may throw org.apache.nemo.common.exception.BlockWriteException for any error occurred while trying to write a block. (This exception will be thrown to the scheduler through Executor and have to be handled by the scheduler with fault tolerance mechanism.)

      • writeSerializedPartitions

        void writeSerializedPartitions​(java.lang.Iterable<SerializedPartition<K>> partitions)
        Stores SerializedPartitions to this block. Invariant: This should not be invoked after this block is committed. Invariant: This method does not support concurrent write.
        Parameters:
        partitions - the SerializedPartitions to store. Classes implementing this interface may throw org.apache.nemo.common.exception.BlockWriteException for any error occurred while trying to write a block. (This exception will be thrown to the scheduler through Executor and have to be handled by the scheduler with fault tolerance mechanism.)

      • readPartitions

        java.lang.Iterable<NonSerializedPartition<K>> readPartitions​(KeyRange<K> keyRange)
        Retrieves the NonSerializedPartitions in a specific key range from this block. If the data is serialized, deserializes it. Invariant: This should not be invoked before this block is committed.
        Parameters:
        keyRange - the key range to retrieve.
        Returns:
        an iterable of NonSerializedPartitions. Classes implementing this interface may throw org.apache.nemo.common.exception.BlockFetchException for any error occurred while trying to fetch a block. (This exception will be thrown to the scheduler through Executor and have to be handled by the scheduler with fault tolerance mechanism.)

      • readSerializedPartitions

        java.lang.Iterable<SerializedPartition<K>> readSerializedPartitions​(KeyRange<K> keyRange)
        Retrieves the SerializedPartitions in a specific key range. Invariant: This should not be invoked before this block is committed.
        Parameters:
        keyRange - the hash range to retrieve.
        Returns:
        an iterable of SerializedPartitions. Classes implementing this interface may throw org.apache.nemo.common.exception.BlockFetchException for any error occurred while trying to fetch a block. (This exception will be thrown to the scheduler through Executor and have to be handled by the scheduler with fault tolerance mechanism.)

      • commit

        java.util.Optional<java.util.Map<K,​java.lang.Long>> commit()
        Commits this block to prevent further write.
        Returns:
        the size of each partition if the data in the block is serialized. Classes implementing this interface may throw org.apache.nemo.common.exception.BlockWriteException for any error occurred while trying to commit a block. (This exception will be thrown to the scheduler through Executor and have to be handled by the scheduler with fault tolerance mechanism.)

      • commitPartitions

        void commitPartitions()
        Commits all un-committed partitions. This method can be useful if partitions in a block should be committed before the block is committed totally. For example, non-committed partitions in a file block can be flushed to storage from memory. If another element is written after this method is called, a new non-committed partition should be created for the element even if a partition with the same key is committed already. Classes implementing this interface may throw org.apache.nemo.common.exception.BlockWriteException for any error occurred while trying to commit partitions. (This exception will be thrown to the scheduler through Executor and have to be handled by the scheduler with fault tolerance mechanism.)

      • getId

        java.lang.String getId()
        Returns:
        the ID of this block.

      • isCommitted

        boolean isCommitted()
        Returns:
        whether this block is committed or not.