Class PartitionSetProperty

  • All Implemented Interfaces:
    java.io.Serializable

    public final class PartitionSetProperty
    extends EdgeExecutionProperty<java.util.ArrayList<KeyRange>>
    This property decides which partitions the tasks of the destination IRVertex should fetch. The position of a KeyRange in the list corresponds to the offset of the destination task.

    For example, in the following setup: Source IRVertex (Parallelism=2) - IREdge (Partitioner.Num=4) - Destination IRVertex (Parallelism=2)

    Setting PartitionSetProperty([0, 3), [3, 3)) on the IREdge with will enforce the following behaviors. - The first destination task fetches the first 3 partitions from each of the 2 data blocks - The second destination task fetches the last partitions from each of the 2 data blocks

    This property is useful for handling data skews. For example, if the size ratio of the 4 partitions in the above setup are (17%, 16%, 17%, 50%), then each of the destination task will evenly handle 50% of the load.

    See Also:
    Serialized Form
    • Method Detail

      • of

        public static PartitionSetProperty of​(java.util.ArrayList<KeyRange> value)
        Static method exposing the constructor.
        Parameters:
        value - value of the new execution property.
        Returns:
        the newly created execution property.