Class ResourceSitePass

  • All Implemented Interfaces:
    Function<IRDAG,​IRDAG>

    public final class ResourceSitePass
    extends AnnotatingPass
    Computes and assigns appropriate share of nodes to each irVertex to minimize shuffle time, with respect to bandwidth restrictions of nodes. If bandwidth information is not given, this pass does nothing. This pass optimizes task assignment considering nonuniform network bandwidths between resources. Ref. http://pages.cs.wisc.edu/~akella/papers/gda-sigcomm15.pdf

    Assumptions

    This pass assumes no skew in input or intermediate data, so that the number of Task assigned to a node is proportional to the data size handled by the node. Also, this pass assumes stages with empty map as ResourceSiteProperty are assigned to nodes evenly. For example, if source splits are not distributed evenly, any source location-aware scheduling policy will assign TaskGroups unevenly. Also, this pass assumes network bandwidth to be the bottleneck. Each node should have enough capacity to run TaskGroups immediately as scheduler attempts to schedule a TaskGroup.
    • Constructor Detail

      • ResourceSitePass

        public ResourceSitePass()
        Default constructor.
    • Method Detail

      • setBandwidthSpecificationString

        public static void setBandwidthSpecificationString​(String value)
        Parameters:
        value - bandwidth information in serialized JSON string.