public class NodeSorterHostProximity extends Object implements INodeSorter
Modifier and Type | Class and Description |
---|---|
static interface |
NodeSorterHostProximity.ExistingScheduleFunc
interface for calculating the number of existing executors scheduled on a object (rack or node).
|
Modifier and Type | Field and Description |
---|---|
protected Cluster |
cluster |
protected List<String> |
favoredNodeIds |
protected List<String> |
greyListedSupervisorIds |
protected BaseResourceAwareStrategy.NodeSortType |
nodeSortType |
protected TopologyDetails |
topologyDetails |
protected List<String> |
unFavoredNodeIds |
Constructor and Description |
---|
NodeSorterHostProximity(Cluster cluster,
TopologyDetails topologyDetails) |
NodeSorterHostProximity(Cluster cluster,
TopologyDetails topologyDetails,
BaseResourceAwareStrategy.NodeSortType nodeSortType)
Initialize for the default implementation node sorting.
|
Modifier and Type | Method and Description |
---|---|
Map<String,Set<String>> |
getRackIdToHosts() |
Map<String,AtomicInteger> |
getScheduledExecCntByRackId() |
Iterable<ObjectResourcesItem> |
getSortedRacks()
Racks are sorted by two criteria.
|
List<RasNode> |
hostnameToNodes(String hostname)
hostname to Ids.
|
protected List<String> |
makeHostToNodeIds(List<String> hosts) |
void |
prepare(ExecutorDetails exec)
Prepare for node sorting.
|
Iterable<String> |
sortAllNodes() |
protected Iterable<ObjectResourcesItem> |
sortObjectResources(ObjectResourcesSummary resourcesSummary,
ExecutorDetails exec,
NodeSorterHostProximity.ExistingScheduleFunc existingScheduleFunc)
Scheduling uses
sortAllNodes() which eventually calls this method whose behavior can be altered by setting nodeSortType . |
protected final BaseResourceAwareStrategy.NodeSortType nodeSortType
protected Cluster cluster
protected TopologyDetails topologyDetails
public NodeSorterHostProximity(Cluster cluster, TopologyDetails topologyDetails)
public NodeSorterHostProximity(Cluster cluster, TopologyDetails topologyDetails, BaseResourceAwareStrategy.NodeSortType nodeSortType)
Initialize for the default implementation node sorting.
BaseResourceAwareStrategy.NodeSortType#GENERIC_RAS
sorting implemented in
sortObjectResourcesGeneric(ObjectResourcesSummary, ExecutorDetails, NodeSorterHostProximity.ExistingScheduleFunc)
BaseResourceAwareStrategy.NodeSortType#DEFAULT_RAS
sorting implemented in
sortObjectResourcesDefault(ObjectResourcesSummary, NodeSorterHostProximity.ExistingScheduleFunc)
BaseResourceAwareStrategy.NodeSortType#COMMON
sorting implemented in
sortObjectResourcesCommon(ObjectResourcesSummary, ExecutorDetails, NodeSorterHostProximity.ExistingScheduleFunc)
cluster
- for which nodes will be sorted.topologyDetails
- the topology to sort for.nodeSortType
- type of sorting to be applied to object resource collection BaseResourceAwareStrategy.NodeSortType
.public void prepare(ExecutorDetails exec)
INodeSorter
Prepare for node sorting. This method must be called before INodeSorter.getSortedRacks()
and INodeSorter.sortAllNodes()
.
prepare
in interface INodeSorter
exec
- optional, may be null.protected Iterable<ObjectResourcesItem> sortObjectResources(ObjectResourcesSummary resourcesSummary, ExecutorDetails exec, NodeSorterHostProximity.ExistingScheduleFunc existingScheduleFunc)
Scheduling uses sortAllNodes()
which eventually calls this method whose behavior can be altered by setting nodeSortType
.
resourcesSummary
- contains all individual ObjectResourcesItem
as well as cumulative statsexec
- executor for which the sorting is doneexistingScheduleFunc
- a function to get existing executors already scheduled on this objectIterable
of sorted ObjectResourcesItem
public Iterable<String> sortAllNodes()
sortAllNodes
in interface INodeSorter
public Map<String,AtomicInteger> getScheduledExecCntByRackId()
public Iterable<ObjectResourcesItem> getSortedRacks()
Racks are sorted by two criteria.
1) the number executors of the topology that needs to be scheduled is already on the rack in descending order. The reasoning to sort based on criterion 1 is so we schedule the rest of a topology on the same rack as the existing executors of the topology.
2) the subordinate/subservient resource availability percentage of a rack in descending order We calculate the resource availability percentage by dividing the resource availability on the rack by the resource availability of the entire cluster By doing this calculation, racks that have exhausted or little of one of the resources mentioned above will be ranked after racks that have more balanced resource availability. So we will be less likely to pick a rack that have a lot of one resource but a low amount of another.
getSortedRacks
in interface INodeSorter
Copyright © 2022 The Apache Software Foundation. All rights reserved.