Uploaded image for project: 'Apache YuniKorn'
  1. Apache YuniKorn
  2. YUNIKORN-1715

Yunikorn performance improvements

    XMLWordPrintableJSON

Details

    Description

      There are some methods/functions in Yunikorn which are called frequently and often unnecessarily. On a large, busy cluster, eliminating these calls can result in faster scheduling cycle, therefore better throughput.

      In the listed cases below, we can re-use a previously computed value and the expensive copy/sort phase can be eliminated completely.

      Retrieving node iterators: in baseNodeCollection.getNodeIteratorInternal(), we always clone the tree of sorted nodes, then we build a slice. The node tree is only modified when a node gets a new score (plus node add/removal). By reusing the sorted list, we avoid cloning an *btree.BTree structure and creating []*Node slices.

      Queue sorting: only need sorting if the following occurred:

      • Allocated resource changed in one of the child queues (most common)
      • Pending resource changed from 0 to "n", or from "n" to 0 (affects filtering)
      • Child queue got stopped (affects filtering)
      • Child queue structure changed on config update

      Application sorting: in Queue.TryAllocate() and Queue.TryPlaceholderAllocate(), sortApplications() always runs. In every iteration, it calls Queue.GetCopyOfApps() then proceeds to sort the apps. It only has to run if something relevant happens from the sorting POV:

      • Application added/removed
      • Ask added to an application
      • Ask max priority changed in at least one application
      • Allocated resource changed in at least one application

      Request sorting: request (ask) sorting is only necessary when the following occurs:

      • Ask added
      • pendingAskRepeat gets 0 in an ask

      Misc: we can have a bunch of other stuff that helps performance.

      Attachments

        There are no Sub-Tasks for this issue.

        Activity

          People

            pbacsko Peter Bacsko
            pbacsko Peter Bacsko
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: