Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-866

Optimize Yarn GetApplications Query to prevent additional load on Yarn and Livy

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • 0.9.0
    • Server
    • None

    Description

      Currently Livy queries Yarn applications by applicationType : Spark. This will put heavy load on Yarn clusters if there are thousands or more Spark application in all states (running, finished, failed, queued etc.).

      A better approach would be to query the applications by tags in addition to job type since Livy only needs to track application with certain application tags. However, YarnClient does not expose any API to query applications by tags.

       

      We can extend the YarnClientImpl which would allow us to use protected rmClient in YarnClient and we can implement getApplications method with GetApplicationsRequest where we can include tags.

      Attachments

        Activity

          People

            Unassigned Unassigned
            akbordia Akshat Bordia
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m