Uploaded image for project: 'Tajo (Retired)'
  1. Tajo (Retired)
  2. TAJO-54

SubQuery::allocateContainers() may ask 0 containers

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 0.2-incubating
    • None

    Description

      SubQuery::allocateContainers() calculates a number of containers to be requested for some subquery and then requests containers as follows:

      SubQuery.java
          public static void allocateContainers(SubQuery subQuery) {
            ExecutionBlock execBlock = subQuery.getBlock();
            QueryUnit [] tasks = subQuery.getQueryUnits();
      
            int numRequest = Math.min(tasks.length,
                subQuery.context.getNumClusterNode() * 4);
      

      In allocateContainers subQuery.context.getNumClusterNode() method internally invokes AMRMClient::getClusterNodeCount(). allocateContainers() requests 0 container to RM if AMRMClient::getClusterNodeCount() returns 0. If it does so, AppSchedulingInfo regards ApplicationMaster as deactive. As a result, ApplicationMaster cannot acquire any containers.

      In the current Hadoop Yarn, AMRMClient::getClusterNodeCount() temporarily returns 0 due to unknown reason even though there are available cluster nodes. This problem causes the integration test (i.e., 'mvn verify') to be hanging. This patch solves this problem by enabling RMContainerAllocator to wait for available cluster nodes.

      Attachments

        1. TAJO-54.patch
          5 kB
          Hyunsik Choi
        2. TAJO-54_2.patch
          5 kB
          Hyunsik Choi

        Activity

          People

            hyunsik Hyunsik Choi
            hyunsik Hyunsik Choi
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: