Major Description
The following issue occurs:
ERROR org.apache.livy.utils.SparkYarnApp: Error whiling refreshing YARN state java.lang.IllegalStateException: No YARN application is found with tag livy-batch-2-per2hlwa in 120 seconds. This may be because 1) spark-submit fail to submit application to YARN; or 2) YARN cluster doesn't have enough resources to start the application in time. Please check Livy log and YARN log to know the details.
if the below configuration parameter is set in YARN:
yarn.resourcemanager.application-tag-based-placement.force-lowercase=false
We can fix this in InteractiveSession by updating app tag generation logic to generate appTag always in lowercase characters. InteractiveSession code can be changed
from
val appTag = s"livy-session-$id-${Random.alphanumeric.take(8).mkString}"
to
val appTag = s"livy-session-$id-${Random.alphanumeric.take(8).mkString}".toLowerCase()
