Livy Supports Different Versions of Spark

Currently, Livy is built against specific version of Spark and can be run with different Spark 1.X versions without recompilation. But with the compatibility broken version of Spark 2 released, Livy���s code base should be changed to be compiled with Spark 2, which means Livy should maintain 2 different code bases to support different versions of Spark. So according to this, Livy should be compatible with different versions of Spark:

1. Make Livy be compatible with different Spark versions, like Spark 1.6, Spark 2.0, etc.
2. Make Livy support for different versions of Scala, Scala 2.10, 2.11, etc.
3. Different Livy sessions can run against different Spark versions, which means Livy could launch different version of Spark applications simultaneously.

Based on the requirements we proposed above, several Livy parts should be changed:

1. Change the pom file to support different Spark profiles, like -Pspark1.6, -Pspark2.0, etc.
2. Change the livy scala API pom to support to cross-compile with different Scala versions, this will make Livy code work against Spark with different Scala versions.
3. Support SparkSession in Batch API.
4. Change the repl related codes to use reflection to make it version independent.
5. Make SPARK_HOME configuration session independent, so that different session could launch different versions of Spark.