Major Description
Problem Description :-
Our requirement is that: in a single Livy Interactive Session, User wants to execute "x" number of SQL Files.User wants to explicitly pass these SQL files to the interactive session and perform execution. These SQL Files consists of contain multiple DDL, DML statements.
Below is how the User wants to run multiple SQLs using an interactive session:-
1) Open a Interactive Session with a predefined session name.
/usr/bin/curl --silent --negotiate -u:$USER --max-time 30 ${LIVY_SERVER}${uri} -X POST -H "X-Requested-By: $USER" -H 'Content-Type: application/json' -d "${json_data}"
2) Using the session name opened in the above step, User wants to perform execution of multiple SQL Files.
/usr/bin/curl -v --negotiate -u:$USER --max-time 30 "${LIVY_SERVER}${uri}/upload-statements?outputPath=$output_file" -H "X-Requested-By: $USER" -X POST -F file=@${code_file} -H 'Content-Type: multipart/form-data'
In the above curl command, code_file is the SQL File which the User has passed for execution.
Suggested Solution :-
The following code would support the new endpoint (upload-statements) for Livy Rest API:-
Below code needs to be added in "server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala"
post("/:id/upload-statements") {
withModifyAccessSession { session =>
fileParams.get("file") match {
case Some(file) =>
val fileContent = new String(file.get) val outputPath = params.get("outputPath")
val req = new ExecuteRequest(fileContent, None, outputPath)
val statement = session.executeStatement(req)
Created(statement,
headers = Map(
"Location" -> url(getStatement,
"id" -> session.id.toString,
"statementId" -> statement.id.toString)))
case None =>
log("Could not get the content of the file")
}