Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
I am trying to use logback with spark-thrift-server to log the application logs in elasticsearch and for that i am using log4j-over-slf4j along with logback-classic and logback-core.
It seems Hive use WriterAppender.setWriter() of log4j for logging but the Log4j bridge (log4j-over-slf4j library JAR) does not implement this method.
Getting this error:-
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hive.service.cli.operation.LogDivertAppender.setWriter(Ljava/io/Writer;)V
at org.apache.hive.service.cli.operation.LogDivertAppender.<init>(LogDivertAppender.java:166)
at org.apache.hive.service.cli.operation.OperationManager.initOperationLogCapture(OperationManager.java:84)
at org.apache.hive.service.cli.operation.OperationManager.init(OperationManager.java:62)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:83)
at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.init(SparkSQLSessionManager.scala:44)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService.$anonfun$initCompositeService$1(SparkSQLCLIService.scala:123)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService.$anonfun$initCompositeService$1$adapted(SparkSQLCLIService.scala:123)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService.initCompositeService(SparkSQLCLIService.scala:123)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService.initCompositeService$(SparkSQLCLIService.scala:120)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIService.initCompositeService(SparkSQLCLIService.scala:41)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIService.init(SparkSQLCLIService.scala:93)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService.$anonfun$initCompositeService$1(SparkSQLCLIService.scala:123)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService.$anonfun$initCompositeService$1$adapted(SparkSQLCLIService.scala:123)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService.initCompositeService(SparkSQLCLIService.scala:123)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService.initCompositeService$(SparkSQLCLIService.scala:120)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.initCompositeService(HiveThriftServer2.scala:124)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.init(HiveThriftServer2.scala:144)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.startWithContext(HiveThriftServer2.scala:63)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:104)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
i have reported same this on stackoverflow, for more details please refer:-
https://stackoverflow.com/questions/76611022/spark-thrift-server-fails-to-start
My question is how to resolve this issue, i searched online and people suggesting to disable this property[{*}hive.server2.logging.operation.enabled{*}] but disabling this property won't work for me.
i need a solution so that i can use logback with hiveserver2/spark-thrift-server