Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
3.2.1, 3.3.0
-
None
-
None
Description
Hi All,
Was trying to run spark application on the cluster mode using log4j2 and Spark 3.3.0.
When I run the below spark-submit command, only one worker (out of 3) starts executing the job.
// code placeholder spark-submit --master spark://spark-master-svc:7077 \ --conf spark.cores.max=4 \ --conf spark.sql.broadcastTimeout=3600 \ --conf spark.executor.cores=1 \ --jars /opt/spark/work-dir/<main_jar_name>.jar \ --deploy-mode cluster \ --class <class_name> \ --properties-file /opt/spark/conf/spark-defaults.conf \ --conf spark.driver.extraJavaOptions="-Dcom.amazonaws.sdk.disableCertChecking=true -Dlog4j.configurationFile=file:/opt/spark/work-dir/<log4j2>.properties" \ --conf spark.executor.extraJavaOptions="-Dcom.amazonaws.sdk.disableCertChecking=true -Dlog4j.configurationFile=file:/opt/spark/work-dir/<log4j2>.properties" \ --files "/opt/spark/work-dir/<log4j2>.properties" \ /opt/spark/work-dir/<jar_name>.jar /opt/spark/work-dir/application.properties >> /var/log/containers/hourly.log 2>&1
It means, on only one worker, I can see the driver logs and the other workers are idle and there are no app or executors logs created on other workers.
Below is the log4j2.properties file being used.
// code placeholder rootLogger.level = INFO rootLogger.appenderRef.rolling.ref = loggerId appender.rolling.type = RollingFile appender.rolling.name = loggerId appender.rolling.fileName=/var/log/containers/hourly.log appender.rolling.filePattern=hourly-.%d{yyyyMMdd}.log.gz appender.rolling.layout.type = PatternLayout appender.rolling.layout.pattern=%d [%t] %-5p (%F:%L) - %m%n appender.rolling.policies.type = Policies appender.rolling.policies.size.type = TimeBasedTriggeringPolicy appender.rolling.strategy.type = DefaultRolloverStrategy appender.rolling.strategy.max = 5 logger.spark.name = org.apache.spark logger.spark.level = WARN logger.spark.additivity = false logger.spark.repl.SparkIMain$exprTyper.level = INFO logger.spark.repl.SparkILoop$SparkILoopInterpreter.level = INFO # Settings to quiet third party logs that are too verbose logger.jetty.name = org.eclipse.jetty logger.jetty.level = WARN logger.jetty.util.component.AbstractLifeCycle.level = ERROR logger.parquet.name = org.apache.parquet logger.parquet.level = ERROR logger.kafka.name = org.apache.kafka logger.kafka.level = WARN logger.kafka.clients.consumer.internals.Fetcher.level=WARN
All log4j2 jars are included in the Spark home classpath under the jars directory.
- log4j-1.2-api-2.17.2.jar
- log4j-api-2.17.2.jar
- log4j-api-scala_2.12-12.0.jar
- log4j-core-2.17.2.jar
- log4j-slf4j-impl-2.17.2.jar
Can you please check and let me know whether I need to add or update anything to start the job in cluster mode supporting log4j2
Note : Things work fine with log4j1