While running Spark application with huge data or more number columns we can see java.lang.StackOverflowError. To solve this issue we need to increase the stack size.
First we need to identify where the StackOverflowError is occurred in driver side or executor side. If issue is occurred in driver side then we need to set the stack size using ‘spark.driver.extraJavaOptions’. If issue is occurred in executor side then we need to set the stack size using ‘spark.executor.extraJavaOptions’. If you dont know where to set, then set the stack size at both driver and executor side.
--conf spark.driver.extraJavaOptions="-Xss4m"
--conf spark.executor.extraJavaOptions="-Xss4m"
If issue is not fixed then we need to increase the stack size like 4m, 8m, 16m, 32m etc. After increasing the some level of stack size for example 1024m issue is still persists then leak is happen at code level. In order to fix the issue, we need to solve the leak at code.
Spark has two JVM process i.e Driver and Executor. To print any JVM process information, JVM supports 3 types of verbose options i.e -verbose:class, -verbose:gc and -verbose:jni.
To enable the verbose class output we need to use -verbose:class option. This option will help us to find out any class loading errors like ClassNotFoundException and NoClassDefFoundError.
To enable from spark side, we need to add the following two parameters:
--conf "spark.driver.extraJavaOptions=-verbose:class" \
--conf "spark.executor.extraJavaOptions=-verbose:class" \
Note:
--verbose - Prints the verbose information like spark configuration.-verbose:class - Prints the details about class loader activity.SparkPi is a simple spark example used to calculate the PI value.
The following are steps to use this tool at various clusters:
Client mode:
$SPARK_HOME/bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 2 \
$SPARK_HOME/examples/jars/spark-examples_*.jar 1000
Cluster mode:
$SPARK_HOME/bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 2 \
$SPARK_HOME/examples/jars/spark-examples_*.jar 1000
Client mode:
spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-example*.jar 10
Cluster mode:
spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-example*.jar 10
Client mode:
spark3-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
/opt/cloudera/parcels/SPARK3/lib/spark3/examples/jars/spark-example*.jar 10
Cluster mode:
spark3-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
/opt/cloudera/parcels/SPARK3/lib/spark3/examples/jars/spark-example*.jar 10
Client mode:
spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-examples_*.jar 10
Cluster mode:
spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-examples_*.jar 10
Client mode:
spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
/usr/hdp/current/spark2-client/examples/jars/spark-examples_*.jar 10
Cluster mode:
spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
/usr/hdp/current/spark2-client/examples/jars/spark-examples_*.jar 10
By default, Spark uses $SPARK_HOME/conf/log4j.properties file to configure log4j and this log4j configuration set at cluster level. Sometimes you need to troubleshoot or fix the performance issues then you need to customize the logs. This blog post will help how to customize the Spark logs for both driver and executor.
The following are steps to customize the logs:
log4j.properties for driver and executorsStep1: Copy the log4.properties to temporary directory for example /tmp
cp $SPARK_HOME/conf/log4.properties /tmp
Step2: Update the log4j.properties file
In the following properties, i have modified logging level from INFO to DEBUG mode.
vi /tmp/log4j.properties
log4j.rootLogger=${root.logger}
root.logger=DEBUG,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
shell.log.level=WARN
log4j.logger.org.spark-project.jetty=WARN
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
log4j.logger.org.apache.spark.repl.Main=${shell.log.level}
log4j.logger.org.apache.spark.api.python.PythonGatewayServer=${shell.log.level}
Step3: Run the Spark Application
Client mode
spark-submit \
--verbose \
--master yarn \
--deploy-mode client \
--files /tmp/log4j.properties#log4j.properties \
--driver-java-options "-Dlog4j.configuration=/tmp/log4j.properties" \
--conf spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j.properties" \
--class org.apache.spark.examples.SparkPi \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-example*.jar 10
Cluster mode
spark-submit \
--verbose \
--master yarn \
--deploy-mode cluster \
--files /tmp/log4j.properties#log4j.properties \
--conf spark.driver.extraJavaOptions="-Dlog4j.configuration=log4j.properties" \
--conf spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j.properties" \
--class org.apache.spark.examples.SparkPi \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-example*.jar 10
log4j.properties for driver and executorsStep1: For driver, create the seperate log4j.properties file say log4j_driver.properties and update the configuration
cp $SPARK_HOME/conf/log4.properties /tmp/log4j_driver.properties
Step2: For executor, create the seperate log4j.properties file say log4j_executor.properties and update the configuration
cp $SPARK_HOME/conf/log4.properties /tmp/log4j_executor.properties
Step3: Run the Spark Application
Client mode
spark-submit \
--verbose \
--master yarn \
--deploy-mode client \
--files /tmp/log4j_driver.properties,/tmp/log4j_executor.properties \
--driver-java-options "-Dlog4j.configuration=/tmp/log4j_driver.properties" \
--conf spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j_executor.properties" \
--class org.apache.spark.examples.SparkPi \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-example*.jar 10
Cluster mode
spark-submit \
--verbose \
--master yarn \
--deploy-mode cluster \
--files /tmp/log4j_driver.properties,/tmp/log4j_executor.properties \
--conf spark.driver.extraJavaOptions="-Dlog4j.configuration=log4j_driver.properties" \
--conf spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j_executor.properties" \
--class org.apache.spark.examples.SparkPi \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-example*.jar 10
Spark Logs Extractor is a simple shell script tool used to collect the Spark Application Logs and Event Logs with the compressed format.
Advantages:
The following are steps to use this tool:
Step1:
Download the spark_logs_extractor.sh script to any location and give the execute permission.
wget https://raw.githubusercontent.com/rangareddy/spark-logs-extractor/main/spark_logs_extractor.sh
chmod +x spark_logs_extractor.sh
Step2:
While Runing the spark_logs_extractor.sh script, provide the application_id.
sh spark_logs_extractor.sh <application_id>
Replace application_id with your spark application id.