Yarn/Spark Log aggregation - log4j, RootLogger, Appenders, yarn.log-aggregation-enable

 

Reference: https://medium.com/@iacomini.riccardo/spark-logging-configuration-in-yarn-faf5ba5fdb01

Where are my log files?

The application will be probably executed inside several containers, which will make logs management really hard without a little help.

Here comes YARN log aggregation feature: YARN will collect all the logs associated to your application, and expose them using the yarn command line client

Reference: https://docs.cloudera.com/runtime/7.2.6/yarn-monitoring-clusters-applications/yarn-monitoring-clusters-applications.pdf

In Cloudera yarn.log-aggregation-enable is enabled by default

yarn-site.xml

<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property><property>
    <name>yarn.nodemanager.log-aggregation.roll-monitoring-interval-    seconds</name>
 <value>3600</value>
</property>