Where are logs in Spark on YARN?

by

You can access logs through the command

yarn logs -applicationId <application ID> [OPTIONS]

general options are:

  • appOwner <Application Owner> – AppOwner (assumed to be current user if not specified)
  • containerId <Container ID> – ContainerId (must be specified if node
    address is specified)
  • nodeAddress <Node Address> – NodeAddress in the format nodename:port (must be specified if container id is specified)

Examples:

yarn logs -applicationId application_1414530900704_0003                                      
yarn logs -applicationId application_1414530900704_0003 myuserid

// the user ids are different
yarn logs -applicationId <appid> -appOwner <userid>

More Related Contents:

  1. Spark on yarn concept understanding
  2. Apache Spark: The number of cores vs. the number of executors
  3. Apache Hadoop Yarn – Underutilization of cores
  4. Begenner at spark Big data programming (spark code)
  5. Container is running beyond memory limits
  6. How to open/stream .zip files through Spark?
  7. How to get ID of a map task in Spark?
  8. How can I access S3/S3n from a local Hadoop 2.6 installation?
  9. Pyspark: get list of files/directories on HDFS path
  10. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  11. How to access s3a:// files from Apache Spark?
  12. Is it better to have one large parquet file or lots of smaller parquet files?
  13. Easiest way to install Python dependencies on Spark executor nodes?
  14. Spark iterate HDFS directory
  15. How to log using log4j to local file system inside a Spark application that runs on YARN?
  16. Cannot Read a file from HDFS using Spark
  17. Information about big data and hadoop [closed]
  18. Spark – load CSV file as DataFrame?
  19. How to turn off INFO logging in Spark?
  20. Chaining multiple MapReduce jobs in Hadoop
  21. Set hadoop system user for client embedded in Java webapp
  22. How to get the input file name in the mapper in a Hadoop program?
  23. How do I run graphx with Python / pyspark?
  24. Namenode not getting started
  25. Hive: Add partitions for existing folder structure
  26. How do I log from my Python Spark script
  27. How to load data to hive from HDFS without removing the source file?
  28. Default number of reducers
  29. Create HIVE Table with multi character delimiter
  30. hadoop: difference between 0 reducer and identity reducer?

Leave a Comment