How to stop INFO messages displaying on spark console?

by

Edit your conf/log4j.properties file and change the following line:

log4j.rootCategory=INFO, console

to

log4j.rootCategory=ERROR, console

Another approach would be to :

Start spark-shell and type in the following:

import org.apache.log4j.Logger
import org.apache.log4j.Level

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

You won’t see any logs after that.

Other options for Level include: all, debug, error, fatal, info, off, trace, trace_int, warn

Details about each can be found in the documentation.

More Related Contents:

  1. Airflow BashOperator return exit code 0 even when task failed and return exit code 1
  2. Using a column value as a parameter to a spark DataFrame function
  3. How to serve a Spark MLlib model?
  4. Overwrite specific partitions in spark dataframe write method
  5. Avoid performance impact of a single partition mode in Spark window functions
  6. Convert date from String to Date format in Dataframes
  7. Spark DataFrame: count distinct values of every column
  8. Spark Standalone Number Executors/Cores Control
  9. pyspark dataframe filter or include based on list
  10. PySpark logging from the executor
  11. How to get word details from TF Vector RDD in Spark ML Lib?
  12. Use collect_list and collect_set in Spark SQL
  13. Apache Spark: setting executor instances does not change the executors
  14. How to find mean of grouped Vector columns in Spark SQL?
  15. Generate a Spark StructType / Schema from a case class
  16. How to loop through each row of dataFrame in pyspark
  17. Fill in null with previously known good value with pyspark
  18. Spark using python: How to resolve Stage x contains a task of very large size (xxx KB). The maximum recommended task size is 100 KB
  19. Keep only duplicates from a DataFrame regarding some field
  20. reading json file in pyspark
  21. Which cluster type should I choose for Spark? [closed]
  22. Overwrite only some partitions in a partitioned spark Dataset
  23. Using windowing functions in Spark
  24. How to assign unique contiguous numbers to elements in a Spark RDD
  25. What is spark.driver.maxResultSize?
  26. Spark + s3 – error – java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
  27. How to get Kafka offsets for structured query for manual and reliable offset management?
  28. How to use groupBy to collect rows into a map?
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment