Spark losing println() on stdout

by

This is a conceptual question…

Imagine You have a big cluster, composed of many workers let’s say n workers and those workers store a partition of an RDD or DataFrame, imagine You start a map task across that data, and inside that map you have a print statement, first of all:

  • Where will that data be printed out?
  • What node has priority and what partition?
  • If all nodes are running in parallel, who will be printed first?
  • How will be this print queue created?

Those are too many questions, thus the designers/maintainers of apache-spark decided logically to drop any support to print statements inside any map-reduce operation (this include accumulators and even broadcast variables).

This also makes sense because Spark is a language designed for very large datasets. While printing can be useful for testing and debugging, you wouldn’t want to print every line of a DataFrame or RDD because they are built to have millions or billions of rows! So why deal with these complicated questions when you wouldn’t even want to print in the first place?

In order to prove this you can run this scala code for example:

// Let's create a simple RDD
val rdd = sc.parallelize(1 to 10000)

def printStuff(x:Int):Int = {
  println(x)
  x + 1
}

// It doesn't print anything! because of a logic design limitation!
rdd.map(printStuff)

// But you can print the RDD by doing the following:
rdd.take(10).foreach(println)

More Related Contents:

  1. Apache Spark – Scala – ReduceByKey – with keys repeating up to twice only
  2. How MapReduce work in Apache Spark and Scala?
  3. Write to multiple outputs by key Spark – one Spark job
  4. How does HashPartitioner work?
  5. Spark 2.0 Dataset vs DataFrame
  6. Spark – load CSV file as DataFrame?
  7. Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
  8. How to define partitioning of DataFrame?
  9. Case class equality in Apache Spark
  10. Why accesing DataFrame from UDF results in NullPointerException?
  11. Dropping a nested column from Spark DataFrame
  12. How to read records in JSON format from Kafka using Structured Streaming?
  13. Spark dataframe write method writing many small files
  14. Change nullable property of column in spark dataframe
  15. Reading DataFrame from partitioned parquet file
  16. How to convert unix timestamp to date in Spark
  17. Perform a typed join in Scala with Spark Datasets
  18. Apache Spark: Get number of records per partition
  19. Left Anti join in Spark?
  20. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  21. DataFrame-ified zipWithIndex
  22. Derive multiple columns from a single column in a Spark DataFrame
  23. How to save a spark DataFrame as csv on disk?
  24. How to sort by column in descending order in Spark SQL?
  25. How to explode an array into multiple columns in Spark
  26. Why is join not possible after show operator?
  27. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  28. How to create DataFrame from Scala’s List of Iterables?
  29. How to create a Dataset of Maps?
  30. Spark ML VectorAssembler returns strange output

Leave a Comment