What do the numbers on the progress bar mean in spark-shell?

by

What you get is a Console Progress Bar,
[Stage 7: shows the stage you are in now, and
(14174 + 5) / 62500] is (numCompletedTasks + numActiveTasks) / totalNumOfTasksInThisStage]. The progress bar shows numCompletedTasks / totalNumOfTasksInThisStage.

It will be shown when both spark.ui.showConsoleProgress is true (by default) and log level in conf/log4j.properties is ERROR or WARN (!log.isInfoEnabled is true).

Let’s see the code in ConsoleProgressBar.scala that shows it out:

private def show(now: Long, stages: Seq[SparkStageInfo]) {
  val width = TerminalWidth / stages.size
  val bar = stages.map { s =>
    val total = s.numTasks()
    val header = s"[Stage ${s.stageId()}:"
    val tailer = s"(${s.numCompletedTasks()} + ${s.numActiveTasks()}) / $total]"
    val w = width - header.length - tailer.length
    val bar = if (w > 0) {
      val percent = w * s.numCompletedTasks() / total
      (0 until w).map { i =>
        if (i < percent) "=" else if (i == percent) ">" else " "
      }.mkString("")
    } else {
    ""
    }
    header + bar + tailer
  }.mkString("")

  // only refresh if it's changed of after 1 minute (or the ssh connection will be closed
  // after idle some time)
  if (bar != lastProgressBar || now - lastUpdateTime > 60 * 1000L) {
    System.err.print(CR + bar)
    lastUpdateTime = now
  }
  lastProgressBar = bar
}

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Spark – repartition() vs coalesce()
  3. Find maximum row per group in Spark DataFrame
  4. How do I split an RDD into two or more RDDs?
  5. Integrating Spark Structured Streaming with the Confluent Schema Registry
  6. What is the difference between map and flatMap and a good use case for each?
  7. Best Practice to launch Spark Applications via Web Application?
  8. How to save/insert each DStream into a permanent table
  9. Default Partitioning Scheme in Spark
  10. How DAG works under the covers in RDD?
  11. Spark yarn cluster vs client – how to choose which one to use?
  12. Spark on YARN + Secured hbase
  13. Spark load data and add filename as dataframe column
  14. PySpark: how to resample frequencies
  15. Explode array data into rows in spark [duplicate]
  16. Spark: how to get the number of written rows?
  17. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  18. When are accumulators truly reliable?
  19. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  20. How to get day of week in SparkSQL?
  21. Array Intersection in Spark SQL
  22. Mac spark-shell Error initializing SparkContext
  23. pyspark: rolling average using timeseries data
  24. Why spark application fail with “executor.CoarseGrainedExecutorBackend: Driver Disassociated”?
  25. What does “Correlated scalar subqueries must be Aggregated” mean?
  26. How to improve broadcast Join speed with between condition in Spark
  27. Spark ALS predictAll returns empty
  28. Spark SQL referencing attributes of UDT
  29. Why is the fold action necessary in Spark?
  30. What will spark do if I don’t have enough memory?

Leave a Comment