Spark: how to get the number of written rows?

by

If you really want you can add custom listener and extract number of written rows from outputMetrics. Very simple example can look like this:

import org.apache.spark.scheduler.{SparkListener, SparkListenerTaskEnd}

var recordsWrittenCount = 0L

sc.addSparkListener(new SparkListener() { 
  override def onTaskEnd(taskEnd: SparkListenerTaskEnd) {
    synchronized {
      recordsWrittenCount += taskEnd.taskMetrics.outputMetrics.recordsWritten 
    }
  }
})

sc.parallelize(1 to 10, 2).saveAsTextFile("/tmp/foobar")
recordsWrittenCount
// Long = 10

but this part of the API is intended for internal usage.

More Related Contents:

  1. Airflow BashOperator return exit code 0 even when task failed and return exit code 1
  2. Unable to fetch the value of Println in apache spark
  3. How to serve a Spark MLlib model?
  4. Avoid performance impact of a single partition mode in Spark window functions
  5. Spark DataFrame: count distinct values of every column
  6. Spark Standalone Number Executors/Cores Control
  7. pyspark dataframe filter or include based on list
  8. Spark: Reading files using different delimiter than new line
  9. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  10. How to get word details from TF Vector RDD in Spark ML Lib?
  11. Use collect_list and collect_set in Spark SQL
  12. Apache Spark: setting executor instances does not change the executors
  13. How to find mean of grouped Vector columns in Spark SQL?
  14. Generate a Spark StructType / Schema from a case class
  15. How to loop through each row of dataFrame in pyspark
  16. Fill in null with previously known good value with pyspark
  17. How to group by common element in array?
  18. Spark: Best practice for retrieving big data from RDD to local machine
  19. Understanding Spark serialization
  20. Pyspark: Pass multiple columns in UDF
  21. How does Spark aggregate function – aggregateByKey work?
  22. Convert null values to empty array in Spark DataFrame
  23. spark-shell error on Windows – can it be ignored if not using hadoop?
  24. How can I access python variable in Spark SQL?
  25. What do the numbers on the progress bar mean in spark-shell?
  26. pyspark: rolling average using timeseries data
  27. How to improve broadcast Join speed with between condition in Spark
  28. Spark ALS predictAll returns empty
  29. Spark SQL referencing attributes of UDT
  30. How to add a SparkListener from pySpark in Python?

Leave a Comment