How to use COGROUP for large datasets

by

When you use collect() you are basically telling spark to move all the resulting data back to the master node, which can easily produce a bottleneck. You are no longer using Spark at that point, just a plain array in a single machine.

To trigger computation just use something that requires the data at every node, that’s why executors live on top of a distributed file system. For instance saveAsTextFile().

Here are some basic examples.

Remember, the entire objective here (that is, if you have big data) is to move the code to your data and compute there, not to bring all the data to the computation.

More Related Contents:

  1. How to convert rdd object to dataframe in spark
  2. Spark 2.0 Dataset vs DataFrame
  3. Spark – load CSV file as DataFrame?
  4. How to define partitioning of DataFrame?
  5. Automatically and Elegantly flatten DataFrame in Spark SQL
  6. Spark extracting values from a Row
  7. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  8. SparkSQL: How to deal with null values in user defined function?
  9. How to aggregate values into collection after groupBy?
  10. Dropping a nested column from Spark DataFrame
  11. How to read records in JSON format from Kafka using Structured Streaming?
  12. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  13. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  14. Processing multiple files as independent RDD’s in parallel
  15. MatchError while accessing vector column in Spark 2.0
  16. Spark / Scala: forward fill with last observation
  17. Filling gaps in timeseries Spark
  18. How to write unit tests in Spark 2.0+?
  19. How to compare two dataframe and print columns that are different in scala
  20. Perform a typed join in Scala with Spark Datasets
  21. Apache Spark: Get number of records per partition
  22. Renaming column names of a DataFrame in Spark Scala
  23. DataFrame-ified zipWithIndex
  24. Derive multiple columns from a single column in a Spark DataFrame
  25. How to save a spark DataFrame as csv on disk?
  26. What is going wrong with `unionAll` of Spark `DataFrame`?
  27. How to sort by column in descending order in Spark SQL?
  28. How to get keys and values from MapType column in SparkSQL DataFrame
  29. Why is join not possible after show operator?
  30. Difference between two rows in Spark dataframe

Leave a Comment