How to assign unique contiguous numbers to elements in a Spark RDD

by

Starting with Spark 1.0 there are two methods you can use to solve this easily:

  • RDD.zipWithIndex is just like Seq.zipWithIndex, it adds contiguous (Long) numbers. This needs to count the elements in each partition first, so your input will be evaluated twice. Cache your input RDD if you want to use this.
  • RDD.zipWithUniqueId also gives you unique Long IDs, but they are not guaranteed to be contiguous. (They will only be contiguous if each partition has the same number of elements.) The upside is that this does not need to know anything about the input, so it will not cause double-evaluation.

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. How to serve a Spark MLlib model?
  3. How to handle categorical features with spark-ml?
  4. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  5. How to get word details from TF Vector RDD in Spark ML Lib?
  6. ALS model – how to generate full_u * v^t * v?
  7. Save ML model for future usage
  8. What’s the difference between Spark ML and MLLIB packages
  9. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  10. Spark ALS predictAll returns empty
  11. Using a column value as a parameter to a spark DataFrame function
  12. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  13. What does “Stage Skipped” mean in Apache Spark web UI?
  14. Encode and assemble multiple features in PySpark
  15. How to import multiple csv files in a single load?
  16. Which operations preserve RDD order?
  17. How to split a list to multiple columns in Pyspark?
  18. Matrix Multiplication in Apache Spark [closed]
  19. What is the maximum size for a broadcast object in Spark?
  20. How does createOrReplaceTempView work in Spark?
  21. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  22. PySpark – get row number for each row in a group
  23. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  24. reduce result datasets into single dataset
  25. spark createOrReplaceTempView vs createGlobalTempView
  26. Why is predicate pushdown not used in typed Dataset API (vs untyped DataFrame API)?
  27. Convert between spark.SQL DataFrame and pandas DataFrame [duplicate]
  28. How to turn off scientific notation in pyspark?
  29. What is the concept of application, job, stage and task in spark?
  30. What is an optimized way of joining large tables in Spark SQL

Leave a Comment