Operate on neighbor elements in RDD in Spark

by

The most efficient solution is to use sliding method:

import org.apache.spark.mllib.rdd.RDDFunctions._

val rdd = sc.parallelize(Seq(1, 3,-1, 0, 2, -4, 6))
  .sortBy(identity)
  .sliding(2)
  .map{case Array(x, y) => y - x}

More Related Contents:

  1. Encoder error while trying to map dataframe row to updated row
  2. How to define schema for custom type in Spark SQL?
  3. Specifying the filename when saving a DataFrame as a CSV [duplicate]
  4. NullPointerException in Scala Spark, appears to be caused be collection type?
  5. how to make saveAsTextFile NOT split output into multiple file?
  6. What does setMaster `local[*]` mean in spark?
  7. Spark unionAll multiple dataframes
  8. How can I pass extra parameters to UDFs in Spark SQL?
  9. How to split a dataframe into dataframes with same column values?
  10. Spark performance for Scala vs Python
  11. How to use regex to include/exclude some input files in sc.textFile?
  12. How to define a custom aggregation function to sum a column of Vectors?
  13. How to write spark streaming DF to Kafka topic
  14. Stackoverflow due to long RDD Lineage
  15. Better way to convert a string field into timestamp in Spark
  16. Apache Spark how to append new column from list/array to Spark dataframe
  17. How to find spark RDD/Dataframe size?
  18. Encode an ADT / sealed trait hierarchy into Spark DataSet column
  19. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  20. Replace null values in Spark DataFrame
  21. Spark RDD default number of partitions
  22. Why does Spark RDD partition has 2GB limit for HDFS?
  23. Calculate Cosine Similarity Spark Dataframe
  24. How can I connect to a postgreSQL database into Apache Spark using scala?
  25. Spark Dataframe Random UUID changes after every transformation/action
  26. Spark column string replace when present in other column (row)
  27. How to vectorize DataFrame columns for ML algorithms?
  28. How to set hadoop configuration values from pyspark
  29. Spark DataFrames when udf functions do not accept large enough input variables
  30. Filter spark DataFrame on string contains

Leave a Comment