Spark Dataframe :How to add a index Column : Aka Distributed Data Index

by

With Scala you can use:

import org.apache.spark.sql.functions._ 

df.withColumn("id",monotonicallyIncreasingId)

You can refer to this exemple and scala docs.

With Pyspark you can use: 

from pyspark.sql.functions import monotonically_increasing_id 

df_index = df.select("*").withColumn("id", monotonically_increasing_id())

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to define partitioning of DataFrame?
  3. How do I detect if a Spark DataFrame has a column
  4. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  5. How to split a dataframe into dataframes with same column values?
  6. Provide schema while reading csv file as a dataframe
  7. Dropping a nested column from Spark DataFrame
  8. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  9. How to zip two (or more) DataFrame in Spark
  10. Joining Spark dataframes on the key
  11. Apache Spark how to append new column from list/array to Spark dataframe
  12. Create new Dataframe with empty/null field values
  13. Provide schema while reading csv file as a dataframe in Scala Spark
  14. Renaming column names of a DataFrame in Spark Scala
  15. Derive multiple columns from a single column in a Spark DataFrame
  16. Replace missing values with mean – Spark Dataframe
  17. What is going wrong with `unionAll` of Spark `DataFrame`?
  18. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  19. How to get keys and values from MapType column in SparkSQL DataFrame
  20. Spark DataFrames when udf functions do not accept large enough input variables
  21. Filter spark DataFrame on string contains
  22. Spark 2.0 Dataset vs DataFrame
  23. SparkSQL: How to deal with null values in user defined function?
  24. How to aggregate values into collection after groupBy?
  25. How to read records in JSON format from Kafka using Structured Streaming?
  26. Processing multiple files as independent RDD’s in parallel
  27. MatchError while accessing vector column in Spark 2.0
  28. How to save a spark DataFrame as csv on disk?
  29. How to sort by column in descending order in Spark SQL?
  30. How to create a Dataset of Maps?

Leave a Comment