how to add Row id in pySpark dataframes [duplicate]

by

You can use also use a function from sql package. It will generate a unique id, however it will not be sequential as it depends on the number of partitions. I believe it is available in Spark 1.5 +

from pyspark.sql.functions import monotonicallyIncreasingId

# This will return a new DF with all the columns + id
res = df.withColumn("id", monotonicallyIncreasingId())

Edit: 19/1/2017

As commented by @Sean

Use monotonically_increasing_id() instead from Spark 1.6 and on

More Related Contents:

  1. How to add a constant column in a Spark DataFrame?
  2. Load CSV file with Spark
  3. How do I add a new column to a Spark DataFrame (using PySpark)?
  4. How to perform union on two DataFrames with different amounts of columns in spark?
  5. Does spark predicate pushdown work with JDBC?
  6. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  7. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  8. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  9. Passing a data frame column and external list to udf under withColumn
  10. Updating a dataframe column in spark
  11. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  12. Spark add new column to dataframe with value from previous row
  13. Best way to get the max value in a Spark dataframe column
  14. Pyspark: Replacing value in a column by searching a dictionary
  15. Apache Spark Python Cosine Similarity over DataFrames
  16. Pivot String column on Pyspark Dataframe
  17. GroupBy column and filter rows with maximum value in Pyspark
  18. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  19. Cannot find col function in pyspark
  20. How to pass a constant value to Python UDF?
  21. How to zip two array columns in Spark SQL
  22. Concatenate two PySpark dataframes
  23. How do I convert an array (i.e. list) column to Vector
  24. Filtering DataFrame using the length of a column
  25. Add column sum as new column in PySpark dataframe
  26. When to cache a DataFrame?
  27. How to add suffix and prefix to all columns in python/pyspark dataframe
  28. How to drop all columns with null values in a PySpark DataFrame?
  29. Pyspark changing type of column from date to string
  30. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe

Leave a Comment