Spark-Monotonically increasing id not working as expected in dataframe?

by

It works as expected. This function is not intended for generating consecutive values. Instead it encodes partition number and index by partition

The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the record number within each partition in the lower 33 bits. The assumption is that the data frame has less than 1 billion partitions, and each partition has less than 8 billion records.

As an example, consider a DataFrame with two partitions, each with 3 records. This expression would return the following IDs:

0, 1, 2, 8589934592 (1L << 33), 8589934593, 8589934594.

If you want consecutive numbers, use RDD.zipWithIndex.

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to convert rdd object to dataframe in spark
  3. How to define and use a User-Defined Aggregate Function in Spark SQL?
  4. How do I detect if a Spark DataFrame has a column
  5. How to define schema for custom type in Spark SQL?
  6. How to avoid duplicate columns after join?
  7. Flattening Rows in Spark
  8. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  9. Spark unionAll multiple dataframes
  10. How can I pass extra parameters to UDFs in Spark SQL?
  11. How to split a dataframe into dataframes with same column values?
  12. Provide schema while reading csv file as a dataframe
  13. How to zip two (or more) DataFrame in Spark
  14. How to flatmap a nested Dataframe in Spark
  15. How to save DataFrame directly to Hive?
  16. About how to add a new column to an existing DataFrame with random values in Scala
  17. Joining Spark dataframes on the key
  18. How to use Column.isin with list?
  19. How to write unit tests in Spark 2.0+?
  20. how to filter out a null value from spark dataframe
  21. How to compare two dataframe and print columns that are different in scala
  22. What are the various join types in Spark?
  23. Provide schema while reading csv file as a dataframe in Scala Spark
  24. Renaming column names of a DataFrame in Spark Scala
  25. Replace missing values with mean – Spark Dataframe
  26. What is going wrong with `unionAll` of Spark `DataFrame`?
  27. How to calculate the size of dataframe in bytes in Spark?
  28. Exploding nested Struct in Spark dataframe
  29. How to get keys and values from MapType column in SparkSQL DataFrame
  30. How to create DataFrame from Scala’s List of Iterables?

Leave a Comment