How to bucket the range of values from a column and count how many values fall into each interval in scala?

by

You can use the scala Bucketizer. There’s a good example here:
https://spark.apache.org/docs/2.2.0/ml-features.html#bucketizer

After you use the bucketizer you have a dataframe with a bucket index (i.e index 1, 2, and 3 might correspond to values 1-5, 6-10, 11-15, respectively). You can do a .groupBy and .agg (or use SQL) to get a count of records in each index group).

More Related Contents:

  1. Apache Spark – Scala – ReduceByKey – with keys repeating up to twice only
  2. Write to multiple outputs by key Spark – one Spark job
  3. How does HashPartitioner work?
  4. Spark 2.0 Dataset vs DataFrame
  5. Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
  6. Automatically and Elegantly flatten DataFrame in Spark SQL
  7. Spark extracting values from a Row
  8. Why accesing DataFrame from UDF results in NullPointerException?
  9. How to pass -D parameter or environment variable to Spark job?
  10. SparkSQL: How to deal with null values in user defined function?
  11. How to aggregate values into collection after groupBy?
  12. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  13. How to read records in JSON format from Kafka using Structured Streaming?
  14. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  15. MatchError while accessing vector column in Spark 2.0
  16. Scala spark, listbuffer is empty
  17. Spark / Scala: forward fill with last observation
  18. Change nullable property of column in spark dataframe
  19. Reading DataFrame from partitioned parquet file
  20. Perform a typed join in Scala with Spark Datasets
  21. Apache Spark: Get number of records per partition
  22. Left Anti join in Spark?
  23. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  24. DataFrame-ified zipWithIndex
  25. How to explode an array into multiple columns in Spark
  26. Why is join not possible after show operator?
  27. Spark : how to run spark file from spark shell
  28. How can I connect to a postgreSQL database into Apache Spark using scala?
  29. Spark column string replace when present in other column (row)
  30. How to vectorize DataFrame columns for ML algorithms?

Leave a Comment