Spark DataFrame: count distinct values of every column

by

In pySpark you could do something like this, using countDistinct():

from pyspark.sql.functions import col, countDistinct

df.agg(*(countDistinct(col(c)).alias(c) for c in df.columns))

Similarly in Scala :

import org.apache.spark.sql.functions.countDistinct
import org.apache.spark.sql.functions.col

df.select(df.columns.map(c => countDistinct(col(c)).alias(c)): _*)

If you want to speed things up at the potential loss of accuracy, you could also use approxCountDistinct().

More Related Contents:

  1. pyspark: count distinct over a window
  2. Finding duplicates from large data set using Apache Spark
  3. Find maximum row per group in Spark DataFrame
  4. How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?
  5. Spark SQL – load data with JDBC using SQL statement, not table name
  6. Spark Dataframe validating column names for parquet writes
  7. How to optimize partitioning when migrating data from JDBC source?
  8. How to split a list to multiple columns in Pyspark?
  9. Filtering a spark dataframe based on date
  10. How to save/insert each DStream into a permanent table
  11. Spark DataFrame Schema Nullable Fields
  12. Spark load data and add filename as dataframe column
  13. PySpark: how to resample frequencies
  14. Why does Spark think this is a cross / Cartesian join
  15. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  16. Spark SQL broadcast hash join
  17. What is the difference between Apache Spark SQLContext vs HiveContext?
  18. What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?
  19. Spark lists all leaf node even in partitioned data
  20. Array Intersection in Spark SQL
  21. reduce result datasets into single dataset
  22. How to exclude multiple columns in Spark dataframe in Python
  23. How to get Kafka offsets for structured query for manual and reliable offset management?
  24. Stratified sampling with pyspark
  25. How to use groupBy to collect rows into a map?
  26. Convert between spark.SQL DataFrame and pandas DataFrame [duplicate]
  27. How to turn off scientific notation in pyspark?
  28. When to use Spark DataFrame/Dataset API and when to use plain RDD?
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment