Finding duplicates from large data set using Apache Spark

by

Load the data into spark and apply group by on the email column.. after that check for bag and apply any distance algorithm on the first name and last name columns. This should be pretty straight forward in spark

val df = sc.textFile("hdfs path of data");

df.mapToPair("email", <whole_record>)
  .groupBy(//will be done based on key)
  .map(//will run on each bag )

More Related Contents:

  1. How to save/insert each DStream into a permanent table
  2. How to melt Spark DataFrame?
  3. Using a column value as a parameter to a spark DataFrame function
  4. Unpivot in spark-sql/pyspark
  5. Split Spark Dataframe string column into multiple columns
  6. Overwrite specific partitions in spark dataframe write method
  7. Avoid performance impact of a single partition mode in Spark window functions
  8. How to import multiple csv files in a single load?
  9. How to access element of a VectorUDT column in a Spark DataFrame?
  10. Spark Dataframe validating column names for parquet writes
  11. How to check if spark dataframe is empty?
  12. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  13. Python Spark Cumulative Sum by Group Using DataFrame
  14. How to split a list to multiple columns in Pyspark?
  15. pyspark dataframe filter or include based on list
  16. How to loop through each row of dataFrame in pyspark
  17. Fill in null with previously known good value with pyspark
  18. PySpark – get row number for each row in a group
  19. Save ML model for future usage
  20. Pyspark: Pass multiple columns in UDF
  21. Apache spark dealing with case statements
  22. Keep only duplicates from a DataFrame regarding some field
  23. pyspark: count distinct over a window
  24. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  25. How can I access python variable in Spark SQL?
  26. How to exclude multiple columns in Spark dataframe in Python
  27. Stratified sampling with pyspark
  28. How to turn off scientific notation in pyspark?
  29. Spark ALS predictAll returns empty
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment