Why is Spark faster than Hadoop Map Reduce

by

bafna’s answer provides the memory-side of the story, but I want to add other two important facts：DAG and ecosystem

  1. Spark uses “lazy evaluation” to form a directed acyclic graph (DAG) of consecutive computation stages. In this way, the execution plan can be optimized, e.g. to minimize shuffling data around. In contrast, this should be done manually in MapReduce by tuning each MR step. (It would be easier to understand this point if you are familiar with the execution plan optimization in RDBMS or the DAG-style execution of Apache Tez)
  2. Spark ecosystem has established a versatile stack of components to handle SQL, ML, Streaming, Graph Mining tasks. But in the hadoop ecosystem you have to install other packages to do these individual things.

And I want to add that, even if your data is too big for main memory, you can still use spark by choosing to persist you data on disk. Although by doing this you give up the advantages of in-memory processing, you can still benefit from the DAG execution optimization.

Some informative answers on Quora:
here and here.

More Related Contents:

  1. How MapReduce work in Apache Spark and Scala?
  2. Reduce a key-value pair into a key-list pair with Apache Spark
  3. Is gzip format supported in Spark?
  4. Change output filename prefix for DataFrame.write()
  5. Find maximum row per group in Spark DataFrame
  6. How do I skip a header from CSV files in Spark?
  7. How to change dataframe column names in pyspark?
  8. Check if every element in array matches condition
  9. Apache Spark Moving Average
  10. Spark RDD to DataFrame python
  11. Create a custom Transformer in PySpark ML
  12. Processing multiple files as independent RDD’s in parallel
  13. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  14. Spark yarn cluster vs client – how to choose which one to use?
  15. How to convert unix timestamp to date in Spark
  16. How to get other columns when using Spark DataFrame groupby?
  17. Spark load data and add filename as dataframe column
  18. Spark groupByKey alternative
  19. PySpark create new column with mapping from a dict
  20. spark submit add multiple jars in classpath
  21. Spark iteration time increasing exponentially when using join
  22. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  23. Sparklyr: how to center a Spark table based on column?
  24. Derive multiple columns from a single column in a Spark DataFrame
  25. Getting Spark, Python, and MongoDB to work together
  26. What is going wrong with `unionAll` of Spark `DataFrame`?
  27. spark dataframe drop duplicates and keep first
  28. pyspark: count distinct over a window
  29. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  30. Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:

Leave a Comment