Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?

by

sparkContext is a Scala implementation entry point and JavaSparkContext is a java wrapper of sparkContext.

SQLContext is entry point of SparkSQL which can be received from sparkContext.Prior to 2.x.x, RDD ,DataFrame and Data-set were three different data abstractions.Since Spark 2.x.x, All three data abstractions are unified and SparkSession is the unified entry point of Spark.

An additional note is , RDD meant for unstructured data, strongly typed data and DataFrames are for structured and loosely typed data. You can check

Is there any method to convert or create Context using Sparksession ?

yes. its sparkSession.sparkContext() and for SQL, sparkSession.sqlContext()

Can I completely replace all the Context using one single entry SparkSession ?

yes. you can get respective contexs from sparkSession.

Does all the functions in SQLContext, SparkContext,JavaSparkContext etc are added in SparkSession?

Not directly. you got to get respective context and make use of it.something like backward compatibility

How to use such function in SparkSession?

get respective context and make use of it.

How to create the following using SparkSession?

  1. RDD can be created from sparkSession.sparkContext.parallelize(???)
  2. JavaRDD same applies with this but in java implementation
  3. JavaPairRDD sparkSession.sparkContext.parallelize(???).map(//making your data as key-value pair here is one way)
  4. Dataset what sparkSession returns is Dataset if it is structured data.

More Related Contents:

  1. Spark read file from S3 using sc.textFile (“s3n://…)
  2. Matrix Multiplication in Apache Spark [closed]
  3. Resolving dependency problems in Apache Spark
  4. Add JAR files to a Spark job – spark-submit
  5. Spark: How to map Python with Scala or Java User Defined Functions?
  6. Spark Strutured Streaming automatically converts timestamp to local time
  7. How can I update a broadcast variable in spark streaming?
  8. Apache Spark – foreach Vs foreachPartition When to use What?
  9. What are the Spark transformations that causes a Shuffle?
  10. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  11. Is gzip format supported in Spark?
  12. Change output filename prefix for DataFrame.write()
  13. How can I force Spark to execute code?
  14. How to get element by Index in Spark RDD (Java)
  15. How to save models from ML Pipeline to S3 or HDFS?
  16. Encoder error while trying to map dataframe row to updated row
  17. Jvm takes a long time to resolve ip-address for localhost
  18. Apache Spark: map vs mapPartitions?
  19. “eval” in Scala
  20. Get the changed HTML content after it’s updated by Javascript? (htmlunit)
  21. Spark performance for Scala vs Python
  22. Writing Custom Kafka Serializer
  23. Stackoverflow due to long RDD Lineage
  24. How to find spark RDD/Dataframe size?
  25. PySpark: java.lang.OutofMemoryError: Java heap space
  26. java.lang.NoClassDefFoundError: org/apache/spark/Logging
  27. Exception: Unexpected end of ZLIB input stream
  28. How to optimize for-comprehensions and loops in Scala?
  29. Reading data from Azure Blob with Spark
  30. Java compile speed vs Scala compile speed

Leave a Comment