Link Spark with iPython Notebook

by

I have Jupyter installed, and indeed It is simpler than you think:

  1. Install anaconda for OSX.

  2. Install jupyter typing the next line in your terminal Click me for more info.

    ilovejobs@mymac:~$ conda install jupyter

  3. Update jupyter just in case.

    ilovejobs@mymac:~$ conda update jupyter

  4. Download Apache Spark and compile it, or download and uncompress Apache Spark 1.5.1 + Hadoop 2.6.

    ilovejobs@mymac:~$ cd Downloads 
ilovejobs@mymac:~/Downloads$ wget http://www.apache.org/dyn/closer.lua/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.6.tgz

  5. Create an Apps folder on your home (i.e):

    ilovejobs@mymac:~/Downloads$ mkdir ~/Apps

  6. Move the uncompressed folder spark-1.5.1 to the ~/Apps directory. 

    ilovejobs@mymac:~/Downloads$ mv spark-1.5.1/ ~/Apps

  7. Move to the ~/Apps directory and verify that spark is there.

    ilovejobs@mymac:~/Downloads$ cd ~/Apps
ilovejobs@mymac:~/Apps$ ls -l
drwxr-xr-x ?? ilovejobs ilovejobs 4096 ?? ?? ??:?? spark-1.5.1

  8. Here is the first tricky part. Add the spark binaries to your $PATH:

    ilovejobs@mymac:~/Apps$ cd
ilovejobs@mymac:~$ echo "export $HOME/apps/spark/bin:$PATH" >> .profile

  9. Here is the second tricky part. Add this environment variables also:

    ilovejobs@mymac:~$ echo "export PYSPARK_DRIVER_PYTHON=ipython" >> .profile
ilovejobs@mymac:~$ echo "export PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark" >> .profile

  10. Source the profile to make these variables available for this terminal

    ilovejobs@mymac:~$ source .profile

  11. Create a ~/notebooks directory.

    ilovejobs@mymac:~$ mkdir notebooks

  12. Move to ~/notebooks and run pyspark:

    ilovejobs@mymac:~$ cd notebooks
ilovejobs@mymac:~/notebooks$ pyspark

Notice that you can add those variables to the .bashrc located in your home.
Now be happy, You should be able to run jupyter with a pyspark kernel (It will show it as a python 2 but it will use spark)

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Find maximum row per group in Spark DataFrame
  3. How do I split an RDD into two or more RDDs?
  4. How to access element of a VectorUDT column in a Spark DataFrame?
  5. Spark Dataframe validating column names for parquet writes
  6. How to improve performance for slow Spark jobs using DataFrame and JDBC connection?
  7. TypeError: Column is not iterable – How to iterate over ArrayType()?
  8. Python Spark Cumulative Sum by Group Using DataFrame
  9. How to save/insert each DStream into a permanent table
  10. Spark load data and add filename as dataframe column
  11. PySpark: how to resample frequencies
  12. Explode array data into rows in spark [duplicate]
  13. Why does Spark think this is a cross / Cartesian join
  14. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  15. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  16. Save ML model for future usage
  17. Apache spark dealing with case statements
  18. pyspark: count distinct over a window
  19. pyspark: Efficiently have partitionBy write to same number of total partitions as original table
  20. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  21. AWS EMR – ModuleNotFoundError: No module named ‘pyarrow’
  22. Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:
  23. pyspark: rolling average using timeseries data
  24. How to exclude multiple columns in Spark dataframe in Python
  25. Stratified sampling with pyspark
  26. Get CSV to Spark dataframe
  27. How to turn off scientific notation in pyspark?
  28. Spark ALS predictAll returns empty
  29. Why is the fold action necessary in Spark?
  30. How to add a SparkListener from pySpark in Python?

Leave a Comment