How do I run graphx with Python / pyspark?

by

You should look at GraphFrames (https://github.com/graphframes/graphframes), which wraps GraphX algorithms under the DataFrames API and it provides Python interface.

Here is a quick example from https://graphframes.github.io/graphframes/docs/_site/quick-start.html, with slight modification so that it works

first start pyspark with the graphframes pkg loaded

pyspark --packages graphframes:graphframes:0.1.0-spark1.6

python code:

from graphframes import *

# Create a Vertex DataFrame with unique ID column "id"
v = sqlContext.createDataFrame([
  ("a", "Alice", 34),
  ("b", "Bob", 36),
  ("c", "Charlie", 30),
], ["id", "name", "age"])

# Create an Edge DataFrame with "src" and "dst" columns
e = sqlContext.createDataFrame([
  ("a", "b", "friend"),
  ("b", "c", "follow"),
  ("c", "b", "follow"),
], ["src", "dst", "relationship"])
# Create a GraphFrame
g = GraphFrame(v, e)

# Query: Get in-degree of each vertex.
g.inDegrees.show()

# Query: Count the number of "follow" connections in the graph.
g.edges.filter("relationship = 'follow'").count()

# Run PageRank algorithm, and show results.
results = g.pageRank(resetProbability=0.01, maxIter=20)
results.vertices.select("id", "pagerank").show()

More Related Contents:

  1. How to turn off INFO logging in Spark?
  2. How can PySpark be called in debug mode?
  3. Convert pyspark string to date format
  4. Calling Java/Scala function from a task
  5. How to use JDBC source to write and read data in (Py)Spark?
  6. How to link PyCharm with PySpark?
  7. Applying UDFs on GroupedData in PySpark (with functioning python example)
  8. Pyspark: Split multiple array columns into rows
  9. How to change a dataframe column from String type to Double type in PySpark?
  10. importing pyspark in python shell
  11. collect_list by preserving order based on another variable
  12. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  13. How to use a Scala class inside Pyspark
  14. How to add third-party Java JAR files for use in PySpark
  15. Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
  16. Filter Pyspark dataframe column with None value
  17. PySpark logging from the executor
  18. Convert spark DataFrame column to python list
  19. Comparing columns in Pyspark
  20. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  21. Cannot find col function in pyspark
  22. How to pass a constant value to Python UDF?
  23. How to zip two array columns in Spark SQL
  24. How do I convert an array (i.e. list) column to Vector
  25. Add column sum as new column in PySpark dataframe
  26. How to add suffix and prefix to all columns in python/pyspark dataframe
  27. Spark DAG differs with ‘withColumn’ vs ‘select’
  28. How to drop all columns with null values in a PySpark DataFrame?
  29. Pyspark changing type of column from date to string
  30. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe

Leave a Comment