Explode in PySpark

by

explode and split are SQL functions. Both operate on SQL Column. split takes a Java regular expression as a second argument. If you want to separate data on arbitrary whitespace you’ll need something like this:

df = sqlContext.createDataFrame(
    [('cat \n\n elephant rat \n rat cat', )], ['word']
)

df.select(explode(split(col("word"), "\s+")).alias("word")).show()

## +--------+
## |    word|
## +--------+
## |     cat|
## |elephant|
## |     rat|
## |     rat|
## |     cat|
## +--------+

More Related Contents:

  1. Convert pyspark string to date format
  2. How to split Vector into columns – using PySpark
  3. How to use JDBC source to write and read data in (Py)Spark?
  4. Applying UDFs on GroupedData in PySpark (with functioning python example)
  5. Spark Dataframe distinguish columns with duplicated name
  6. Pyspark: Split multiple array columns into rows
  7. How to change dataframe column names in pyspark?
  8. How to change a dataframe column from String type to Double type in PySpark?
  9. Retrieve top n in each group of a DataFrame in pyspark
  10. Pyspark: explode json in column to multiple columns
  11. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  12. How to use a Scala class inside Pyspark
  13. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  14. Filter Pyspark dataframe column with None value
  15. Cast column containing multiple string date formats to DateTime in Spark
  16. How to explode multiple columns of a dataframe in pyspark
  17. PySpark: multiple conditions in when clause
  18. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  19. Create Spark DataFrame. Can not infer schema for type
  20. Load CSV file with PySpark
  21. Create single row dataframe from list of list PySpark
  22. Cannot find col function in pyspark
  23. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  24. How do I convert an array (i.e. list) column to Vector
  25. Add column sum as new column in PySpark dataframe
  26. When to cache a DataFrame?
  27. How to add suffix and prefix to all columns in python/pyspark dataframe
  28. How to drop all columns with null values in a PySpark DataFrame?
  29. Pyspark changing type of column from date to string
  30. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe

Leave a Comment