Create single row dataframe from list of list PySpark

by

I find it’s useful to think of the argument to createDataFrame() as a list of tuples where each entry in the list corresponds to a row in the DataFrame and each element of the tuple corresponds to a column.

You can get your desired output by making each element in the list a tuple:

data = [([1.1, 1.2],), ([1.3, 1.4],), ([1.5, 1.6],)]
dataframe = sqlCtx.createDataFrame(data, ['features'])
dataframe.show()
#+----------+
#|  features|
#+----------+
#|[1.1, 1.2]|
#|[1.3, 1.4]|
#|[1.5, 1.6]|
#+----------+

Or if changing the source is cumbersome, you can equivalently do:

data = [[1.1, 1.2], [1.3, 1.4], [1.5, 1.6]]
dataframe = sqlCtx.createDataFrame(map(lambda x: (x, ), data), ['features'])
dataframe.show()
#+----------+
#|  features|
#+----------+
#|[1.1, 1.2]|
#|[1.3, 1.4]|
#|[1.5, 1.6]|
#+----------+

More Related Contents:

  1. Spark Equivalent of IF Then ELSE
  2. How to add a constant column in a Spark DataFrame?
  3. Load CSV file with Spark
  4. How do I add a new column to a Spark DataFrame (using PySpark)?
  5. How to perform union on two DataFrames with different amounts of columns in spark?
  6. Does spark predicate pushdown work with JDBC?
  7. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  8. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  9. Explode in PySpark
  10. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  11. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  12. Passing a data frame column and external list to udf under withColumn
  13. Updating a dataframe column in spark
  14. Dividing complex rows of dataframe to simple rows in Pyspark
  15. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  16. Spark add new column to dataframe with value from previous row
  17. Best way to get the max value in a Spark dataframe column
  18. Pyspark: Replacing value in a column by searching a dictionary
  19. Apache Spark Python Cosine Similarity over DataFrames
  20. Pivot String column on Pyspark Dataframe
  21. GroupBy column and filter rows with maximum value in Pyspark
  22. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  23. Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
  24. How do I convert an array (i.e. list) column to Vector
  25. Add column sum as new column in PySpark dataframe
  26. When to cache a DataFrame?
  27. How to add suffix and prefix to all columns in python/pyspark dataframe
  28. How to drop all columns with null values in a PySpark DataFrame?
  29. Pyspark changing type of column from date to string
  30. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe

Leave a Comment