PySpark Throwing error Method __getnewargs__([]) does not exist

by

Using spark inside flatMap or any transformation that occures on executors is not allowed (spark session is available on driver only). It is also not possible to create RDD of RDDs (see: Is it possible to create nested RDDs in Apache Spark?)

But you can achieve this transformation in another way – read all content of all_files.txt into dataframe, use local map to make them dataframes and local reduce to union all, see example:

>>> filenames = spark.read.text('all_files.txt').collect()
>>> dataframes = map(lambda r: spark.read.text(r[0]), filenames)
>>> all_lines_df = reduce(lambda df1, df2: df1.unionAll(df2), dataframes)

More Related Contents:

  1. Calling Java/Scala function from a task
  2. Efficient string matching in Apache Spark
  3. Load CSV file with Spark
  4. How to link PyCharm with PySpark?
  5. How to turn off INFO logging in Spark?
  6. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  7. importing pyspark in python shell
  8. collect_list by preserving order based on another variable
  9. Rename nested field in spark dataframe
  10. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  11. How to determine if object is a valid key-value pair in PySpark
  12. Updating a dataframe column in spark
  13. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  14. Best way to get the max value in a Spark dataframe column
  15. How to map features from the output of a VectorAssembler back to the column names in Spark ML?
  16. Apache Spark Python Cosine Similarity over DataFrames
  17. Comparing columns in Pyspark
  18. GroupBy column and filter rows with maximum value in Pyspark
  19. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  20. Cannot find col function in pyspark
  21. How to pass a constant value to Python UDF?
  22. How to zip two array columns in Spark SQL
  23. How do I convert an array (i.e. list) column to Vector
  24. Add column sum as new column in PySpark dataframe
  25. Pyspark – converting json string to DataFrame
  26. How to add suffix and prefix to all columns in python/pyspark dataframe
  27. Spark DAG differs with ‘withColumn’ vs ‘select’
  28. Pyspark changing type of column from date to string
  29. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe
  30. Random numbers generation in PySpark

Leave a Comment