Spark Dataframe validating column names for parquet writes

by

For everyone experiencing this in pyspark: this even happened to me after renaming the columns. One way I could get this to work after some iterations is this:

file = "/opt/myfile.parquet"
df = spark.read.parquet(file)
for c in df.columns:
    df = df.withColumnRenamed(c, c.replace(" ", ""))

df = spark.read.schema(df.schema).parquet(file)

More Related Contents:

  1. How to save/insert each DStream into a permanent table
  2. Finding duplicates from large data set using Apache Spark
  3. Using a column value as a parameter to a spark DataFrame function
  4. Find maximum row per group in Spark DataFrame
  5. Split Spark Dataframe string column into multiple columns
  6. Avoid performance impact of a single partition mode in Spark window functions
  7. How to access element of a VectorUDT column in a Spark DataFrame?
  8. How to check if spark dataframe is empty?
  9. How to split a list to multiple columns in Pyspark?
  10. pyspark dataframe filter or include based on list
  11. Pyspark : forward fill with last observation for a DataFrame
  12. Spark load data and add filename as dataframe column
  13. PySpark: how to resample frequencies
  14. How to loop through each row of dataFrame in pyspark
  15. Fill in null with previously known good value with pyspark
  16. Why does Spark think this is a cross / Cartesian join
  17. PySpark – get row number for each row in a group
  18. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  19. Rename more than one column using withColumnRenamed
  20. Keep only duplicates from a DataFrame regarding some field
  21. reading json file in pyspark
  22. pyspark: count distinct over a window
  23. Spark lists all leaf node even in partitioned data
  24. How can I access python variable in Spark SQL?
  25. PySpark: How to fillna values in dataframe for specific columns?
  26. How to exclude multiple columns in Spark dataframe in Python
  27. Stratified sampling with pyspark
  28. How to turn off scientific notation in pyspark?
  29. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment