_corrupt_record error when reading a JSON file into Spark

by

If you want to leave your JSON file as it is (without stripping new lines characters \n), include multiLine=True keyword argument

sc = SparkContext() 
sqlc = SQLContext(sc)

df = sqlc.read.json('my_file.json', multiLine=True)

print df.show()

More Related Contents:

  1. How to add a constant column in a Spark DataFrame?
  2. How to extract multiple JSON objects from one file?
  3. Convert Pandas Dataframe to nested JSON
  4. Spark Dataframe distinguish columns with duplicated name
  5. Pyspark: Split multiple array columns into rows
  6. How do I add a new column to a Spark DataFrame (using PySpark)?
  7. How to change a dataframe column from String type to Double type in PySpark?
  8. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  9. Python – How to convert JSON File to Dataframe
  10. Retrieve top n in each group of a DataFrame in pyspark
  11. How to flatten a pandas dataframe with some columns as json?
  12. Rename nested field in spark dataframe
  13. Filtering a Pyspark DataFrame with SQL-like IN clause
  14. Updating a dataframe column in spark
  15. Dividing complex rows of dataframe to simple rows in Pyspark
  16. Filter Pyspark dataframe column with None value
  17. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  18. How to explode multiple columns of a dataframe in pyspark
  19. Spark add new column to dataframe with value from previous row
  20. PySpark: multiple conditions in when clause
  21. Pyspark: Replacing value in a column by searching a dictionary
  22. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  23. Pivot String column on Pyspark Dataframe
  24. Pyspark: Parse a column of json strings
  25. FastAPI is very slow in returning a large amount of JSON data
  26. Calculating the cosine similarity between all the rows of a dataframe in pyspark
  27. How to make JSON flattening memory efficient?
  28. Filtering DataFrame using the length of a column
  29. Multiple condition filter on dataframe
  30. How to return a “Tuple type” in a UDF in PySpark?

Leave a Comment