pyspark parse fixed width text file

by

Spark’s substr function can handle fixed-width columns, for example:

df = spark.read.text("/tmp/sample.txt")
df.select(
    df.value.substr(1,3).alias('id'),
    df.value.substr(4,8).alias('date'),
    df.value.substr(12,3).alias('string'),
    df.value.substr(15,4).cast('integer').alias('integer')
).show()

will result in:

+---+--------+------+-------+
| id|    date|string|integer|
+---+--------+------+-------+
|001|01292017|   you|   1234|
|002|01302017|    me|   5678|
+---+--------+------+-------+

Having splitted columns you can reformat and use them as in normal spark dataframe.

More Related Contents:

  1. Efficient string matching in Apache Spark
  2. Load CSV file with Spark
  3. How to turn off INFO logging in Spark?
  4. How do I add a new column to a Spark DataFrame (using PySpark)?
  5. How to perform union on two DataFrames with different amounts of columns in spark?
  6. Does spark predicate pushdown work with JDBC?
  7. Rename nested field in spark dataframe
  8. Passing a data frame column and external list to udf under withColumn
  9. Updating a dataframe column in spark
  10. How to run multiple jobs in one Sparkcontext from separate threads in PySpark?
  11. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  12. A list as a key for PySpark’s reduceByKey
  13. Best way to get the max value in a Spark dataframe column
  14. How to map features from the output of a VectorAssembler back to the column names in Spark ML?
  15. Pyspark replace strings in Spark dataframe column
  16. Pyspark: Replacing value in a column by searching a dictionary
  17. Apache Spark Python Cosine Similarity over DataFrames
  18. Pyspark: Parse a column of json strings
  19. Reduce a key-value pair into a key-list pair with Apache Spark
  20. GroupBy column and filter rows with maximum value in Pyspark
  21. PySpark in iPython notebook raises Py4JJavaError when using count() and first()
  22. Add Jar to standalone pyspark
  23. PySpark: match the values of a DataFrame column against another DataFrame column
  24. Spark mllib predicting weird number or NaN
  25. Multiple condition filter on dataframe
  26. How to return a “Tuple type” in a UDF in PySpark?
  27. How to connect HBase and Spark using Python?
  28. Pyspark 2.4.0, read avro from kafka with read stream – Python
  29. How to pivot on multiple columns in Spark SQL?
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment