Split 1 column into 3 columns in spark scala

by

Try:

import sparkObject.spark.implicits._
import org.apache.spark.sql.functions.split

df.withColumn("_tmp", split($"columnToSplit", "\\.")).select(
  $"_tmp".getItem(0).as("col1"),
  $"_tmp".getItem(1).as("col2"),
  $"_tmp".getItem(2).as("col3")
)

The important point to note here is that the sparkObject is the SparkSession object you might have already initialized. So, the (1) import statement has to be compulsorily put inline within the code, not before the class definition.

More Related Contents:

  1. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  2. How to define schema for custom type in Spark SQL?
  3. Specifying the filename when saving a DataFrame as a CSV [duplicate]
  4. How can I change column types in Spark SQL’s DataFrame?
  5. NullPointerException in Scala Spark, appears to be caused be collection type?
  6. how to make saveAsTextFile NOT split output into multiple file?
  7. Dropping a nested column from Spark DataFrame
  8. Spark losing println() on stdout
  9. Spark DataFrame: does groupBy after orderBy maintain that order?
  10. Filling gaps in timeseries Spark
  11. Spark UDAF with ArrayType as bufferSchema performance issues
  12. Apache Spark how to append new column from list/array to Spark dataframe
  13. java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ while running TwitterPopularTags
  14. How to get ID of a map task in Spark?
  15. Spark: produce RDD[(X, X)] of all possible combinations from RDD[X]
  16. Encode an ADT / sealed trait hierarchy into Spark DataSet column
  17. Spark: Add column to dataframe conditionally
  18. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  19. Replace null values in Spark DataFrame
  20. Spark RDD default number of partitions
  21. Why does Spark RDD partition has 2GB limit for HDFS?
  22. Calculate Cosine Similarity Spark Dataframe
  23. Spark : how to run spark file from spark shell
  24. How can I connect to a postgreSQL database into Apache Spark using scala?
  25. Spark Dataframe Random UUID changes after every transformation/action
  26. Spark column string replace when present in other column (row)
  27. How to vectorize DataFrame columns for ML algorithms?
  28. How to set hadoop configuration values from pyspark
  29. Spark DataFrames when udf functions do not accept large enough input variables
  30. Filter spark DataFrame on string contains

Leave a Comment