dataframe

Writing UDF for looks up in the Map in java giving Unsupported literal type class java.util.HashMap

You can pass the look up map or array etc. to the udf by using partial. check out this example. from functools import partial from pyspark.sql.functions import udf fruit_dict = {“O”: “Orange”, “A”: “Apple”, “G”: “Grape”} df = spark.createDataFrame([(“A”, 20), (“G”, 30), (“O”, 10)], [“Label”, “Count”]) def decipher_fruit(label, fruit_map): label_names = list(fruit_map.keys()) if label in … Read more

pySpark mapping multiple columns

From my understanding, you can create a map based on columns from reference_df (I assumed this is not a very big dataframe): map_key = concat_ws(‘\0’, PrimaryLookupAttributeName, PrimaryLookupAttributeValue) map_value = OutputItemNameByValue and then use this mapping to get the corresponding values in df1: from itertools import chain from pyspark.sql.functions import collect_set, array, concat_ws, lit, col, create_map … Read more

Filter Spark DataFrame based on another DataFrame that specifies denylist criteria

You’ll need to use a left_anti join in this case. The left anti join is the opposite of a left semi join. It filters out data from the right table in the left table according to a given key : largeDataFrame .join(smallDataFrame, Seq(“some_identifier”),”left_anti”) .show // +—————+———-+ // |some_identifier|first_name| // +—————+———-+ // | 222| mary| // … Read more

Difference between DataFrame, Dataset, and RDD in Spark

A DataFrame is defined well with a google search for “DataFrame definition”: A data frame is a table, or two-dimensional array-like structure, in which each column contains measurements on one variable, and each row contains one case. So, a DataFrame has additional metadata due to its tabular format, which allows Spark to run certain optimizations … Read more

How to create an empty DataFrame with a specified schema?

Lets assume you want a data frame with the following schema: root |– k: string (nullable = true) |– v: integer (nullable = false) You simply define schema for a data frame and use empty RDD[Row]: import org.apache.spark.sql.types.{ StructType, StructField, StringType, IntegerType} import org.apache.spark.sql.Row val schema = StructType( StructField(“k”, StringType, true) :: StructField(“v”, IntegerType, false) … Read more

Difference between DataFrame, Dataset, and RDD in Spark

A DataFrame is defined well with a google search for “DataFrame definition”: A data frame is a table, or two-dimensional array-like structure, in which each column contains measurements on one variable, and each row contains one case. So, a DataFrame has additional metadata due to its tabular format, which allows Spark to run certain optimizations … Read more

How to pivot Spark DataFrame?

As mentioned by David Anderson Spark provides pivot function since version 1.6. General syntax looks as follows: df .groupBy(grouping_columns) .pivot(pivot_column, [values]) .agg(aggregate_expressions) Usage examples using nycflights13 and csv format: Python: from pyspark.sql.functions import avg flights = (sqlContext .read .format(“csv”) .options(inferSchema=”true”, header=”true”) .load(“flights.csv”) .na.drop()) flights.registerTempTable(“flights”) sqlContext.cacheTable(“flights”) gexprs = (“origin”, “dest”, “carrier”) aggexpr = avg(“arr_delay”) flights.count() ## … Read more