Transpose column to row with Spark

It is relatively simple to do with basic Spark SQL functions. Python from pyspark.sql.functions import array, col, explode, struct, lit df = sc.parallelize([(1, 0.0, 0.6), (1, 0.6, 0.7)]).toDF([“A”, “col_1”, “col_2”]) def to_long(df, by): # Filter dtypes and split into column names and type description cols, dtypes = zip(*((c, t) for (c, t) in df.dtypes if … Read more

Transposing a 1D NumPy array

It’s working exactly as it’s supposed to. The transpose of a 1D array is still a 1D array! (If you’re used to matlab, it fundamentally doesn’t have a concept of a 1D array. Matlab’s “1D” arrays are 2D.) If you want to turn your 1D vector into a 2D array and then transpose it, just … Read more