While writing to hdfs path getting error java.io.IOException: Failed to rename

by

You can do all the selects in one single job, get all the selects and union in a single table.

Dataset<Row> resultDs = givenItemList.parallelStream().map( item -> {   
    String query = "select $item  as itemCol , avg($item) as mean groupBy year";
    return sparkSession.sql(query);
}).reduce((a, b) -> a.union(b)).get

saveDsToHdfs(hdfsPath, resultDs );

More Related Contents:

  1. Spark – load CSV file as DataFrame?
  2. Finding duplicates from large data set using Apache Spark
  3. Using a column value as a parameter to a spark DataFrame function
  4. Find maximum row per group in Spark DataFrame
  5. Split Spark Dataframe string column into multiple columns
  6. Multiple Aggregate operations on the same column of a spark dataframe
  7. Partitioning in spark while reading from RDBMS via JDBC
  8. How to control partition size in Spark SQL
  9. How to import multiple csv files in a single load?
  10. How does Hadoop Namenode failover process works?
  11. How to access element of a VectorUDT column in a Spark DataFrame?
  12. Requirements for converting Spark dataframe to Pandas/R dataframe
  13. Spark DataFrame: count distinct values of every column
  14. How to split a list to multiple columns in Pyspark?
  15. pyspark dataframe filter or include based on list
  16. Use collect_list and collect_set in Spark SQL
  17. What is the maximum size for a broadcast object in Spark?
  18. How to find mean of grouped Vector columns in Spark SQL?
  19. Generate a Spark StructType / Schema from a case class
  20. How does createOrReplaceTempView work in Spark?
  21. PySpark – get row number for each row in a group
  22. Keep only duplicates from a DataFrame regarding some field
  23. Spark iterate HDFS directory
  24. pyspark: count distinct over a window
  25. Using windowing functions in Spark
  26. PySpark: How to fillna values in dataframe for specific columns?
  27. How to calculate Median in spark sqlContext for column of data type double
  28. Spark: disk I/O on stage boundaries explanation
  29. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment