What is the meaning of partitionColumn, lowerBound, upperBound, numPartitions parameters?

by

Actually the list above misses a couple of things, specifically the first and the last query.

Without them you would loose some data (the data before the lowerBound and that after upperBound). From the example is not clear because the lower bound is 0.

The complete list should be: 

SELECT * FROM table WHERE partitionColumn < 100

SELECT * FROM table WHERE partitionColumn BETWEEN 0 AND 100  
SELECT * FROM table WHERE partitionColumn BETWEEN 100 AND 200

…

SELECT * FROM table WHERE partitionColumn > 9000

More Related Contents:

  1. Partitioning in spark while reading from RDBMS via JDBC
  2. How to optimize partitioning when migrating data from JDBC source?
  3. Spark SQL: apply aggregate functions to a list of columns
  4. Unpivot in spark-sql/pyspark
  5. Spark SQL replacement for MySQL’s GROUP_CONCAT aggregate function
  6. DataFrame join optimization – Broadcast Hash Join
  7. Overwrite specific partitions in spark dataframe write method
  8. Avoid performance impact of a single partition mode in Spark window functions
  9. Convert date from String to Date format in Dataframes
  10. How to connect to remote hive server from spark [duplicate]
  11. Pyspark : forward fill with last observation for a DataFrame
  12. How to use SQL query to define table in dbtable?
  13. Why does format(“kafka”) fail with “Failed to find data source: kafka.” (even with uber-jar)?
  14. How to loop through each row of dataFrame in pyspark
  15. multiple conditions for filter in spark data frames
  16. How to group by common element in array?
  17. How do I add an persistent column of row ids to Spark DataFrame?
  18. SPARK SQL – update MySql table using DataFrames and JDBC
  19. Convert null values to empty array in Spark DataFrame
  20. DataFrame partitionBy to a single Parquet file (per partition)
  21. Temp table caching with spark-sql
  22. How can I access python variable in Spark SQL?
  23. Why spark application fail with “executor.CoarseGrainedExecutorBackend: Driver Disassociated”?
  24. Why is predicate pushdown not used in typed Dataset API (vs untyped DataFrame API)?
  25. Convert between spark.SQL DataFrame and pandas DataFrame [duplicate]
  26. How to turn off scientific notation in pyspark?
  27. What does “Correlated scalar subqueries must be Aggregated” mean?
  28. How to improve broadcast Join speed with between condition in Spark
  29. Spark SQL referencing attributes of UDT
  30. What is an optimized way of joining large tables in Spark SQL

Leave a Comment