How to save/insert each DStream into a permanent table

by

Vanilla Spark does not provide a way to persist data unless you’ve downloaded the version packaged with HDFS (although they appear to be playing with the idea in Spark 2.0). One way to store the results to a permanent table and query those results later is to use one of the various databases in the Spark Database Ecosystem. There are pros and cons to each and your use case matters. I’ll provide something close to a master list. These are segmented by:

Type of data managment, form data is stored in, connection to Spark

Database, SQL, Integrated

Database, SQL, Connector

Database, NoSQL, Connector

Database, Document, Connector

Database, Graph, Connector

Search, Document, Connector

Data grid, SQL, Connector

Data grid, NoSQL, Connector

File System, Files, Integrated

  • HDFS

File System, Files, Connector

Datawarehouse, SQL, Connector

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Spark Dataframe validating column names for parquet writes
  3. How to melt Spark DataFrame?
  4. Using a column value as a parameter to a spark DataFrame function
  5. Find maximum row per group in Spark DataFrame
  6. Unpivot in spark-sql/pyspark
  7. Split Spark Dataframe string column into multiple columns
  8. Overwrite specific partitions in spark dataframe write method
  9. Avoid performance impact of a single partition mode in Spark window functions
  10. How to import multiple csv files in a single load?
  11. How to access element of a VectorUDT column in a Spark DataFrame?
  12. Python Spark Cumulative Sum by Group Using DataFrame
  13. How to split a list to multiple columns in Pyspark?
  14. pyspark dataframe filter or include based on list
  15. Spark DataFrame: does groupBy after orderBy maintain that order?
  16. Pyspark : forward fill with last observation for a DataFrame
  17. How to loop through each row of dataFrame in pyspark
  18. Fill in null with previously known good value with pyspark
  19. PySpark – get row number for each row in a group
  20. Pyspark: Pass multiple columns in UDF
  21. Apache spark dealing with case statements
  22. Rename more than one column using withColumnRenamed
  23. Keep only duplicates from a DataFrame regarding some field
  24. reading json file in pyspark
  25. pyspark: count distinct over a window
  26. How can I access python variable in Spark SQL?
  27. How to exclude multiple columns in Spark dataframe in Python
  28. Stratified sampling with pyspark
  29. How to turn off scientific notation in pyspark?
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment