If merge in ACID mode is not applicable, then it’s possible to update using FULL OUTER JOIN or using UNION ALL + row_number. To find all entries that will be updated you can join increment data with old data: insert overwrite target_data [partition() if applicable] SELECT –select new if exists, old if not exists case … Read more
Interesting question, I spent some time looking at the code for the details and here are my thoughts. The splits are handled by the client by InputFormat.getSplits, so a look at FileInputFormat gives the following info: For each input file, get the file length, the block size and calculate the split size as max(minSize, min(maxSize, … Read more
spark-csv is part of core Spark functionality and doesn’t require a separate library. So you could just do for example df = spark.read.format(“csv”).option(“header”, “true”).load(“csvfile.csv”) In scala,(this works for any format-in delimiter mention “,” for csv, “\t” for tsv etc) val df = sqlContext.read.format(“com.databricks.spark.csv”) .option(“delimiter”, “,”) .load(“csvfile.csv”)
If you use Spark 1.4+, this has become much, much easier thanks to the DataFrame API. (DataFrames were introduced in Spark 1.3, but partitionBy(), which we need, was introduced in 1.4.) If you’re starting out with an RDD, you’ll first need to convert it to a DataFrame: val people_rdd = sc.parallelize(Seq((1, “alice”), (1, “bob”), (2, … Read more
I assume you’re running Hadoop on 64bit CentOS. The reason you saw that warning is the native Hadoop library $HADOOP_HOME/lib/native/libhadoop.so.1.0.0 was actually compiled on 32 bit. Anyway, it’s just a warning, and won’t impact Hadoop’s functionalities. Here is the way if you do want to eliminate this warning, download the source code of Hadoop and … Read more
As mentioned by PinoSan this question is probably too generic, and you should be able to find your answer in any Spark Getting Started, or Tutorial. Let me point you to some interesting content: Spark Quick Start Guide Getting Started with Apache Spark, ebook Introduction to Apache Spark with Examples and Use Cases Disclaimer: I … Read more
I find the tutorials from hortonworks a rather good starting point http://hortonworks.com/tutorials/#tuts-developers To deep dive a must read is Tom White’s ‘Hadoop: The Definitve Guide’. ‘Hadoop in Practice’ shows a lot of cookbook like examples.