Provide small sample data, that can be easily recreated. At the very least, posters should provide a couple of rows and columns on their dataframe and code that can be used to easily create it. By easy, I mean cut and paste. Make it as small as possible to demonstrate your problem. I have the … Read more
As mentioned by PinoSan this question is probably too generic, and you should be able to find your answer in any Spark Getting Started, or Tutorial. Let me point you to some interesting content: Spark Quick Start Guide Getting Started with Apache Spark, ebook Introduction to Apache Spark with Examples and Use Cases Disclaimer: I … Read more
If you are going to make all the groupBy with the same columns, and you want to create all the aggregation column references as avg(col_interested ).as(col_interested_avg) with all the elements of columns_interestedList you can create all the references with the stream, and pass them to the gag method. List<Column> avgCols = columns_interestedList.stream() .map(col_interested -> avg(col_interested).as(col_interested … Read more
You can use the scala Bucketizer. There’s a good example here: https://spark.apache.org/docs/2.2.0/ml-features.html#bucketizer After you use the bucketizer you have a dataframe with a bucket index (i.e index 1, 2, and 3 might correspond to values 1-5, 6-10, 11-15, respectively). You can do a .groupBy and .agg (or use SQL) to get a count of records … Read more
Load the data into spark and apply group by on the email column.. after that check for bag and apply any distance algorithm on the first name and last name columns. This should be pretty straight forward in spark val df = sc.textFile(“hdfs path of data”); df.mapToPair(“email”, <whole_record>) .groupBy(//will be done based on key) .map(//will … Read more
I know you write that no aggregation is (should be?) done, but I’d wager that is precisely what you need and want to do. The point of distributed computing is largely that partial results are computed, well, distributed at each node. For very big data sets, each node (often) sees only a subset of the … Read more
You have an extrat upper case try this : rddUnion.foreach(println)
Assume – for example – that these are coordinates. Are (x,y) and (y,x) the same coordinates? Certainly not! Therefore, mapreduce must not assume that the order of a tuple is irrelevant by default. (That does not say it can’t be done, just that the system must not assume this as default behavior) If you want … Read more
Join both datasets based on company_id and select all columns from second dataset. Code should look something as below: (Not Tested) Dataset<Row> finalDf = firstDataset.join(secondDataset ,firstDataset.col(“companyId”).equalTo(secondDataset.col(“companyid”), “inner”).select(secondDataset .col(“*)) finalDF.show()
What (probably) takes the time here is shuffling the data: when you want to group two or more records together, they must reside within the same partition, so Spark has to first shuffle the records so that all records with same key are in a single partition. Now, even if each key has two records … Read more