You have to go through at least one extra step in MPI to do this. The problem is that the most general of the gather/scatter routines, MPI_Scatterv and MPI_Gatherv, allow you to pass a “vector” (v) of counts/displacements, rather than just one count for Scatter and Gather, but the types are all assumed to be … Read more
The evils of too much coupling between services are far worse than the problems caused by code duplication The author is very unspecific when he uses the generic word “coupling”. I would agree with certain types of coupling being a strict no-no (like sharing databases or using internal interfaces). However the use of common libraries … Read more
When you create the SparkContext, each worker starts an executor. This is a separate process (JVM), and it loads your jar too. The executors connect back to your driver program. Now the driver can send them commands, like flatMap, map and reduceByKey in your example. When the driver quits, the executors shut down. RDDs are … Read more
I named this the Remote Session pattern in my 2001 book. The idea is to have a singleton RMI object, bound in the Registry, with nothing but a login() method. That method, if successful, returns a new RemoteSession object for every call, that contains the API you need for the session. RemoteSession is another remote … Read more
Since I don’t know what information you are looking for, I’ll try to describe how to set up a socket program and what pitfalls I’ve run into. To start with, *read the Winsock tutorial at MSDN. This is a basic program to connect, send a message and disconnect. It’s great for getting a feel for … Read more
Spark uses a master/slave architecture. As you can see in the figure, it has one central coordinator (Driver) that communicates with many distributed workers (executors). The driver and each of the executors run in their own Java processes. DRIVER The driver is the process where the main method runs. First it converts the user program … Read more
It is a bit more complex than you described. The auto.offset.reset config kicks in ONLY if your consumer group does not have a valid offset committed somewhere (2 supported offset storages now are Kafka and Zookeeper), and it also depends on what sort of consumer you use. If you use a high-level java consumer then … Read more
The inter_op_parallelism_threads and intra_op_parallelism_threads options are documented in the source of the tf.ConfigProto protocol buffer. These options configure two thread pools used by TensorFlow to parallelize execution, as the comments describe: // The execution of an individual op (for some op types) can be // parallelized on a pool of intra_op_parallelism_threads. // 0 means the … Read more
You can use explode function: scala> import org.apache.spark.sql.functions.explode import org.apache.spark.sql.functions.explode scala> val test = sqlContext.read.json(sc.parallelize(Seq(“””{“a”:1,”b”:[2,3]}”””))) test: org.apache.spark.sql.DataFrame = [a: bigint, b: array<bigint>] scala> test.printSchema root |– a: long (nullable = true) |– b: array (nullable = true) | |– element: long (containsNull = true) scala> val flattened = test.withColumn(“b”, explode($”b”)) flattened: org.apache.spark.sql.DataFrame = [a: bigint, … Read more
It avoids a full shuffle. If it’s known that the number is decreasing then the executor can safely keep data on the minimum number of partitions, only moving the data off the extra nodes, onto the nodes that we kept. So, it would go something like this: Node 1 = 1,2,3 Node 2 = 4,5,6 … Read more