It depends on your workload By default Dask creates a single process with as many threads as you have logical cores on your machine (as determined by multiprocessing.cpu_count()). dask-worker … –nprocs 1 –nthreads 8 # assuming you have eight cores dask-worker … # this is actually the default setting Using few processes and many threads … Read more
If you want a very easy solution, there isn’t one. However, there is a solution that has the multiprocessing interface — pathos — which has the ability to establish connections to remote servers through a parallel map, and to do multiprocessing. If you want to have a ssh-tunneled connection, you can do that… or if … Read more
assuming that a worker wants to send 4G of data to the driver, then having spark.driver.maxResultSize=1G, will cause the worker to send 4 messages (instead of 1 with unlimited spark.driver.maxResultSize). No. If estimated size of the data is larger than maxResultSize given job will be aborted. The goal here is to protect your application from … Read more
The parameter mapred.max.split.size which can be set per job individually is what you looking for. Don’t change dfs.block.size because this is global for HDFS and can lead to problems.
In my opinion the most elegant solution is to star expand a Struct using a select operator as shown below: var explodedDf2 = explodedDf.select(“department.*”,”*”) https://docs.databricks.com/spark/latest/spark-sql/complex-types.html
It is very easy to misunderstand the CAP properties, hence I’m providing some illustrations to make it easier. Consistency: A query Q will produce the same answer A regardless the node that handles the request. In order to guarantee full consistency we need to ensure that all nodes agree on the same value at all … Read more
Actually having actually tried it, I think the link I posted as a comment doesn’t do exactly what you want with dependencies. What you are quite reasonably asking for is a way to have Spark play nicely with setuptools and pip regarding installing dependencies. It blows my mind that this isn’t supported better in Spark. … Read more
With cache(), you use only the default storage level : MEMORY_ONLY for RDD MEMORY_AND_DISK for Dataset With persist(), you can specify which storage level you want for both RDD and Dataset. From the official docs: You can mark an RDD to be persisted using the persist() or cache() methods on it. each persisted RDD can … Read more
aggregateByKey() is quite different from reduceByKey. What happens is that reduceByKey is sort of a particular case of aggregateByKey. aggregateByKey() will combine the values for a particular key, and the result of such combination can be any object that you specify. You have to specify how the values are combined (“added”) inside one partition (that … Read more
Yes, a spark application has one and only Driver. What is the relationship between numWorkerNodes and numExecutors? A worker can host multiple executors, you can think of it like the worker to be the machine/node of your cluster and the executor to be a process (executing in a core) that runs on that worker. So … Read more