If you have CPU bound tasks, as you increase the number of threads you get increasing overhead and slower performances. Note: having more threads than waiting tasks is just a waste of resources, but may not slow down the tasks so much. I would use a multiple (e.g. 1 or 2) of the number of … Read more
I used to work on an app with long running threads. We do this at shutdown, BlockingQueue<Runnable> queue = threadPool.getQueue(); List<Runnable> list = new ArrayList<Runnable>(); int tasks = queue.drainTo(list); The list is saved to a file. On startup, the list is added back to the pool so we don’t lose any jobs.
Actually, in the current implementation of Spark Streaming and under default configuration, only job is active (i.e. under execution) at any point of time. So if one batch’s processing takes longer than 10 seconds, then then next batch’s jobs will stay queued. This can be changed with an experimental Spark property “spark.streaming.concurrentJobs” which is by … Read more
In some very narrow circumstances, you can implement a java.util.concurrent.RejectedExecutionHandler that does what you need. RejectedExecutionHandler block = new RejectedExecutionHandler() { rejectedExecution(Runnable r, ThreadPoolExecutor executor) { executor.getQueue().put( r ); } }; ThreadPoolExecutor pool = new … pool.setRejectedExecutionHandler(block); Now. This is a very bad idea for the following reasons It’s prone to deadlock because all the … Read more
In this case, there is no way to cancel the Future once it has actually started running, because you’re relying on the behavior of concurrent.futures.Future, and its docs state the following: cancel() Attempt to cancel the call. If the call is currently being executed and cannot be cancelled then the method will return False, otherwise … Read more
It’s pretty simple. The idea is that you have an executor object that’s a bean, which is passed into whatever object wants to fire the new task (in a new thread). The nice thing is that you can modify what type of task executor to use just by changing the Spring config. In the example … Read more
To give some history, Executors were only added as part of the java standard in Java 1.5. So in some ways Executors can be seen as a new better abstraction for dealing with Runnable tasks. A bit of an over-simplification coming… – Executors are threads done right so use them in preference.
In some very narrow circumstances, you can implement a java.util.concurrent.RejectedExecutionHandler that does what you need. RejectedExecutionHandler block = new RejectedExecutionHandler() { rejectedExecution(Runnable r, ThreadPoolExecutor executor) { executor.getQueue().put( r ); } }; ThreadPoolExecutor pool = new … pool.setRejectedExecutionHandler(block); Now. This is a very bad idea for the following reasons It’s prone to deadlock because all the … Read more
Currently the only concrete implementations of the Executor interface are the ThreadPoolExecutor and the ScheduledThreadpoolExecutor Instead of using the utility / factory class Executors, you should create an instance using a constructor. You can pass a BlockingQueue to the constructors of the ThreadPoolExecutor. One of the implementations of the BlockingQueue, the PriorityBlockingQueue lets you pass … Read more
One of the possible solutions I’ve just found: public class BoundedExecutor { private final Executor exec; private final Semaphore semaphore; public BoundedExecutor(Executor exec, int bound) { this.exec = exec; this.semaphore = new Semaphore(bound); } public void submitTask(final Runnable command) throws InterruptedException, RejectedExecutionException { semaphore.acquire(); try { exec.execute(new Runnable() { public void run() { try { … Read more