This error message…
java.lang.OutOfMemoryError: unable to create new native thread
…implies that JVM was unable to create any new native thread as your system have run OutOfMemory
OutOfMemoryError
Out of Memory error messages can appear when you attempt to start new programs or you try to use programs that are already running, even though you still have plenty of physical and pagefile memory available. When you run a java program or java application, os allocates some memory to the JVM. JVM divides this memory into two parts. One is the Stack Memory and another is the Heap Memory. Stack Memory is used for execution of methods while Heap Memory is used to store the objects. When the heap becomes full, JVM throws java.lang.OutOfMemoryError.
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError is thrown when the Heap Memory is full and JVM is unable to allocate the memory to new objects. As the objects you create in Java are stored in the Heap Memory, so when ever the objects are no more required, they must be removed from the memory. Garbage Collector automatically removes the unwanted objects from the Heap Memory. If your objects have live references, garbage collector doesn’t remove them. It removes only those objects which don’t have live references. in this process if at any point of time there is no space left for new objects in the heap memory then JVM will throw java.lang.OutOfMemoryError.
This usecase
As per your code trials, you have implemented Multithreading using ThreadPoolExecutor to create and use a thread for each request while opening a google-chrome-headless browsing context.
Multithreading using Selenium
As you mentioned, within 1 hour it breaks agian with the same error, I suspect this due to the fact that WebDriver is not thread-safe. Having said that, if you can serialize access to the underlying driver instance, you can share a reference in more than one thread. This is not advisable. But you can always instantiate one WebDriver instance for each thread.
Ideally the issue of thread-safety isn’t in your code but in the actual browser bindings. They all assume there will only be one command at a time (e.g. like a real user). But on the other hand you can always instantiate one WebDriver instance for each thread which will launch multiple browsing tabs/windows. Till this point it seems your program is perfect.
Now, different threads can be run on same Webdriver, but then the results of the tests would not be what you expect. The reason behind is, when you use multi-threading to run different tests on different tabs/windows a little bit of thread safety coding is required or else the actions you will perform like click() or send_keys() will go to the opened tab/window that is currently having the focus regardless of the thread you expect to be running. Which essentially means all the test will run simultaneously on the same tab/window that has focus but not on the intended tab/window.
Additional Consideration
However another issue is the incompatibility between the version of the binaries you are using as follows:
- You are using chromedriver=2.35
- Release Notes of chromedriver=2.35 clearly mentions the following :
Supports Chrome v62-64
- You are using chrome=70.0
- Release Notes of ChromeDriver vv2.44 clearly mentions the following :
Supports Chrome v69-71
- Your Selenium Client version is 3.9.1 which is almost 2 years older.
So there is a clear mismatch between Selenium Client v3.9.1 , ChromeDriver v2.35 and the Chrome Browser v70.0
Solution
Ensure that:
- JDK is upgraded to current levels JDK 8u241.
- Selenium is upgraded to current levels Version 3.141.59.
- ChromeDriver is updated to current ChromeDriver v80.0 level.
- Chrome is updated to current Chrome Version 80.0 level. (as per ChromeDriver v80.0 release notes)
- If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
- Take a System Reboot.
- Always invoke
driver.quit()withintearDown(){}method to close & destroy the WebDriver and Web Client instances gracefully.