Python Postgres psycopg2 ThreadedConnectionPool exhausted

by

I’ve struggled to find really detailed information on how the ThreadedConnectionPool works. https://bbengfort.github.io/observations/2017/12/06/psycopg2-transactions.html ain’t bad, but it turns out that its claim that getconn blocks until a connection becomes available is incorrect. Checking the code, all ThreadedConnectionPool adds is a lock around the AbstractConnectionPool methods to prevent race conditions. If more than maxconn connections are attempted used at any point, the connection pool exhausted PoolError will be raised.

If you want something a bit simpler than the accepted answer, further wrapping the methods in a Semaphore providing the blocking until a connection becomes available should do the trick:

from psycopg2.pool import ThreadedConnectionPool
from threading import Semaphore

class ReallyThreadedConnectionPool(ThreadedConnectionPool):
    def __init__(self, minconn, maxconn, *args, **kwargs):
        self._semaphore = Semaphore(maxconn)
        super().__init__(minconn, maxconn, *args, **kwargs)

    def getconn(self, *args, **kwargs):
        self._semaphore.acquire()
        return super().getconn(*args, **kwargs)

    def putconn(self, *args, **kwargs):
        super().putconn(*args, **kwargs)
        self._semaphore.release()

More Related Contents:

  1. psycopg2: insert multiple rows with one query
  2. Parameterized queries with psycopg2 / Python DB-API and PostgreSQL
  3. Passing table name as a parameter in psycopg2
  4. Executing multiple statements with Postgresql via SQLAlchemy does not persist changes
  5. How to create an SQL View with SQLAlchemy?
  6. Use Django ORM as standalone [duplicate]
  7. Django connection to postgres by docker-compose
  8. How to set up a PostgreSQL database in Django
  9. Import psycopg2 Library not loaded: libssl.1.0.0.dylib
  10. Composite primary key in django
  11. Psycopg2 image not found
  12. How do you join two tables on a foreign key field using django ORM?
  13. python pip install psycopg2 install error
  14. sqlalchemy.exc.NoSuchModuleError: Can’t load plugin: sqlalchemy.dialects:postgres
  15. Create a Postgres database using python
  16. SQLAlchemy – performing a bulk upsert (if exists, update, else insert) in postgresql
  17. Strange SQLAlchemy error message: TypeError: ‘dict’ object does not support indexing
  18. Creating partial unique index with sqlalchemy on Postgres
  19. Filling Many2many field (odoo 8)
  20. Django ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
  21. SQLAlchemy func.count on boolean column
  22. Django-DB-Migrations: cannot ALTER TABLE because it has pending trigger events
  23. Python/postgres/psycopg2: getting ID of row just inserted
  24. How to set connection timeout in SQLAlchemy
  25. Deploying Django to Heroku (Psycopg2 Error)
  26. “Failed building wheel for psycopg2” – MacOSX using virtualenv and pip
  27. How to perform filtering with a Django JSONField?
  28. Psycopg2, Postgresql, Python: Fastest way to bulk-insert
  29. Insert Python Dictionary using Psycopg2
  30. How Postgresql COPY TO STDIN With CSV do on conflic do update?

Leave a Comment