using sqlalchemy to load csv file into a database

by

In case your CSV is quite large, using INSERTS is very ineffective. You should use a bulk loading mechanisms, which differ from base to base. E.g. in PostgreSQL you should use “COPY FROM” method:

with open(csv_file_path, 'r') as f:    
    conn = create_engine('postgresql+psycopg2://...').raw_connection()
    cursor = conn.cursor()
    cmd = 'COPY tbl_name(col1, col2, col3) FROM STDIN WITH (FORMAT CSV, HEADER FALSE)'
    cursor.copy_expert(cmd, f)
    conn.commit()

More Related Contents:

  1. SQLAlchemy: cascade delete
  2. Bulk insert with SQLAlchemy ORM
  3. How can I verify Column data types in the SQLAlchemy ORM?
  4. SQLAlchemy: how to filter date field?
  5. Getting random row through SQLAlchemy
  6. Select NULL Values in SQLAlchemy
  7. Variable table name in sqlite
  8. jsonify a SQLAlchemy result set in Flask [duplicate]
  9. Using OR in SQLAlchemy
  10. SQLAlchemy: unexpected results when using `and` and `or`
  11. sqlalchemy: how to join several tables by one query?
  12. How to create an SQL View with SQLAlchemy?
  13. Serve image stored in SQLAlchemy LargeBinary column
  14. When do I need to use sqlalchemy back_populates?
  15. SQLAlchemy ManyToMany secondary table with additional fields
  16. Can SQLAlchemy be used with Google Cloud SQL?
  17. Updates to JSON field don’t persist to DB
  18. ImportError: No module named MySQLdb
  19. How do I select literal values in an sqlalchemy query?
  20. Get inserted key before commit session
  21. How to move a model between two Django apps (Django 1.7)
  22. SQLAlchemy multiple foreign keys in one mapped class to the same primary key
  23. SQLite Performance Benchmark — why is :memory: so slow…only 1.5X as fast as disk?
  24. SQLAlchemy default DateTime
  25. Setting SQLAlchemy autoincrement start value
  26. How to close a SQLAlchemy session?
  27. sqlalchemy existing database query
  28. Using SQLAlchemy models in and out of Flask
  29. django makemigrations and migrate on heroku server don’t create tables
  30. SQLAlchemy ordering by count on a many to many relationship

Leave a Comment