Use binary COPY table FROM with psycopg2

Here is the binary equivalent of COPY FROM for Python 3: from io import BytesIO from struct import pack import psycopg2 # Two rows of data; “id” is not in the upstream data source # Columns: node, ts, val1, val2 data = [(23253, 342, -15.336734, 2494627.949375), (23256, 348, 43.23524, 2494827.949375)] conn = psycopg2.connect(“dbname=mydb user=postgres”) curs … Read more

Performance of bcp/BULK INSERT vs. Table-Valued Parameters

I don’t really have experience with TVP yet, however there is an nice performance comparison chart vs. BULK INSERT in MSDN here. They say that BULK INSERT has higher startup cost, but is faster thereafter. In a remote client scenario they draw the line at around 1000 rows (for “simple” server logic). Judging from their … Read more

Bulk insert using stored procedure

There’s nothing wrong with your stored procedure code – the point is: the BULK INSERT command cannot accept a file name as a variable. This does work: BULK INSERT ZIPCodes FROM ‘e:\5-digit Commercial.csv’ WITH but this never works – within a stored proc or not: DECLARE @filename VARCHAR(255) SET @filename=”e:\5-digit Commercial.csv” BULK INSERT ZIPCodes FROM … Read more

How can I get a trigger to fire on each inserted row during an INSERT INTO Table (etc) SELECT * FROM Table2?

The insert trigger is called once for bulk inserts, but on the trigger you can use the special inserted table to get all the inserted rows. So, imagine you have an insert trigger like this one, that logs all the rows inserted into table create trigger trgInsertTable on dbo.table for insert as insert tableLog(name) select … Read more

Slow bulk insert for table with many indexes

You can disable and enable the indexes. Note that disabling them can have unwanted side-effects (such as having duplicate primary keys or unique indices etc.) which will only be found when re-enabling the indexes. –Disable Index ALTER INDEX [IXYourIndex] ON YourTable DISABLE GO –Enable Index ALTER INDEX [IXYourIndex] ON YourTable REBUILD GO