Process very big csv file without timeout and memory error

by

I’ve used fgetcsv to read a 120MB csv in a stream-wise-manner (is that correct english?). That reads in line by line and then I’ve inserted every line into a database. That way only one line is hold in memory on each iteration. The script still needed 20 min. to run. Maybe I try Python next time… Don’t try to load a huge csv-file into an array, that really would consume a lot of memory.

// WDI_GDF_Data.csv (120.4MB) are the World Bank collection of development indicators:
// http://data.worldbank.org/data-catalog/world-development-indicators
if(($handle = fopen('WDI_GDF_Data.csv', 'r')) !== false)
{
    // get the first row, which contains the column-titles (if necessary)
    $header = fgetcsv($handle);

    // loop through the file line-by-line
    while(($data = fgetcsv($handle)) !== false)
    {
        // resort/rewrite data and insert into DB here
        // try to use conditions sparingly here, as those will cause slow-performance

        // I don't know if this is really necessary, but it couldn't harm;
        // see also: http://php.net/manual/en/features.gc.php
        unset($data);
    }
    fclose($handle);
}

More Related Contents:

  1. Importing CSV data using PHP/MySQL
  2. Import CSV file directly into MySQL
  3. Remove BOM () from imported .csv file
  4. Export to CSV via PHP
  5. How to extract data from csv file in PHP
  6. Create a CSV File for a user in PHP
  7. How to parse a CSV file using PHP [duplicate]
  8. Setting Curl’s Timeout in PHP
  9. How to create an array from a CSV file using PHP and the fgetcsv function
  10. How does the keyword “use” work in PHP and can I import classes with it?
  11. Copy Image from Remote Server Over HTTP
  12. Convert array into csv
  13. MySQL server has gone away – in exactly 60 seconds
  14. Dynamically display a CSV file as an HTML table on a web page
  15. PHP CSV string to array
  16. PHP sessions default timeout [duplicate]
  17. Read large data from csv file in php [duplicate]
  18. How to upload and parse a CSV file in php
  19. PHP how to import all classes from another namespace
  20. Why am I getting mime-type of .csv file as “application/octet-stream”?
  21. PHP – Import CSV file to mysql database Using LOAD DATA INFILE
  22. Creating csv file with php
  23. Create a comma-separated string from a single column of an array of objects
  24. Group subarrays by one column, make comma-separated values from other column within groups
  25. Forcing fputcsv to Use Enclosure For *all* Fields
  26. How do I get csv file to download on IE? Works on firefox
  27. Premature end of script headers: index.php, mod_fcgid: read data timeout in 61 seconds
  28. Reports in Codeigniter
  29. Write CSV To File Without Enclosures In PHP
  30. Converting HTML Table to a CSV automatically using PHP?

Leave a Comment