How to insert big data on the laravel?

by

As it was stated, chunks won’t really help you in this case if it is a time execution problem. I think that bulk insert you are trying to use cannot handle that amount of data , so I see 2 options:

1 – Reorganise your code to properly use chunks, this will look something like this:

$insert_data = [];

foreach ($json['value'] as $value) {
    $posting_date = Carbon::parse($value['Posting_Date']);

    $posting_date = $posting_date->format('Y-m-d');

    $data = [
        'item_no'                   => $value['Item_No'],
        'entry_no'                  => $value['Entry_No'], 
        'document_no'               => $value['Document_No'],
        'posting_date'              => $posting_date,
        ....
    ];

    $insert_data[] = $data;
}

$insert_data = collect($insert_data); // Make a collection to use the chunk method

// it will chunk the dataset in smaller collections containing 500 values each. 
// Play with the value to get best result
$chunks = $insert_data->chunk(500);

foreach ($chunks as $chunk)
{
   \DB::table('items_details')->insert($chunk->toArray());
}

This way your bulk insert will contain less data, and be able to process it in a rather quick way.

2 – In case your host supports runtime overloads, you can add a directive right before the code starts to execute : 

ini_set('max_execution_time', 120 ) ; // time in seconds

$insert_data = [];

foreach ($json['value'] as $value)
{
   ...
}

To read more go to the official docs

More Related Contents:

  1. Can Anyone Explain Laravel 5.2 Multi Auth with Example
  2. Laravel 5.6 – Passport JWT httponly cookie SPA authentication for self consuming API?
  3. How to protect image from public view in Laravel 5?
  4. Laravel – Route::resource vs Route::controller
  5. How can I convert many statement mysql to laravel eloquent?
  6. Laravel Redirect Back with() Message
  7. Persisting sessions across subdomains in Laravel 5
  8. How do I catch exceptions / missing pages in Laravel 5?
  9. Undefined variable: errors in Laravel
  10. Laravel Session always changes every refresh / request in Laravel 5.4
  11. Class ‘Illuminate\Html\HtmlServiceProvider’ not found Laravel 5
  12. Rollback one specific migration in Laravel
  13. Laravel Migration Change to Make a Column Nullable
  14. How to compare two Carbon Timestamps?
  15. Laravel 5.4^ – How to customize notification email layout?
  16. Laravel 5 – Manual pagination
  17. What is difference between use env(‘APP_ENV’), config(‘app.env’) or App::environment() to get app environment?
  18. What is the difference between queue:work –daemon and queue:listen
  19. How to select specific columns in laravel eloquent
  20. 404 Not Found, but route exist in Laravel 5.4
  21. “Please provide a valid cache path” error in laravel
  22. Including a css file in a blade template?
  23. Laravel 5 – artisan seed [ReflectionException] Class SongsTableSeeder does not exist
  24. There is no existing directory at /storage/logs and its not buildable: Permission denied
  25. I’m getting error “Class ‘Predis\Client’ not found” in Laravel 5.2
  26. Has many through many-to-many
  27. How to pass multiple parameters to middleware with OR condition in Laravel 5.2
  28. Modify existing Authorization module (email to username)
  29. Laravel: how to set date format on model attribute casting?
  30. Laravel: How can I change the default Auth Password field name?

Leave a Comment