Go http.Get, concurrency, and “Connection reset by peer”

by

The message connection reset by peer indicates that the remote server sent an RST to forcefully close the connection, either deliberately as a mechanism to limit connections, or as a result of a lack of resources. Either way you are likely opening too many connections, or reconnecting too fast.

Starting 1000-2000 connections in parallel is rarely the most efficient way to download that many pages, especially if most or all are coming from a single server. If you test the throughput you will find an optimal concurrency level that is far lower.

You will also want to set the Transport.MaxIdleConnsPerHost to match your level of concurrency. If MaxIdleConnsPerHost is lower than the expected number of concurrent connections, the server connections will often be closed after a request, only to be immediately opened again — this will slow your progress significantly and possibly reach connection limits imposed by the server.

More Related Contents:

  1. How to idiomatically log auth information with middleware
  2. Why do I need to use http.StripPrefix to access my static files?
  3. Golang read request body multiple times [duplicate]
  4. Correct way of getting Client’s IP Addresses from http.Request
  5. Why is my Hello World go server getting crushed by ApacheBench?
  6. Process Management for the Go Webserver
  7. Setting HTTP headers
  8. How can I start the browser AFTER the server started listening?
  9. Go client program generates a lot a sockets in TIME_WAIT state
  10. How to serve http partial content with Go?
  11. Why this simple web server is called even number times?
  12. Make a URL-encoded POST request using `http.NewRequest(…)`
  13. Do we need to close the response object if an error occurs while calling http.Get(url)?
  14. What is the difference between POST and PUT in HTTP?
  15. How long do browsers cache HTTP 301s?
  16. REST HTTP status codes for failed validation or invalid duplicate
  17. How to $http Synchronous call with AngularJS
  18. Why is it said that “HTTP is a stateless protocol”?
  19. What is the difference between HTTP status code 200 (cache) vs status code 304?
  20. Why shouldn’t data be modified on an HTTP GET request?
  21. How to use angular2 http API for tracking upload/download progress
  22. Why can’t I set a cookie and redirect?
  23. What is the function of the “Vary: Accept” HTTP header?
  24. Doing a HTTP PUT from a browser
  25. angular — accessing data of multiple http calls – how to resolve the promises
  26. Compressing HTTP Post Data sent from browser
  27. Escaping username characters in basic auth URLs
  28. Can I stream a file upload to S3 without a content-length header?
  29. How to evaluate http response codes from bash/shell script?
  30. How do you add query parameters to a Dart http request?

Leave a Comment