Script to get the HTTP status code of a list of urls?

by

Curl has a specific option, --write-out, for this:

$ curl -o /dev/null --silent --head --write-out '%{http_code}\n' <url>
200
  • -o /dev/null throws away the usual output
  • --silent throws away the progress meter
  • --head makes a HEAD HTTP request, instead of GET
  • --write-out '%{http_code}\n' prints the required status code

To wrap this up in a complete Bash script:

#!/bin/bash
while read LINE; do
  curl -o /dev/null --silent --head --write-out "%{http_code} $LINE\n" "$LINE"
done < url-list.txt

(Eagle-eyed readers will notice that this uses one curl process per URL, which imposes fork and TCP connection penalties. It would be faster if multiple URLs were combined in a single curl, but there isn’t space to write out the monsterous repetition of options that curl requires to do this.)

More Related Contents:

  1. How to urlencode data for curl command?
  2. How to properly handle a gzipped page when using curl?
  3. How to include an ‘&’ character in a bash curl statement
  4. Percent sign % not working in crontab
  5. curl: argument list too long
  6. Save file to specific folder with curl command
  7. Assign output to variable in Bash [duplicate]
  8. Can I export a variable to the environment from a Bash script without sourcing it?
  9. Passing arguments to an interactive program non-interactively
  10. How to execute a bash command stored as a string with quotes and asterisk [duplicate]
  11. How to define hash tables in Bash?
  12. Why isn’t tilde (~) expanding inside double quotes? [duplicate]
  13. sed search and replace strings containing / [duplicate]
  14. What is the difference between “$@” and “$*” in Bash? [duplicate]
  15. Count occurrences of a char in a string using Bash
  16. How to redirect output of an entire shell script within the script itself?
  17. Getting “command not found” error in bash script
  18. What is the exact meaning of IFS=$’\n’?
  19. How to check if running in Cygwin, Mac or Linux?
  20. Using getopts inside a Bash function
  21. choosing between $0 and BASH_SOURCE
  22. What does ${ARGUMENT+x} mean in Bash? [duplicate]
  23. What is
  24. How to write a bash script to set global environment variable?
  25. Hiding secret from command line parameter on Unix
  26. Bash Script Binary Search
  27. Escaping forward slashes in sed command [duplicate]
  28. Convert a string into an array with bash, honoring quotes for grouping [duplicate]
  29. Remove all files except some from a directory
  30. Groups of compound conditions in Bash test [duplicate]

Leave a Comment