Fastest possible grep

by

Try with GNU parallel, which includes an example of how to use it with grep:

grep -r greps recursively through directories. On multicore CPUs GNU
parallel can often speed this up.

find . -type f | parallel -k -j150% -n 1000 -m grep -H -n STRING {}

This will run 1.5 job per core, and give 1000 arguments to grep.

For big files, it can split it the input in several chunks with the --pipe and --block arguments:

 parallel --pipe --block 2M grep foo < bigfile

You could also run it on several different machines through SSH (ssh-agent needed to avoid passwords):

parallel --pipe --sshlogin server.example.com,server2.example.net grep foo < bigfile

More Related Contents:

  1. Piping tail output though grep twice
  2. Waiting for background processes to finish before exiting script
  3. Some troubles with using sed and awk [closed]
  4. What does 2>&1 mean here?
  5. How can I pipe stderr, and not stdout?
  6. Can I export a variable to the environment from a Bash script without sourcing it?
  7. Bash command line and input limit
  8. Delete all but the most recent X files in bash
  9. How do I check if a directory exists in a Bash shell script?
  10. Checking from shell script if a directory contains files
  11. How to determine the current interactive shell that I’m in (command-line)
  12. How to use ‘find’ to search for files created on a specific date? [closed]
  13. Removing colors from output
  14. How to get the part of a file after the first line that matches a regular expression
  15. How to represent multiple conditions in a shell if statement?
  16. How to process each output line in a loop?
  17. How to search filenames by regex with “find”
  18. How does this bash fork bomb work? [duplicate]
  19. How to do a logical OR operation for integer comparison in shell scripting?
  20. How to go to each directory and execute a command?
  21. What’s a concise way to check that environment variables are set in a Unix shell script?
  22. executing shell command in background from script [duplicate]
  23. Variables as commands in Bash scripts
  24. How to split a file into equal parts, without breaking individual lines? [duplicate]
  25. Copy all files with a certain extension from all subdirectories
  26. How can I convert tabs to spaces in every file of a directory?
  27. File content into unix variable with newlines
  28. How to give a pattern for new line in grep?
  29. Prevent grep returning an error when input doesn’t match
  30. Left side of pipe is the subshell?

Leave a Comment