How to split a file into equal parts, without breaking individual lines? [duplicate]

by

If you mean an equal number of lines, split has an option for this:

split --lines=75

If you need to know what that 75 should really be for N equal parts, its:

lines_per_part = int(total_lines + N - 1) / N

where total lines can be obtained with wc -l.

See the following script for an example:

#!/usr/bin/bash

# Configuration stuff

fspec=qq.c
num_files=6

# Work out lines per file.

total_lines=$(wc -l <${fspec})
((lines_per_file = (total_lines + num_files - 1) / num_files))

# Split the actual file, maintaining lines.

split --lines=${lines_per_file} ${fspec} xyzzy.

# Debug information

echo "Total lines     = ${total_lines}"
echo "Lines  per file = ${lines_per_file}"    
wc -l xyzzy.*

This outputs:

Total lines     = 70
Lines  per file = 12
  12 xyzzy.aa
  12 xyzzy.ab
  12 xyzzy.ac
  12 xyzzy.ad
  12 xyzzy.ae
  10 xyzzy.af
  70 total

More recent versions of split allow you to specify a number of CHUNKS with the -n/--number option. You can therefore use something like:

split --number=l/6 ${fspec} xyzzy.

(that’s ell-slash-six, meaning lines, not one-slash-six).

That will give you roughly equal files in terms of size, with no mid-line splits.

I mention that last point because it doesn’t give you roughly the same number of lines in each file, more the same number of characters.

So, if you have one 20-character line and 19 1-character lines (twenty lines in total) and split to five files, you most likely won’t get four lines in every file.

More Related Contents:

  1. What does 2>&1 mean here?
  2. In the shell, what does ” 2>&1 ” mean?
  3. How do I split a string on a delimiter in Bash?
  4. Can I export a variable to the environment from a Bash script without sourcing it?
  5. Bash tool to get nth line from a file
  6. Bash command line and input limit
  7. Which characters need to be escaped when using Bash?
  8. How to run the sftp command with a password from Bash script?
  9. “~/Desktop/test.txt: No such file or directory”
  10. How do I check if a directory exists in a Bash shell script?
  11. Checking from shell script if a directory contains files
  12. How to determine the current interactive shell that I’m in (command-line)
  13. How to represent multiple conditions in a shell if statement?
  14. How to split one string into multiple strings separated by at least one space in bash shell?
  15. Delete files older than 10 days using shell script in Unix [duplicate]
  16. “Invalid Arithmetic Operator” when doing floating-point math in bash
  17. Piping tail output though grep twice
  18. How to go to each directory and execute a command?
  19. Is there a way to make bash job control quiet?
  20. What’s a concise way to check that environment variables are set in a Unix shell script?
  21. executing shell command in background from script [duplicate]
  22. Waiting for background processes to finish before exiting script
  23. Variables as commands in Bash scripts
  24. How can I convert tabs to spaces in every file of a directory?
  25. Convert Unix timestamp to a date string
  26. Using Bash to display a progress indicator (spinner) [duplicate]
  27. Extract version number from a string
  28. How to properly nest Bash backticks
  29. How to include nohup inside a bash script?
  30. Subtract two variables in Bash

Leave a Comment