You don’t need to make new slices, just append slices of logs to the divided slice. http://play.golang.org/p/vyihJZlDVy var divided [][]string chunkSize := (len(logs) + numCPU – 1) / numCPU for i := 0; i < len(logs); i += chunkSize { end := i + chunkSize if end > len(logs) { end = len(logs) } divided … Read more
Using awk you could: awk ‘/^\+$/ { delim++ } { file = sprintf(“chunk%s.txt”, int(delim / 50000)); print >> file; }’ < input.txt Update: To not include the delimiter, try this: awk ‘/^\+$/ { if(++delim % 50000 == 0) { next } } { file = sprintf(“chunk%s.txt”, int(delim / 50000)); print > file; }’ < input.txt … Read more
CORRECTED TWICE: A simpler way to un-chunk a lazy sequence: (defn unchunk [s] (when (seq s) (lazy-seq (cons (first s) (unchunk (next s)))))) First version omitted (when … so it returned an infinite seq of nil’s after the input sequence ended. Second version used first instead of seq so it stopped on nil. RE: your … Read more
This is probably more efficient (faster) def batch(iterable, n=1): l = len(iterable) for ndx in range(0, l, n): yield iterable[ndx:min(ndx + n, l)] for x in batch(range(0, 10), 3): print x Example using list data = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] # list of data for x in batch(data, … Read more