How to find a good/optimal dictionary for zlib ‘setDictionary’ when processing a given set of data?

John Reiser explained on comp.compression: For the dictionary: make a histogram of short substrings, sort by payoff (number of occurrences times number of bits saved when compressed) and put the highest-payoff substrings into the dictionary. For example, if k is the length of the shortest substring that can be compressed (usually 3==k or 2==k), then … Read more

Deflate compression browser compatibility and advantages over GZIP

UPDATE: Browsers have been dropping support for raw deflate. zOompf has completed some very thorough research on this very topic here. Unfortunately, it appears that raw deflate is NOT safe to use. Check http://www.vervestudios.co/projects/compression-tests/results for more results. Here are the browsers that have been tested: /* Browser DEFLATE ZLIB */ XP Internet Explorer 6 PASS … Read more

Easy HTTP requests with gzip/deflate compression

For anyone coming across this in recent times, the request library supports gzip decompression out of the box now. Use as follows: request( { method: ‘GET’ , uri: ‘http://www.google.com’ , gzip: true } , function (error, response, body) { // body is the decompressed response body console.log(‘server encoded the data as: ‘ + (response.headers[‘content-encoding’] || … Read more