I'm wondering about pressing relatively small strings, between 1 and 10kb, for storage on my server. The server and receiving client application will not be using my chosen pression during actual requests. The pression will only be used to conserve storage space on the server.

In a case like this, is using gzip with its headers really necessary? Could I use deflate? Maybe even deflate raw, since I know the encoding of the strings in all cases?

The #1 argument I see against deflate is inconsistent implementations in browsers, but this seems irrelevant in my case.

Am I getting something wrong in my thinking?

If deflate is a viable alternative to gzip here, what about deflate raw?

I'm wondering about pressing relatively small strings, between 1 and 10kb, for storage on my server. The server and receiving client application will not be using my chosen pression during actual requests. The pression will only be used to conserve storage space on the server.

In a case like this, is using gzip with its headers really necessary? Could I use deflate? Maybe even deflate raw, since I know the encoding of the strings in all cases?

The #1 argument I see against deflate is inconsistent implementations in browsers, but this seems irrelevant in my case.

Am I getting something wrong in my thinking?

If deflate is a viable alternative to gzip here, what about deflate raw?

• javascript
• node.js
• pression
• gzip
• zlib

Share Improve this question Follow edited Mar 18, 2018 at 16:43 phihag 289k7575 gold badges469469 silver badges483483 bronze badges asked Mar 18, 2018 at 16:00 SlboxSlbox 13.2k1818 gold badges6565 silver badges134134 bronze badges

Add a ment  | 

2 Answers 2

Sorted by: Reset to default 6

First some terminology. Deflate refers to the raw deflate format, as described in RFC 1951. So there is no distinction between "deflate" and "deflate raw". You may be thinking of the misnamed "deflate" HTTP encoding, which is not actually deflate but rather zlib, as described in RFC 1950.

Second, pressing small strings as standalone, depressible files, which it seems you are implying, will result in rather poor pression in most cases. You should be concatenating those strings, along with whatever structure you need to be able to pull them apart again, at least to about the 1 MB level and applying pression to those. You did not say how you want to later access these strings, which would need to be taken into account in such a scheme.

Third, even when pressing your small strings in the 1 KB to 10 KB range, the differences between gzip, zlib, and deflate will be essentially negligible in the space taken. The header and trailers account for 18 bytes, 6 bytes, and 0 bytes respectively for the three formats. So if it is space you are concerned about, there's little gain in going away from gzip.

Fourth, there might be a small speed advantage in not calculating the CRC-32 or Adler-32 check values when pressing (used in gzip and zlib respectively), but it will again be negligible pared to the time spent pressing.

Internally in your web application, you can use any pression - if you send the unpressed data to the HTTP client, there's simply no difference on the wire.

However, doing so is always a trade-off - the implementation of the server will be more plex, and you'll need more CPU power. It also only works if the data is pressible, and large objects (e.g. video files / disk images) tend to be pressed already.

For many applications, disk space is not a problem, and low latency and minimal plexity are much more important concerns. If your application stores a large amount of pressible data which gets requested very infrequently, such a pression scheme may indeed be a good idea.

But before you implement plex code, calculate whether it's really worth the trade-off. Also consider filesystem-level pression (very simple to enable, implementation is somebody else's problem). If your aim is to conserve space, you should also consider various algorithms, from LZ4(very fast) over gzip/deflate/deflateRaw (all identical, just different headers) to LZMA (very slow but very efficient).