20 Aug 2022

[nginx] HTTP Serving Compressed File

Abstract

Most web browsers include built-in support for gzip, deflate, br
Most programming languages include built-in support for gzip and deflate, and br typically has community support

Nginx - ngx_http_gzip_static_module

Conclusions:

Check config: if gzip_static enable or not ?
Check request header: client support gzip or not ?
- when using common algorithm, like “gzip”, skiping this seems no problem
Mapping url to file path with given rule (filename extension is “.gz”)
Setting response’s headers: Content-Length/Last-Modified/Etag?/Content-Type?
Ensuring response’s header Content-Encoding: gzip

// nginx-1.22.0 - src/http/modules/ngx_http_gzip_static_module.c

static ngx_int_t
ngx_http_gzip_static_handler(ngx_http_request_t *r)
{
	...
	// enable gzip_static
    if (gzcf->enable == NGX_HTTP_GZIP_STATIC_ON) {
		// clietn support gzip ? Accept-Encoding: gzip
        rc = ngx_http_gzip_ok(r);

    } else {
        /* always */
        rc = NGX_OK;
    }


    p = ngx_http_map_uri_to_path(r, &path, &root, sizeof(".gz") - 1);
    if (p == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }
	
	// local gzip static file's filename must has ".gz" extension
    *p++ = '.';
    *p++ = 'g';
    *p++ = 'z';
    *p = '\0';
	
	...
	
	// like serving static file, but ensure header "Content-Encoding: gzip"
	
    r->headers_out.status = NGX_HTTP_OK;
    r->headers_out.content_length_n = of.size;
    r->headers_out.last_modified_time = of.mtime;

    if (ngx_http_set_etag(r) != NGX_OK) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    if (ngx_http_set_content_type(r) != NGX_OK) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    h = ngx_list_push(&r->headers_out.headers);
    if (h == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    h->hash = 1;
	
	// response must set header: Content-Encoding: gzip
    ngx_str_set(&h->key, "Content-Encoding");
    ngx_str_set(&h->value, "gzip");
    r->headers_out.content_encoding = h;

 	...

    return ngx_http_output_filter(r, &out);
}

Example - serving brotli static file with nginx

server {
		...

		location /br-static {
			if ($http_accept_encoding ~ "(, br$)|(, br,)|(^br,)|( br,)|(^br$)") {
				add_header Content-Encoding br;
				rewrite ^(.*)$ /$1.br break;
			}
			try_files $uri =404;
		}

		...
}

Metrics - compressor benchmark

Conclusions:

To prevent simply selecting a compressor based on ratio and speed, resource usage and compatibility with other tools (rsync) must also be considered.
zstd and gzip can be rsync friendly with given option (not default)
The ratio is highly dependent on the file’s content, for common log file, >3 is possible

Compressor name	Ratio	Compression	Decompress.
zstd 1.5.1 -1	2.887	530 MB/s	1700 MB/s
zlib 1.2.11 -1	2.743	95 MB/s	400 MB/s
brotli 1.0.9 -0	2.702	395 MB/s	450 MB/s
zstd 1.5.1 –fast=1	2.437	600 MB/s	2150 MB/s
zstd 1.5.1 –fast=3	2.239	670 MB/s	2250 MB/s
quicklz 1.5.0 -1	2.238	540 MB/s	760 MB/s
zstd 1.5.1 –fast=4	2.148	710 MB/s	2300 MB/s
lzo1x 2.10 -1	2.106	660 MB/s	845 MB/s
lz4 1.9.3	2.101	740 MB/s	4500 MB/s
lzf 3.6 -1	2.077	410 MB/s	830 MB/s
snappy 1.1.9	2.073	550 MB/s	1750 MB/s

Credit: zstd

Nutshell

[nginx] HTTP Serving Compressed File

Abstract

Nginx - ngx_http_gzip_static_module

Example - serving brotli static file with nginx

Metrics - compressor benchmark

Reference