RAM usage climbing, should I have a cron job to restart nginx periodically?

[adamJLev]

Hi there,

big fan of this project, its been working out very well for me!
quick q- I noticed that over time the server RAM usage goes up slowly.
Is this normal? I'm using the base nginx.conf with some small modifications (e.g. using nginx disk cache instead of memory cache).

Thanks guys!

[root@0b8f96506c8a imagesweserv]# cat /proc/meminfo

MemTotal:       31821952 kB
MemFree:        17520340 kB
MemAvailable:   28377172 kB
Buffers:          179768 kB
Cached:         10709368 kB
SwapCached:            0 kB
Active:          2846256 kB
Inactive:       10695384 kB
Active(anon):    2733396 kB
Inactive(anon):        0 kB
Active(file):     112860 kB
Inactive(file): 10695384 kB
Unevictable:        4000 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                 4 kB
Writeback:             0 kB
AnonPages:       2528276 kB
Mapped:           237780 kB
Shmem:             80892 kB
KReclaimable:     511360 kB
Slab:             624356 kB
SReclaimable:     511360 kB
SUnreclaim:       112996 kB
KernelStack:        4880 kB
PageTables:        20360 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    15910976 kB
Committed_AS:    1802396 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       19228 kB
VmallocChunk:          0 kB
Percpu:             3584 kB
HardwareCorrupted:     0 kB
AnonHugePages:   2256896 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      206248 kB
DirectMap2M:     8439808 kB
DirectMap1G:    25165824 kB

[kleisauke]

Sounds like a memory fragmentation issue. Are you using Docker? If so, is jemalloc enabled there?

images/docker/Dockerfile

Lines 74 to 75 in 4133e2f

	# Use jemalloc on glibc-based Linux systems to reduce the effects of memory fragmentation
	LD_PRELOAD=/usr/lib64/libjemalloc.so

images/docker/README.md

Lines 48 to 49 in 4133e2f

	# Check jemalloc configuration
	docker exec -e MALLOC_CONF="stats_print:true,stats_print_opts:mdablxeh" weserv bash

The pre-built Docker image at ghcr.io/weserv/images:5.x uses jemalloc to prevent memory fragmentation in long-running, multi-threaded, glibc-based Linux processes. See lovell/sharp#955 to learn more about memory allocation and why the choice of a memory allocator is important.

For the same reason Redis ships with jemalloc by default since version 2.4.

Since we introduced the specially encoded data types Redis started suffering from fragmentation. We tried different things to fix the problem, but basically the Linux default allocator in glibc sucks really, really hard. […] Every single case of fragmentation in real world systems was fixed by this change, and also the amount of memory used dropped a bit.

[kleisauke]

To answer the question in the issue title, I think restarting nginx via a cron job is unnecessary.

If you've already tried this and noticed an improvement in memory usage, it might indicate an issue with the proxy cache. Did you also modify the following nginx configuration?:

images/ngx_conf/imagesweserv.conf

Lines 1 to 3 in 4133e2f

	# Please adjust cache size (max_size=250m) to a value that you can accommodate in RAM! Advised values: 2 GB RAM: 500m, 4 GB RAM: 1g, 8 GB RAM: 2g, etc..
	proxy_cache_path /dev/shm/proxy_cache inactive=8h levels=1:2 keys_zone=images:512m max_size=250m loader_files=5000 use_temp_path=off;

[adamJLev]

Thanks for the info, Im looking into jemalloc now. But I assume its already on because my dockerfile is basically just inheriting from the standard Docker file for this project right?

Here's my Dockerfile

FROM ghcr.io/weserv/images:sha-4133e2f

STOPSIGNAL SIGQUIT

COPY imagesweserv.conf /etc/nginx/imagesweserv.conf
COPY public/* /var/www/imagesweserv/public/

EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

and here's the modifications I made to the nginx conf, mainly to increase the timeouts and also to use disk cache instead of memory cache:

weserv_process_timeout 95s;
weserv_max_size 300m;
weserv_max_pages 0;
weserv_limit_input_pixels 0;
weserv_limit_output_pixels 0;
weserv_savers jpg png webp json;

# Disk cache: grows to fill disk but ensures 4GB space free, caches resized images up to 1 year
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=images:64m min_free=4g inactive=1y use_temp_path=off;

proxy_cache_min_uses 1;
proxy_read_timeout 95s;
proxy_send_timeout 95s;
proxy_buffers 256 8k;
proxy_buffering on;

# Enable proxy cache lock to make sure we only process the same url request once at a time.
proxy_cache_lock on;
proxy_cache_lock_timeout 95s;

thanks for your insight, much appreciated!

[kleisauke]

jemalloc should already be enabled if you're inheriting from the pre-built Docker image. The custom nginx config changes looks fine, except for the following lines:

weserv_limit_input_pixels 0;
weserv_limit_output_pixels 0;

You should be careful with that, especially if you're processing untrusted input. There are PNG decompression bombs available that could burn all the available memory.

[adamJLev]

yeah, probably should set a sensible number there. thanks for the tip.

for the "memory growth" issue, jemalloc I guess cant be the issue since its based on the pre-built docker image.
for lack of a better solution, I added a cron job that runs /usr/sbin/nginx -s reload once a day and that seems to fix the issue. thanks for the help!

jemalloc should already be enabled if you're inheriting from the pre-built Docker image. The custom nginx config changes looks fine, except for the following lines:

weserv_limit_input_pixels 0;
weserv_limit_output_pixels 0;

You should be careful with that, especially if you're processing untrusted input. There are PNG decompression bombs available that could burn all the available memory.