Ah, you saw an OOM, great. That’s definitive. ulimit has nothing to do with this.
Add swap. There’s no reason not to, other than lack of disk space. Add 8G or 16G, try again. You want to get to a working state. Then you can try to measure what process is blowing up, if you wish.
Set up overcommit. It’s good practice, it reduces peak memory problems. You don’t need to understand it or justify it, just do it. It’s part of a good Linux setup. Check first. It’s this easy:
# uname -a
Linux ubuntu-4gb-hel1-1 6.8.0-110-generic #110-Ubuntu SMP PREEMPT_DYNAMIC
Thu Mar 19 17:16:23 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
# cat /proc/sys/vm/overcommit_memory
1