Yep, I’m afraid your little pi needs more memory. Once it is maxed out like that it will freeze.
One option is to add an NVMe drive and increase swap.
You can then increase swap safely.
I think the only other solution is to source an 8GB 5 for this task and use your 4 for something else.