Create a swapfile for your Linux server


(Jeff Atwood) #1

Most cloud virtual machine providers do not set up swapfiles as part of their server provisioning.

In particular, upgrading Discourse produces a lot of memory pressure. With a swap file, rather than randomly terminating processes with an out of memory error, things will slow down instead. Having a swap file is a cheap insurance policy that protects you from many other load related failures.

This can be done at any time from the command line on your server.

Set up a 2GB swap file

We recommend a 2GB swap file for Discourse, unless your server has 4GB or more of memory.

In the shell you have opened to your droplet, do the following:

  1. Create an empty swapfile

     sudo install -o root -g root -m 0600 /dev/null /swapfile
    
  2. write out a 2GB file named ‘swapfile’

     dd if=/dev/zero of=/swapfile bs=1k count=2048k
    
  3. tell linux this is the swap file:

     mkswap /swapfile
    
  4. Activate it

     swapon /swapfile
    
  5. Add it to the file system table so its there after reboot:

     echo "/swapfile       swap    swap    auto      0       0" | sudo tee -a /etc/fstab
    
  6. Set the swappiness to 10 so its only uses as an emergency buffer

     sudo sysctl -w vm.swappiness=10
     echo vm.swappiness = 10 | sudo tee -a /etc/sysctl.conf
    

The whole thing as a single copy and pastable script:

sudo install -o root -g root -m 0600 /dev/null /swapfile
dd if=/dev/zero of=/swapfile bs=1k count=2048k
mkswap /swapfile
swapon /swapfile
echo "/swapfile       swap    swap    auto      0       0" | sudo tee -a /etc/fstab
sudo sysctl -w vm.swappiness=10
echo vm.swappiness = 10 | sudo tee -a /etc/sysctl.conf

Memory Usage and Blank Pages
How to update to Discourse 1.0?
Updating plugin versions from Docker Manager exhausting memory
Is Digital Ocean $10/month Package Good for Discourse?
Installation failing on AWS EC2 with Ubuntu 16.04
Errno::ENOMEM: Cannot allocate memory
Memory Useage Tips
Installing on GitHub Pages
Upgrading problems
Much slower after update
Forum upgrade crashing at Upgrade `docker_manager`
Pups error when boostrapping
Could not fork autovacuum worker process
Discourse Upgrade fails every time
"Cannot allocate memory" when upgrading
ActionView Template Error
Updated Discourse, broke my app.yml?
Updated Discourse, broke my app.yml?
Installing forum on AWS
Self hosting time and skills
Could sidekiq queue be reason for 500 errors?
Mail-in-a-Box email
DigitalOcean does NOT recommend swap for SSD storage
DigitalOcean does NOT recommend swap for SSD storage
Avatars intermittently broken
Upgrade error: Failed to build gem native extension
QuotaExceededError - storage - exceeded the quota
Upgrade stalled, followed advice on forum, now forum not accessible
(NoseyNick) #2

Yes, but you need to remember swap is typically a MILLION TIMES slower than memory. There’s a REASON why “Most cloud virtual machine providers do not set up swapfiles”, because they do know what they’re doing.

If you’re really occasionally hitting memory limits, a swapfile might help, but if it’s more than once a week or so, you should really just pony up the extra cash for extra RAM, it’s cheap compared with swapping hell :frowning:

NN


(Jeff Atwood) #3

The cloud providers do this out of self protection. Any modern OS runs with swap, the difference is you don’t have 5 paying customers sharing resources on your desktop computer – all of whom would be affected by hitting swap.


(NoseyNick) #4

Exactly? I think?

Speaking as both a (private) cloud provider and a public+private cloud user, I know “local” disk I/O (which typically cant actually be local, and yes I’m referring to I/O here, not space) is ridiculously expensive compared with RAM, which is cheap and orders of magnitude faster. Swapping is also about the worst imaginable form of random and heavy disk I/O too. One way or another, directly or indirectly, I’ve got to pay for that as a user, or as a provider I have to pass that cost on to the customer(s). If more than a few customers are swapping, they’re crippling the I/O performance for everyone including themselves. Eventually the only realistic workaround is to use (some form of) RAM as a write-cache, which is complicated, costly, and risks “disk” corruption for “written” data that’ still in RAM. From “outside” the VMs it’s also almost impossible to tell what’s “swap I/O” vs “real disk I/O” so I’d have to juggle the risk vs performance VM-by-VM not file-by-file (with greater risk+cost for the high-I/O VMs obviously). It would have been so much easier (+cheaper) to give the VMs more real RAM in the first place.

As a cloud user, I try to reduce my “disk” I/O as much as practically possible, and as a provider I’m happy to help my customers do the same, and improve cache-ability as much as practically possible. All my customers end up happier and performing better, and spending less, as a result.

TL/DR: swap is bad in clouds.

Again, if this is really occasional, maybe a couple of times a week, fine, but if you’re swapping for more than a few minutes daily, then just get more RAM, I think you’ll find that both you, your provider, and your provider’s other customers will end up happier in the long run.

Three more tips to reduce I/O, you can have these for free:   :wink:

  • Windows bad, Linux good. Never mind the difference in disk, memory, licensing, support, paravirt capabilities… in Windows it’s almost impossible to disable swap, officially, or in a supportable way.
  • Mount all drives/partitions with “noatime” option, otherwise (cheap) reads beget (expensive) writes anyway! There’s almost no app I know that cares about atimes anyway (OK, I have a list of 3 pretty obscure bits of software that care), so 99% of the time you’ll get a free performance boost with no appreciable impact.
  • If you happen to be one of the 1% who really do need atime, try “relatime” instead.

If your cloud provider knows what they’re doing, they already disabled swap, enabled noatime, and charge a lot more for Windows VMs :smiley:


(Luke Larris) #5

Not if your provider uses SSDs.


(NoseyNick) #6

True… If you provider is using SSDs you’re paying about 10 times as much as HDD, but swap is “only” tens of thousands of times slower than RAM rather than millions.

If you’re measuring price vs performace you may prefer to pay the extra… But if performance mattered to you, you’d be paying for more RAM instead of swapping to either ANYWAY :smiley:


(Luke Larris) #7

True, swap is just an emergency fallback and not a longterm solution in my opinion.


(Sam Saffron) #8

One thing you are missing here, is that in general we are fine with 1GB of RAM but during upgrade using /admin/docker we need some extra playroom. Having swap is the difference between this process working and failing if you are memory tight.


(Jeff Atwood) #9

Still, I think you are missing something here. There is a difference between total working set (actual memory needed for all active tasks) and total memory in use. With swap, things that are not needed in memory can be swapped to disk to make room for other more useful active program data. Without a swapfile, if you go one BYTE over the total memory at any time, stuff dies horribly and immediately.

It’s like a life insurance policy: not having one is cheaper, yeah, and will save you some short term resources – but not having one at all is taking an extreme position that can be catastrophic in certain rare circumstances.

I can understand cloud providers taking a hard line here because one bad customer could cripple 20 others on the same box if they cause massive IO thrashing.


(NoseyNick) #10

I understand, hence why I said:

Sure, or 10 bad customers could cripple several hundred on the same cluster/zone/whatever.


(Michael Brown) #12

I was pondering - there’s another benefit to giving more memory to your guests when you know they’re going to be hosting many similar images - they can potentially benefit from Kernel Samepage Merging (memory page sharing at a hypervisor level) which can’t happen if the data was swapped to disk.

Did you have this enabled in your environment? Did you try it? Were there benefits?


(ljpp) #15

Would enabling zram be a good option (ubuntu: sudo apt-get install zram-config)? People seem to claim that it is able to provide ~1:3 compression ratio. On an 1GB Ubuntu box, the zram-config installs a 512mb zram compressed virtual swap. Elementary school math says: 0.5GB * 3 + 0.5GB = 2GB.


(Sam Saffron) #16

Possibly could help on extremely low end RAM, I don’t know, you would have to run benches.

see also


(ljpp) #17

I’ve read a ton of articles on zRam and do know how it works. The articles or benchmarks usually focus on desktop usage.

On my 1GB box and a low traffic website the Discourse Docker installation is already causing a little bit of swapping, so enabling zRam might be beneficial. However, zRam is basically trading CPU cycles for RAM, which is something I am not overly excited about on a single-core VPS.


(Jeff Atwood) #18

I was just about to say this – if you are low on RAM but have lots of CPU (fast dual or quad core) than it might make sense. But if you have both low RAM and not much CPU power this is unlikely to improve things for you.


(Jens Maier) #19

Not to mention that your system will have a meager 512 MB of active RAM left to work with and common wisdom (also: sciency resource allocation models) holds that cutting the available RAM in half increses required swap operations by a LOT more then factor two.

Didn’t we learn that lesson back in the days of the memory-doubling placebo-software for Windows 3.11? :grin:


(Jeff Atwood) #20

I moved 7 posts to a new topic: Why does Docker say “WARNING: No swap limit support!”


(PJH) #28

May I suggest:

fallocate -l 1G /swapfile (or 2G)

Simply reserves as uninitialised, rather than actually writing out 1(2)G worth of zeros. Returns a tad quicker.


(Michael Brown) #29

It returns quicker as it doesn’t right out all the zeros, true. But this might lead to problems on some filesystems if fallocate doesn’t actually allocate the space as it claims - you’ll up with a sparse file which doesn’t work for swap.

It should work fine, just be aware that if swapon complains you’ll need to dd the file to remove the holes.


(Jeff Atwood) #30

I tried it on a new DO install and it did not seem to complain and free -m showed the expected swap. So… working? At least for default Ubuntu 14.04 x64 LTS install?