Yes, swap activity is the best baseline measure, as in “if you see consistent swap activity, you definitely don’t have enough memory”. You can get a bit more performance by ensuring you also have enough memory to store the entire working set of disk pages in memory (I’ve talked about this before, specifically the paragraph that starts, “As far as disk cache goes”), but definitely if you’re swapping, your performance is going to be viciously destroyed.
BTW, for anyone comparing the costs and thinking, “OMFG that’s not worth it”, consider that the colocated server is significantly more powerful than the droplet (8x RAM, etc). The closest equivalent droplet is $160/month, so if you were replacing that droplet size with a colo box you’d make your money back in about five months… sure, time is money, etc etc, but stable hardware doesn’t take that much time to keep an eye on.