How to backup Discourse on a compromised and locked down Digital Ocean droplet?

(Panteen Pro-V) #1

Recently my Digital Ocean’s droplet (server) just got compromised by Trojan that turned my server into a zombie for DDOS attack. What happened now is that Digital Ocean locked down my droplet so that it would not cause further damage. They want me to backup the data and move them to new server instance. I have never done anything like this before in my life thus I have couple of questions

  • Which folder should I back it up ( e.g /var/discourse/ , database, etc?..)
  • How should I deploy my backup into new server in the seamless and least technical involved way?

(Jens Maier) #2
mkdir -p /mnt/root
mount --bind / /mnt/root
tar czf /tmp/backup.tar.gz --ignored-failed-read --numeric-owner -P --one-file-system --exclude=./tmp -C /mnt/root .
umount /mnt/root

This sequence will create a gzip compressed tar archive containing everything in your / filesystem; download this file, keep it safe. As long as you have this, you can recover from any mistakes you may make while restoring your forum. However, this backup will also contain the malicion software, so beware!

Moving Discourse is a bit easier. First, copy your app.yml file and everything in the shared directory (tar czf discourse-shared.tar.gz -C /var/discourse shared). Then provision a new droplet and follow the usual Discourse install instructions. Use your old app.yml instead of creating a new one. After bootstrapping, instead of creating your admin user, stop the container and replace the new shared folder with your backup (rm -rf /var/discourse/shared && tar xzf discourse-shared.tar.gz -C /var/discourse), then restart the container.

If you haven’t updated your old instance in a while, you may need to manually migrate your database by SSHing into the container and running rake db:migrate. Give the container one more restart and you should be good.

Lastly, would you share how your droplet got infected? Was some accident or other software installed on the host involved? Is there any evidence that Discourse may have been used to break into the system?

(Panteen Pro-V) #3

I think my server got brute force. It was hard lesson for me because I did not set SSH key login authentication or any firewall, fail2ban. Also the root password is very simple —> this must be the cause of that. Not sure Discourse create backdoor or anything like that since I am not linux expert… :frowning:

Shared webhosting setup?
(Kane York) #4

Yep, that’d do it. Getting set up with a password database is a good investment of your time, because you can use the password database to generate passwords that are impractical (in terms of $$$) to guess.

(Jens Maier) #5

Ah well, now you know better. A couple tips for the next server:

  • create a regular user account for yourself, disable root logins via SSH, and further restrict SSH logins to members of privileged groups (AllowGroups users wheel in sshd_config).
  • configure PAM so that su requires that the calling user is in the wheel group and add your personal user into that group.
  • lock down sudoers so that only specific commands can be run through sudo.
  • install the Google authenticator PAM module for SSH.

Net effect: logins straight to root are impossible; logins to your admin user without a key will require two-factor authentication. If your user account is compromised, sudo can only run predefined commands and su requires a different password.

(Jens Maier) #6

Why? There’s a generator for Correct Horse Battery Staple passwords…

Tho I would grab a copy off Github and install the thing locally, because paranoia. :wink:

(Kane York) #7

Make sure to add it to the docker group!! Also note that membership in docker is equivalent to membership in adm/wheel/admin/sudo (it just might take a few unorthrodox steps to get there).

(Panteen Pro-V) #8

Thanks for your tips. To be honest with you I’m so new to linux, in order to do all steps I need to learn how to do all of them

(Jens Maier) #9

Worse – with sudo locked down, membership in wheel (or whichever group Ubuntu uses in sudoers) is by itself harmless, while membership in docker (with Docker installed in the system) will give you full root privileges without going through any authentication, PAM will have no say in what’s going on.

With Docker it’s safer to straight up su - and run the docker commands from an interactive root shell.

(Jens Maier) #10

There’s a lot to learn, but it can be quite fun actually. :smile: You’ll probably be especially interested in these:

  • man 5 pam.d
  • man 5 sshd_config
  • man 5 sudoers

(Panteen Pro-V) #11

I just gotta man up. Steep learning curve ahead for newbie like me. Thanks for your help

(mountain) #12

I can vouch for that. Finding new ways to harden a VPS is like a heroin addiction for me. I just started teaching myself Linux around the beginning of December and I’ve been finding really amazing, innovative methods to using default programs, like iptables and fail2ban.

While I was mucking around on vultr I made a few simple rules for the iptables firewall. I cut all incoming and outgoing to only localhost and I made it so only my IP can access ssh. Any other IP that tried was auto-banned and dropped.

(Jens Maier) #13

Funnily enough, I’ve never used fail2ban. I know it’s popular, but I don’t like the idea. Controlling system services by monitoring log files seems… unclean; I’d rather have a PAM module do that for me and, ideally, it wouldn’t just drop the attacker’s packets but send them to a tar pit. :drevil:

(mountain) #14

Said tar pit better have tar sharks with frickin’ laser beams on their foreheads. :drevil:

I’ve seen fail2ban useful for something like WordPress, but I don’t plan on using that anyway. I also already have iptables rules to get rid and/or mitigate some of the bull…stuff (aka bad, malicious packets and weakling script kiddie DDoS attacks) as well as moving my ssh port to something else and only allowing my IP to log in. That cuts out a lot already, so using fail2ban is just an extra cherry on top of this super serial security sundae.

I agree with you, to a point. It’s useful but probably not a front-line defense. I plan on using it to catch the rest that get through my net to put into a banned ipset list.

@Huey_Le, you use DO, correct? There’s a lot of security-related tutorials to look into all this stuff and more on DO’s community area. This is a nice starter tutorial to get your feet wet into some options for securing your VPS.

(system) #15