Automating Server Maintenance


(Joshua Rosenfeld) #1

So I will be leaving the university where I have set up Discourse as an internal knowledge base in a few weeks. As the server is specific to the group I worked with, it isn’t managed by any of the existing sys-admins. My supervisor has requested that I automate some of the maintenance tasks that I do to avoid the need for manual intervention. I’ve been thinking about what I do to the server, and have organized the commands into a weekly cron job. Can anyone (like @mpalmer) think of a reason why this would be dangerous, and is there anything I can add to it? (Note, I know ./lanucher rebuild app shouldn’t be neccessary this often, but she doesn’t know how reliable the new Discourse admins will be in updating via Docker Manager…)

sudo crontab -e:

# Edit this file to introduce tasks to be run by cron.
# 
# **Comments removed from upload as they're irrelevant to post**
#
# m h  dom mon dow   command
0 3 * * 1 /var/www/html/weeklyCleanup > /var/log/weeklyCleanup/cleanup.log 2>&1

/var/www/html/weeklyCleanup:

#!/bin/bash
# Weekly update and Cleanup Script

apt-get update -q
apt-get upgrade -q
apt-get autoclean -y -q
apt-get autoremove -y -q
cd /var/discourse
git pull
./launcher rebuild app
./launcher cleanup
sudo shutdown -r 5 "Server is going down for weekly cleanup. Please save and close any applications you are working with."

/etc/logrotate.d/weeklyCleanup:

/var/log/weeklyCleanup/cleanup.log {
   nocompress
   weekly
   missingok
   rotate 10
}

(Rafael dos Santos Silva) #2

dpkg-reconfigure -plow unattended-upgrades can handle the OS updates, and log rotation should be automatic.

Only thing missing is updating Discourse, that can be updated by the phone, clicking on the new version email.

I think that a weekly automatic rebuild is a bit dangerous, like when a database update is needed.


(Joshua Rosenfeld) #3

That’s already enabled. Are you suggesting removing the following two lines?

apt-get update -q
apt-get upgrade -q

Due to redirecting the crontab output to it’s own log it isn’t automatic from my testing…unless I’ve missed something.

Does git pull and ./launcher rebuild not update Discourse?

What do you mean by phone? I will have no contact with the university once I leave (graduate).

Not possible for me, no way to access the server from off-site, won’t have VPN privileges anymore.

Can you be more specific? Would the database update need user-intervention? What makes it dangerous?


(Rafael dos Santos Silva) #4

I was saying that not using your script, and only dpkg-reconfigure -plow unattended-upgrades should be enough in most cases.

Last time Discourse had a database update, you had to run the rebuild command three times to get it running.

Discourse gets an update, staff gets an email, click on the email link and updates Discourse. Putting Discourse on stable means a click every 6 months, not that bad.


(Joshua Rosenfeld) #5

When was the last database update, and how were site admins alerted to the need to run rebuild 3 times? That’s concerning as no one else is active here on Meta and I worry (even if we’re on stable) that things could break with an update…

We’re currently on tests-passed, and update with the new beta notifications. Stable doesn’t seem even with tests-passed, is it still possible to change the tracked branch?


(Rafael dos Santos Silva) #6

It’s safe right now: Comparing stable...master · discourse/discourse · GitHub


(Joshua Rosenfeld) #7

Sounds good, I’ll check with my boss tomorrow. Can you please also clarify my first question above. Specifically: [quote=“jomaxro, post:5, topic:55368”]
how were site admins alerted to the need to run rebuild 3 times?
[/quote]


(Rafael dos Santos Silva) #8

On the rebuild output.