Synchronizer-base for any backup provider

Plugin
1

:warning: The discourse-sync-base plugin is currently working only with the discourse-sync-to-dropbox and discourse-sync-to-googledrive plugins. :warning:

We built new plugins for making a backup to a provider of your choice. All of our plugins follow the same structure and need to be installed together with a synchronizer-base plugin.

plugins

The discourse-sync-base

This plugin contains the base class for all other synchronizers. This way we made sure that our other synchronizer plugins follow the same structure and inherit the same methods.

The discourse-sync-to-dropbox

We forked @Falco’s backups-to-dropbox plugin and changed its structure in a way that fits our sync-base. Check out our plugin here if you want to do a backup to dropbox.
Please note that since people with the old version of the dropbox plugin from Falco now have issues, because in his repository the naming didn’t change so the plugin is not loaded in the right order. We will revert our pull request to Falco so that you can still use the old version of the plugin, where you don’t need to install the base class.

The discourse-sync-to-googledrive

If you prefer synchronizing your files to your Google Drive account, we built a plugin for this one too. Learn how to install and use it here.

The discourse-sync-to-nextcloud

Nextcloud is a new cloud tool that works together with several different providers you can pick from. We built a plugin that makes it easy to synchronize your backup files to any of these providers.

The discourse-sync-to-box

As promised, we also have a new plugin to sync your backups to box. :slight_smile:

TODO

About us

We (@Jen_Lijo and @kajatiger) are the team Berlin Diamonds from the Rails Girls Summer of Code. If you want to see more about our approach and the process how we built the plugins, read our posts on meta. We will continue making plugins with other providers. If you have any experience with a good provider for plugins, please give us suggestions what we should work on next.

27 Likes
2

Only major provider that I know of that’s not on that list is Box, which is also featured in several WordPress backup plugins,

https://developer.box.com

p.s. Microsoft “Skydrive” doesn’t exist any longer. It’s called OneDrive now.

3 Likes
3

Totally right. Our next plugin will be for Box and while creating an account for skydrive, I realized that it is called “onedrive” now :slight_smile: Thanks for your input!

1 Like
4

@eviltrout Perhaps these plugins can be merged into Discourse? Right now we’re only supporting backups uploads to S3 and supporting more providers is definitely a plus to me.

3 Likes
5

I think they would need to be tested a lot more. Nobody ever got fired for picking Amazon s3.

3 Likes
6

Hey @erlend_sh your message arrives just on time to find the new discourse-sync-to-box fresh from the oven :woman_cook:

5 Likes
7

I feel honored that someone wants to merge our plugins, but the question is “why?”. They are no necessary conditions for Discourse and setting up the app is already a big and heavy thing that at least on my local computer runs the tests for almost 15 minutes.
Our plugins should be a nice extra for people who like customizing the usage to their personal needs. Don’t you think so?

PS. @codinghorror We would be very happy about your evaluation and hints on where to optimize our code. Testing has been a painful but good lesson for us so far.

1 Like
8

Also the plugins require many gems to support syncing – @tgxworld it would mean installing dozens of gems in core discourse which we don’t want. Let’s let the users pick the providers they want!

8 Likes
9

Ah good point :+1:

I felt that having the provider be chosen via a site setting would be easier but I guess it isn’t worth the trade of in this case.

3 Likes
10

Yesterday I installed the sync-base plugin and the sync-to-dropbox plugin by replacing the deprecated dropbox plugin but it does not work.

I tried to trigger the job manually twice on sidekiq but it fails:


Logs errors:

Backtrace 
/var/www/discourse/plugins/discourse-sync-to-dropbox/lib/dropbox_synchronizer.rb:56:in `upload'
/var/www/discourse/plugins/discourse-sync-to-dropbox/lib/dropbox_synchronizer.rb:47:in `block in upload_unique_files'
/var/www/discourse/plugins/discourse-sync-to-dropbox/lib/dropbox_synchronizer.rb:42:in `each'
/var/www/discourse/plugins/discourse-sync-to-dropbox/lib/dropbox_synchronizer.rb:42:in `upload_unique_files'
/var/www/discourse/plugins/discourse-sync-to-dropbox/lib/dropbox_synchronizer.rb:38:in `perform_sync'
/var/www/discourse/plugins/discourse-sync-base/lib/synchronizer.rb:10:in `sync'
/var/www/discourse/plugins/discourse-sync-to-dropbox/app/jobs/regular/sync_backups_to_dropbox.rb:9:in `block in execute'
/var/www/discourse/plugins/discourse-sync-to-dropbox/app/jobs/regular/sync_backups_to_dropbox.rb:8:in `each'
/var/www/discourse/plugins/discourse-sync-to-dropbox/app/jobs/regular/sync_backups_to_dropbox.rb:8:in `execute'
/var/www/discourse/app/jobs/base.rb:153:in `block (2 levels) in perform'
Env 
hostname	adu1-app
process_id	145
application_version	9a861c1b436ca571d06235c07de0f815cd5a90ac
current_db	default
current_hostname	forum.adunanza.net
job	Jobs::SyncBackupsToDropbox
problem_db	default
	
opts	
current_site_id	default

Dropbox settings where I clicked to generate the token:

Backup size: 1.7 Gb

I deleted the app created on Dropbox for the deprecated plugin and created a new one for sync-to-dropbox. It seems dropbox does not like that much, because it continues to recreate the old folder.

This evening I’ll try a rebuild to see if it fix this error.

1 Like
11

So the plugin creates a folder in dropbox but doesn’t upload the files?
In your logs it looks like the upload_unique_files method is causing the problem…
I will look into it.

12

Exactly, every time I try to do the manual trigger on sidekiq, the error immediately appears in my log.

The only thing unclear in the instructions is whether the sync-base plugin should be installed on its own (with a rebuild) and only later add the plugins of the various backup services (with another rebuild) or if they can be installed together by activating before the sync-base.

I did the second thing but maybe I was wrong…Please, let me know which of the two methods is the correct one.

2 Likes
13

I think it makes no difference if you install them both together or individually, but I am not sure. I don’t use vagrant and just manually clone the plugins into my plugins folder…

But for your issue, I think I found the bug. There seems to be a little mistake in this upload_unique_files method because the loop is not handling single elements for some reason, but the whole array. I saw that in the console. I will try to fix it now and push the changes to github and we will see if that fixes your problem.

Thanks for letting us know the issue!

3 Likes
14

if the S3 plugin allowed to specify other API endpoints, it would be perfect :slight_smile:

15

hey @Trash sorry for the issue, the error was in this line:

backup.chunked_upload(folder_name, file_name, full_path)
"Backup does not have “chunk_upload”.

see here:
https://meta.discourse.org/t/discourse-backups-to-dropbox-deprecated/51176/39

It was corrected by @AlbertA in original Falco’s post but we didn’t update our repo.
If there was a big file coming, the sync-to-dropbox wasn’t able to handle it because the upload limit of dropbox + chunked_upload method broken.
It’s all good and working now :slight_smile:

2 Likes
16

I agree, I also thought that part of the instructions was a bit ambiguous. Maybe it can be clarified once it’s confirmed to work with simultaneous install?

17

Documentation will be improved soon. So far, only one backup provider can be installed together with the sync-base. We’ll look into the rebuild matter too, as not being familiar with Discourse we didn’t think about these possibilities!

2 Likes
18

Another thing that could be clarified is that you also have to activate the sync-base plugin (or perhaps you could make it so that activating the sync-to-dropbox plugin automatically also makes sure the sync-base plugin is activated).

3 Likes
19

Everything works perfectly now @Jen_Lijo and @kajatiger, thanks.

@tophee I can confirm that sync-base + sync-to-… can be installed together without problems.

5 Likes
20

Hi All,

I need some help with the discourse-sync-to-box plugin. Here is what I have verified:

  • I have the sync-base plugin installed and activated
  • I have the sync-to-box plugin installed, configured, and activated
  • I have daily backups configured and working
  • I can manually run the sync_backups_to_box job and it works: Jobs.enqueue_in(60, :sync_backups_to_box)
  • All my sidekiq queues are running (critical, default, low)

The behavior I am seeing is that backups never make it to box automatically.

Thanks for the help.

1 Like