Improving import and export support

feature
11

Is there a wiki page, or sticky post somewhere that documents the current import/export format, or how to do an import/export?

Or is the source code itself the only source of this information?

12

I believe it is all possible via the admin web gui now. @zogstrip worked on this and we use it extensively.

13

I just installed the latest Discourse via Docker on a Amazon EC2 box.

I have two dummy posts, with a file upload.

I went to /admin, then Backups, and tried to create a backup.

However, it seemed to stall at “Waiting for 17 jobs…”, then errored out.

Full log is on Gist here:

(Hmm - seems my Gist link isn’t shown on the post? Is it some kind of parsing issue?

gist.github.com/victorhooi/9339617

Any thoughts on what I did wrong?

14

It didn’t start the backup because Sidekiq had some jobs running.

Try visiting /sidekiq on your forum as an admin - what do you see?

15

Hmm,this is my Sidekiq admin:

Screen Shot 2014-03-04 at 3.14.30 pm.png2546×1142 109 KB

Queues shows a single queue, “Default”, with no queued jobs.

However, the Backup job now refuses to start.

When I click the Backup button, nothing happens - it doesn’t put the forum in read-only mode, or anything like that.

Should I restart the docker box or something like that, to give it a kick? Or are there specific logs from the box that I can put here that would help diagnose what’s going on?

Also - this is a pretty small box, memory wise - so could it might simply be an issue with being memory constrained? There is only one user and two posts though.

16

How much memory? 1gb is the minimum.

17

I slightly disagree - 1GB is the minimum if and only if you have at least 1GB of swap as well.

Anything less than 1GB RAM will be unable to run the forum, and anything less than 2GB RAM + swap will be unable to do upgrades or other ‘extra’ tasks.

So, really, 2GB ram + swap is the minimum.

18

This is a EC2 Micro - so around 600 Mb of RAM - I figured it was enough for one user (me) and 2 posts.

It’s obviously not a prod setup or even QA - I only set it up because I was excited to checkout the admin export feature you mentioned above (Improving import and export support).

Is it possible to verify it’s a memory issue here? Or is there some other reason it’s seemingly in this confused state where the “Backup” button doesn’t do anything, or why the original job failed?

Are there any more detailed logs I can retrieve from the box to help troubleshoot?

19

Ok, I’ve put the output from ./launcher logs app on Gist here:

@riking - Aha, just saw your comment - wait - you replied after my last post? Lol. Ok, getting used to Discourse ordering =).

Hmm, so you’re saying Discourse is pretty memory hungry, huh? Bugger. So it wouldn’t even be possible to play around with it on any box with less than 2Gb?

20

I plan to have “low mem” Discourse config that gives up on perf, but its still in the works in my head :smile:

1 Like
21

Hmm - sorry, I didn’t quite follow the sentence - do you mean some kind of warning sentence on the /admin page, just like we do for ImageMagick, missing Facebook tokens etc.?

Or something else?

22

No, any box with less than 2GB mem + swap combined.

I’m running one on a $10/mo DigitalOcean cloud server, which is 1GB of ram, but I also set up a 2GB swapfile.

1 Like
23

Ok, so I have it setup now on a higher-specced box, and I was able to run the backup =).

I downloaded the backup tar.gz file, and extracted it. Inside, I have:

  • dump.sql
  • meta.json

Just to clarify - the meta.json file has a timestamp as the version number - does this mean I need to match the exact same git commit in order to do a successful import/export? Is this going to be the case all the way until 1.0?

And the dump.sql - that’s just a straight PostgreSQL dump, right? So there isn’t really a specific Discourse import/export or backup format per se, it’s just a dump of the database.

Hmm, my original purpose of testing this feature was to see how I might import another forum (www.lefora.com) into Discourse.

If I want to do that - do I need to generate a dump.sql file exactly like this, including all the DDL statements, in order for Discourse to import? There’s no easier way/format?

24

I set up a Digital Ocean box with low memory and enough swap to mock around and I run into the problem where running backups or restores from the admin panel would fail with ‘Waiting for sidekiq to finish running jobs…’. and multiple lines of ‘Waiting for 1 jobs…’. I don’t think it has anything to do with memory. I tried to resize the DO droplet and increase memory to 2GB with no help.

Turns out even though (default) queue was empty, sidekiq had one email worker from my registration email that I needed to remove for it to run. I didn’t setup email for this install since I just wanted to play around and not use the install for anything else than testing. After clicking ‘Clear workers list’ it runs fine even with 512MB memory (+swap of course).

2 Likes
25

The meta.json file currently only holds the current database version (which is the timestamp when the last migration was created). This value is used during the restore to make sure you’re not importing a newer version of the database without migrating first.

Not exactly.

We do use the standard pg_dump command to generate the dump but then we add a slight modification.

In order to limit the amount of downtime during the restoration, we make sure to restore the backup in the restore schema instead of the standard public schema.

This allow us to limit the downtime to the amount of time it takes to switch the public schema to the backup schema, and the restore schema to the public schema :wink:

You will need to write custom code for that. You may want to take a look at

for inspiration.

Unfortunately no. Welcome to a world of pain :rage1:

26

Hi,

Have there been any changes/updates on the Discourse import/migration front?

Or is improving import/export support still on the roadmap somewhere?

There was talk about this Discourse migration service releasing some of their code as open-source, however, that doesn’t appear to have eventuated.

Is anybody aware of any up-to-date migration/import tools for Discourse?

27

There are some migration tools, @techapj can you provide a howto topic for finding it and getting started? Can be a very basic howto for now, just the location of the code basically. Longer term we will produce a detailed howto for migrating a sample PHPbb forum, but it may take a few months for us to get to that.

But in the meantime, the migration / conversion code is open source, you just have to know where in the tree to find it. Warning, it is evolving rapidly and meant only for developers at this time.

1 Like
28

Here is the howto for migration:

2 Likes
29

This is totally implemented and quite rocking.

Backup/Restore on Discourse is quite a lifesaver at times.

3 Likes
30