Uploading 17500 users and their avatars using PHP

Dev
1

I have been tasked with moving a Ning community of 17500 users to Discourse. I have gotten the user adding part working using a modified version of the Discourse API for PHP from here https://github.com/communiteq/discourse-api-php.

I plan to parse in the full 17500 users and then work on them one at a time, looping through with a foreach statement.

The sticking point is that there isn’t a clear example of how to upload each user’s avatar (present on my local disk) to the API using the uploads.json endpoint. I see how it can be done if the file is uploaded through a standard browser upload, but the mechanism of using a file loaded from disk programmatically in PHP instead of a browser upload is eluding me.

If anyone can offer any advice on this issue I would appreciate your help enormously. Thanks!

2

Why aren’t you using the ning import script? :thinking:

4

Because I don’t know Ruby at all and I do know PHP. If it’s possible to use without extensive Ruby knowledge then I’ll gladly use that instead.

5

There is a howto topic here:

6

So I can do the import to a local Discourse instance and then restore that to the production version? Will the restore override the settings on the production version? If so then this won’t be viable as loads of configuration has been done on the production version.

7

Yes, restoring a backup does this.

Yes, except for those settings in your app.yml / configuration file - the backup will be restored as it has been backed up.

I believe you could (not done this myself)…

  1. backup your existing production Discourse instance.
  2. restore to the development instance.
  3. process import.
  4. backup dev
  5. restore to production.

Keep in mind depending on your DB size etc you import can take some time.

4 Likes
8

Dean, your final suggested process is what I settled on. I’m going to give it bash and see how it goes. I’ll set up two local instances and try restoring from the one to the other and, if all goes well, then do the same to the production server. Thanks very much for your advice!

2 Likes
9

Okay thus far I have tried installing Discourse locallt using the Vagrant method mentioned in the howto. This wasn’t successful as I got stuck with a fatal error during the bundle install. The debug_inspector failed to install and the bundle could not complete the build.

I then tried the Bitnami Discourse image, which worked great and I got into Discourse without much trouble. I then ran into a wall with the restore of a backup from the production server, where an error was encountered and the restore failed and was rolled back. The error was

[2017-10-09 22:31:14] EXCEPTION: failed to execute:
pg_dump -s -x -O -f /opt/bitnami/apps/discourse/htdocs/db/structure.sql bitnami_discourse

So I’m now in a quandary. I’ve already exceeded the hours I quoted my client to do this user import from Ning, and I’m no closer to the goal.

All I really need to get right is the avatar upload, as I can already add the users from the json export with the custom fields I’ve set up in Discourse. It’s just the avatars. There is no need for any other data except the user profiles and their pictures. Is there perhaps a way to load an image from disk in PHP and somehow get it into the $_FILES collection programmatically, or fake the $_FILES collection in PHP with an array, so as to be able to get the pictures in?

10

The Bitnami Discourse install is not supported or recommended.

You need the Discourse that you’re restoring to to be the same or newer version as the one that you did the backup from.

I’m pretty sure that the avatar import on Ning works as expected.

4 Likes
11

After starting from scratch with the Discourse Vagrant container and once again hitting a wall, I have decided to cut my losses and upload the users with the PHP Discourse API helper. Once the users are uploaded I will upload the avatars.

Can anyone possibly give me an example of what the curl syntax should look like to include the $_FILES as part of the POST, please?

1 Like