Here’s an update - I did eventually get it working. Some of the lessons learned may help others, hence the detailed write-up below. The Unix permissions gotcha might be worth adding to the OP.
Unix Permissions on the attachments folder
The SMF2 importer was working fine but there was a slight gotcha to do with the Unix permissions of the attachments. When running the import script it is running as the discourse user, doesn’t necessarily have both read and write permissions to the attachments volume mount.
During import it looks like the importer creates small temporary files in that same directory, so you must ensure the correct user has both read and write permissions. I did this by chown-ing the directory on the host recursively to the discourse user, but there are other ways including setting very open permissions, or using Access Control Lists.
Debugging failed uploads is difficult
If the SMF2 importer fails on an upload for any reason it reports failure in the same way, so it can be hard to tell why an attachment/upload has failed.
Reasons can include:
- Attachments are in the wrong place on the file system.
- The permissions on the attachments are insufficient (needs
rw). - The Docker volume mount is not correctly set up.
- The SMF2 DB is reporting there should be an attachment which no longer exists in your attachments folder (in our SMF2 forum, which is about 20 years old, there were quite a few ‘lost’ attachments - there’s not much you can do for these…)
- The upload object is not being correctly saved in the Discourse DB or on disk in
/shared/standalone/uploads - The markdown for the upload link is not being created.
I debugged this by testing every step of the attachment process from end-to-end and adding a huge number of extra debugging puts statements in the smf2.rb code, to work out which part of the upload creation process was not working. In my case there were numerous little issues which eventually got ironed out one-by-one.
Don’t be lazy Marcus
And at the very end I got lazy and wasn’t fully clearing the DB back to zero, so I had the issue that everything was succeeding - but the importer doesn’t overwrite a Topic or Post which has already been imported. So the Markdown that references the upload, which should look like:
---
was not being inserted into posts.
To fix this, I reset the Discourse instance’s DB back to a pre-import state. It’s recommended to do this by simply deleting everything in shared/standalone, but that would have lost me all customisation, branding, and theming, configuration, setup, and a fair bit of manually imported topics from yet another forum they also have. So instead I restored a very early backup from before I did the import, which worked. It’s a slightly more risky approach because old backups aren’t always 100% guaranteed to be compatible unless you also roll back the installed Discourse version. This worked and I was then able to re-run the import, all available attachments succeeded, and the Markdown upload links were correctly inserted.
Testing it’s working
The way I found to test if this was all working end-to-end was to run
Post.where("raw LIKE ?", "%upload://%").count
in the Rails console. It would return the count of Posts that actually have a proper upload link in them and this number gradually goes up throughout the import process, which is reassuring.