I think if there are no records in uploads then the files won’t be removed to tombstone?
I believe Discourse doesn’t do a directory tree scan to find orphaned files…
If you:
- manually insert a record in uploads pointing to a new video file that you copied in
- add the URL in some post
- rebuild html for that post
- does it create post_records?
If you:
- compose new post
- upload an mp4
- fix the URL to get rid of the domain name (must leave the https: in otherwise the video won’t preview)
- save
- verify uploads record is created
- verify post_uploads record is NOT created
1 Like
Just checked.
Posts created via email in are not affected by this problem. Video files attached from emails have the post_uploads record created successfully.
The reason seems to be that, when a video file is attached in an email, the HTML created is a simple <a> tag without https:. So it matches the uploads record’s url field exactly, allowing it to be found.
So this seems to be only a problem with the composer, which creates <video> tags.
2 Likes
Hi, I wonder if there is any potential solution to this issue?
I added a fix for this
https://github.com/discourse/discourse/pull/5925
8 Likes
May I ask when this issue can be resolved?
Will the two PR’s be merged?
Last PR was merged today, so the issue with orphaned uploads should be resolved. The other one which fixes video URLs in the composer is still pending.
4 Likes
@schungx I know its not user friendly, but as a work around and for performance reasons (actually I’m not sure if there is a benefit to oneboxing a large video compared to an uploaded one?) I created a separate azure blob storage container for storing videos and manually upload them to azure (not via discourse in anyway), then I just onebox the video url when composing a post.
2 Likes
Well, as long as it doesn’t get orphaned then it is a minor hassle to correct the url in the composer.
Only problem is users uploading videos who won’t know to do this.
And yes, i believe video onebox is useful because it is simple to view.
1 Like
By the way, ever got to fix the issue of non-ASCII filenames?
All file uploads with non-ASCII filenames fail, coming back with a 403:
Completely missed that one, sorry. Will take a look.
5 Likes
I’m sorry to report that this does not work. Upload a video file and the corresponding uploads record is created correctly, but the matching post_uploads link record is NOT created.
Therefore the file will still be susceptible to being orphaned.
I use this query to monitor the database:
select * from uploads U left outer join post_uploads P
on (P.upload_id = U.id)
where U.url like '%.mp4'
order by U.id desc
3 Likes
I am unable to reproduce this time since post_uploads records are created correctly for me now. Note that they are not created at the same time as uploads, but when ProcessPost job is ran. Did you query right after creating upload?
4 Likes
Hhhhmmmm… I wonder how frequent this runs. It has been 6 hours now and the post_uploads record is still not there.
Will ProcessPost run when I turn off orphaning?
NOTE: ProcessPost is not in sidekiq/scheduler
Scheduler will only show the, well, scheduled jobs types (job is triggered by time, cron-like behavior). A lot of jobs are enqueued programmatically, like when a post is created.
2 Likes
OK, because that job isn’t anywhere, and I don’t get the post_uploads record. So, either the run has not yet run, or it has run and the record is not created.
But it again happens only for videos? If there was a problem with background job, it would affect all the other uploads too.
Meanwhile the fix for broken URLs in the editor was merged, and the other one for failing uploads with non-ascii filenames as well (you’ll have to update the plugin)
Yes, only for videos. That’s because videos are treated separately with different logic.
OK. The post_uploads link is not there.
Is there some way I can trace whether the ProcessPost job is run and whether it comes up with any errors etc.?
I suspect it is not matching the URL on that video…