Azure Blob Storage Plugin

(Stephen Chung) #24

I think if there are no records in uploads then the files won’t be removed to tombstone?

I believe Discourse doesn’t do a directory tree scan to find orphaned files…

(Stephen Chung) #25

If you:

  1. manually insert a record in uploads pointing to a new video file that you copied in
  2. add the URL in some post
  3. rebuild html for that post
  4. does it create post_records?

If you:

  1. compose new post
  2. upload an mp4
  3. fix the URL to get rid of the domain name (must leave the https: in otherwise the video won’t preview)
  4. save
  5. verify uploads record is created
  6. verify post_uploads record is NOT created

(Stephen Chung) #26

Just checked.

Posts created via email in are not affected by this problem. Video files attached from emails have the post_uploads record created successfully.

The reason seems to be that, when a video file is attached in an email, the HTML created is a simple <a> tag without https:. So it matches the uploads record’s url field exactly, allowing it to be found.

So this seems to be only a problem with the composer, which creates <video> tags.

(Stephen Chung) #27

Hi, I wonder if there is any potential solution to this issue?

(Maja) #28

I added a fix for this

Discourse Deleting Uploads moved from Local to S3
(Stephen Chung) #30

May I ask when this issue can be resolved?

Will the two PR’s be merged?

(Maja) #31

Last PR was merged today, so the issue with orphaned uploads should be resolved. The other one which fixes video URLs in the composer is still pending.

(Blake Erickson) #32

@schungx I know its not user friendly, but as a work around and for performance reasons (actually I’m not sure if there is a benefit to oneboxing a large video compared to an uploaded one?) I created a separate azure blob storage container for storing videos and manually upload them to azure (not via discourse in anyway), then I just onebox the video url when composing a post.

(Stephen Chung) #33

Well, as long as it doesn’t get orphaned then it is a minor hassle to correct the url in the composer.

Only problem is users uploading videos who won’t know to do this.

And yes, i believe video onebox is useful because it is simple to view.

(Stephen Chung) #34

By the way, ever got to fix the issue of non-ASCII filenames?

All file uploads with non-ASCII filenames fail, coming back with a 403:


(Maja) #35

Completely missed that one, sorry. Will take a look.

(Stephen Chung) #36

I’m sorry to report that this does not work. Upload a video file and the corresponding uploads record is created correctly, but the matching post_uploads link record is NOT created.

Therefore the file will still be susceptible to being orphaned.

(Stephen Chung) #37

I use this query to monitor the database:

select * from  uploads U left outer join post_uploads P
on (P.upload_id =
where U.url like '%.mp4'
order by desc

(Maja) #38

I am unable to reproduce this time since post_uploads records are created correctly for me now. Note that they are not created at the same time as uploads, but when ProcessPost job is ran. Did you query right after creating upload?

(Stephen Chung) #39

Hhhhmmmm… I wonder how frequent this runs. It has been 6 hours now and the post_uploads record is still not there.

Will ProcessPost run when I turn off orphaning?

NOTE: ProcessPost is not in sidekiq/scheduler

(Rafael dos Santos Silva) #40

Scheduler will only show the, well, scheduled jobs types (job is triggered by time, cron-like behavior). A lot of jobs are enqueued programmatically, like when a post is created.

(Stephen Chung) #41

OK, because that job isn’t anywhere, and I don’t get the post_uploads record. So, either the run has not yet run, or it has run and the record is not created.

(Maja) #42

But it again happens only for videos? If there was a problem with background job, it would affect all the other uploads too.

Meanwhile the fix for broken URLs in the editor was merged, and the other one for failing uploads with non-ascii filenames as well (you’ll have to update the plugin)

(Stephen Chung) #43

Yes, only for videos. That’s because videos are treated separately with different logic.

(Stephen Chung) #44

OK. The post_uploads link is not there.

Is there some way I can trace whether the ProcessPost job is run and whether it comes up with any errors etc.?

I suspect it is not matching the URL on that video…