# Azure Blob Storage Plugin

**URL:** https://meta.discourse.org/t/azure-blob-storage-plugin/79392
**Category:** Plugin
**Tags:** broken
**Created:** [January 30, 2018, 9:28pm UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392 "2018-01-30T21:28:29Z")
**Posts on this page:** 20
**Page:** 2

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [May 27, 2018, 4:17am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/24 "2018-05-27T04:17:24Z")

</div>

> [@maja](#):
>
> If there are no `post_uploads` for `uploads` or even no records for `uploads` themselves, they’ll keep getting removed.

I think if there are no records in uploads then the files won’t be removed to tombstone?

I believe Discourse doesn’t do a directory tree scan to find orphaned files…

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [May 27, 2018, 4:21am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/25 "2018-05-27T04:21:06Z")

</div>

> [@maja](#):
>
> So far I didn’t have any luck reproducing the issue.

If you:

1. manually insert a record in uploads pointing to a new video file that you copied in
2. add the URL in some post
3. rebuild html for that post
4. does it create post\_records?

If you:

1. compose new post
2. upload an mp4
3. fix the URL to get rid of the domain name (must leave the https: in otherwise the video won’t preview)
4. save
5. verify uploads record is created
6. verify post\_uploads record is NOT created

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [May 27, 2018, 4:49am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/26 "2018-05-27T04:49:25Z")

</div>

Just checked.

Posts created via `email in` are not affected by this problem. Video files attached from emails have the `post_uploads` record created successfully.

The reason seems to be that, when a video file is attached in an email, the HTML created is a simple `<a>` tag without `https:`. So it matches the `uploads` record’s `url` field exactly, allowing it to be found.

So this seems to be only a problem with the composer, which creates `<video>` tags.

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [May 31, 2018, 8:23am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/27 "2018-05-31T08:23:52Z")

</div>

Hi, I wonder if there is any potential solution to this issue?

---

<div class="post-metadata">

### Author: ![maja](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/maja/32/129533_2.png) [@maja](https://meta.discourse.org/u/maja)
#### Post date: [June 5, 2018, 1:29pm UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/28 "2018-06-05T13:29:56Z")

</div>

I added a fix for this  
[https://github.com/discourse/discourse/pull/5925](https://github.com/discourse/discourse/pull/5925)

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [June 13, 2018, 6:06am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/30 "2018-06-13T06:06:58Z")

</div>

May I ask when this issue can be resolved?

Will the two PR’s be merged?

---

<div class="post-metadata">

### Author: ![maja](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/maja/32/129533_2.png) [@maja](https://meta.discourse.org/u/maja)
#### Post date: [June 13, 2018, 3:20pm UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/31 "2018-06-13T15:20:16Z")

</div>

Last PR was merged today, so the issue with orphaned uploads should be resolved. The other one which fixes video URLs in the composer is still pending.

---

<div class="post-metadata">

### Author: ![blake](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/blake/32/157322_2.png) [@blake](https://meta.discourse.org/u/blake)
#### Post date: [June 13, 2018, 5:28pm UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/32 "2018-06-13T17:28:09Z")

</div>

> [@schungx](#):
>
> Uploading a video file via the composer will create record in uploads, but no corresponding record in post\_uploads.

> [@maja](#):
>
> The other one which fixes video URLs in the composer is still pending.

@schungx I know its not user friendly, but as a work around and for performance reasons (actually I’m not sure if there is a benefit to oneboxing a large video compared to an uploaded one?) I created a separate azure blob storage container for storing videos and manually upload them to azure (not via discourse in anyway), then I just onebox the video url when composing a post.

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [June 15, 2018, 1:36am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/33 "2018-06-15T01:36:52Z")

</div>

Well, as long as it doesn’t get orphaned then it is a minor hassle to correct the url in the composer.

Only problem is users uploading videos who won’t know to do this.

And yes, i believe video onebox is useful because it is simple to view.

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [June 22, 2018, 7:55am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/34 "2018-06-22T07:55:11Z")

</div>

By the way, ever got to fix the issue of `non-ASCII filenames`?

All file uploads with non-ASCII filenames fail, coming back with a 403:

 ![image](https://global.discourse-cdn.com/meta/original/3X/5/6/5626d43e97812cfddcf54ae4ffbfe744251d3559.png)

---

<div class="post-metadata">

### Author: ![maja](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/maja/32/129533_2.png) [@maja](https://meta.discourse.org/u/maja)
#### Post date: [June 22, 2018, 8:28am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/35 "2018-06-22T08:28:42Z")

</div>

Completely missed that one, sorry. Will take a look.

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [June 25, 2018, 8:05am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/36 "2018-06-25T08:05:04Z")

</div>

> [@maja](#):
>
> Last PR was merged today, so the issue with orphaned uploads should be resolved. The other one which fixes video URLs in the composer is still pending.

I’m sorry to report that this does _not_ work. Upload a video file and the corresponding `uploads` record is created correctly, but the matching `post_uploads` link record is NOT created.

Therefore the file will still be susceptible to being orphaned.

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [June 25, 2018, 8:06am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/37 "2018-06-25T08:06:24Z")

</div>

I use this query to monitor the database:

```plaintext
select * from uploads U left outer join post_uploads P
on (P.upload_id = U.id)
where U.url like '%.mp4'
order by U.id desc

```

---

<div class="post-metadata">

### Author: ![maja](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/maja/32/129533_2.png) [@maja](https://meta.discourse.org/u/maja)
#### Post date: [June 25, 2018, 9:42am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/38 "2018-06-25T09:42:10Z")

</div>

I am unable to reproduce this time since `post_uploads` records are created correctly for me now. Note that they are not created at the same time as `uploads`, but when `ProcessPost` job is ran. Did you query right after creating upload?

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [June 25, 2018, 2:12pm UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/39 "2018-06-25T14:12:08Z")

</div>

> [@maja](#):
>
> `ProcessPost`

Hhhhmmmm… I wonder how frequent this runs. It has been 6 hours now and the `post_uploads` record is still not there.

Will `ProcessPost` run when I turn off orphaning?

NOTE: `ProcessPost` is not in `sidekiq/scheduler`

---

<div class="post-metadata">

### Author: ![Falco](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/falco/32/179432_2.png) [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [June 25, 2018, 2:59pm UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/40 "2018-06-25T14:59:36Z")

</div>

> [@schungx](#):
>
> NOTE: `ProcessPost` is not in `sidekiq/scheduler`

Scheduler will only show the, well, scheduled jobs types (job is triggered by time, cron-like behavior). A lot of jobs are enqueued programmatically, like when a post is created.

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [June 25, 2018, 3:18pm UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/41 "2018-06-25T15:18:30Z")

</div>

OK, because that job isn’t anywhere, and I don’t get the `post_uploads` record. So, either the run has not yet run, or it has run and the record is not created.

---

<div class="post-metadata">

### Author: ![maja](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/maja/32/129533_2.png) [@maja](https://meta.discourse.org/u/maja)
#### Post date: [June 25, 2018, 4:02pm UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/42 "2018-06-25T16:02:49Z")

</div>

But it again happens only for videos? If there was a problem with background job, it would affect all the other uploads too.

Meanwhile the fix for broken URLs in the editor was merged, and the other one for failing uploads with non-ascii filenames as well (you’ll have to update the plugin)

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [June 25, 2018, 4:14pm UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/43 "2018-06-25T16:14:44Z")

</div>

Yes, only for videos. That’s because videos are treated separately with different logic.

---

<div class="post-metadata">

### Author: ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [June 26, 2018, 4:10am UTC](https://meta.discourse.org/t/azure-blob-storage-plugin/79392/44 "2018-06-26T04:10:55Z")

</div>

OK. The `post_uploads` link is not there.

Is there some way I can trace whether the `ProcessPost` job is run and whether it comes up with any errors etc.?

I suspect it is not matching the URL on that video…

[Previous page](https://meta.discourse.org/t/azure-blob-storage-plugin/79392.md?page=1)

[Next page](https://meta.discourse.org/t/azure-blob-storage-plugin/79392.md?page=3)