Azure Blob Storage Plugin


(Stephen Chung) #46

After updating to the latest, it seems to be working now. The correct post_uploads record seems to be created.

:tada:


(Stephen Chung) #47

Now that Azure Blob seems to be working fine, any chance to upload backups to it?

Am I too greedy?


(Sam Saffron) #48

Our first priority is sorting out asset hosting like we have for s3, we will eventually get to backups


(Stephen Chung) #49

The update to S3 seems to break Azure Blob. I’m getting Oops after the recent upgrade.

ActionView::Template::Error (undefined method `split' for nil:NilClass) /var/www/discourse/app/models/site_setting.rb:144:in `s3_base_url'

This commit:

changes s3_base_url:

s3_upload_bucket appears to be null when using Azure Blob.

I believe the Azure Blob plugin needs to override s3_base_url or s3_upload_bucket.


(Stephen Chung) #50

But it is strange because according to this:

s3_base_url should return the azure URL instead of falling into the base implementation – which it does by getting the error for split.


(Maja) #51

Yes, with recent changes this issue should be fixed. Can you make sure you’re using the plugin from this repository GitHub - discourse/discourse-azure-blob-storage: A Discourse plugin for storing images and files in azure blob storage and the last version of it?


(Stephen Chung) #52

OK, I find that I was not using the official one.

I’ll switch to the official one and try again.

EDIT: The official plugin works fine! Thanks!


(Stephen Chung) #53

Just realize that this plugin keeps generating some strange error messages in the logs, but doesn’t seem to be causing any ill effects:

Job exception: BlobNotFound (404): The specified blob does not exist.
RequestId:deb4165d-401e-0133-49b4-27e5ab000000
Time:2018-07-30T03:19:56.6174455Z

/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/http/http_request.rb:153:in `call' 
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/http/signer_filter.rb:28:in `call' 
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/http/signer_filter.rb:28:in `call' 
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/http/http_request.rb:110:in `block in with_filter' 
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/service.rb:36:in `call' 
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/filtered_service.rb:34:in `call' 
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/signed_service.rb:41:in `call' 
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-storage-common-1.0.1/lib/azure/storage/common/service/storage_service.rb:60:in `call' /var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-storage-blob-1.0.1/lib/azure/storage/blob/blob_service.rb:176:in `call' 
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-storage-blob-1.0.1/lib/azure/storage/blob/blob.rb:761:in `copy_blob_from_uri' 
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-storage-blob-1.0.1/lib/azure/storage/blob/blob.rb:831:in `copy_blob' 
/var/www/discourse/plugins/discourse-azure-blob-storage/lib/azure_blob_store.rb:30:in `remove_file' 
/var/www/discourse/lib/file_store/base_store.rb:20:in `remove_upload' 
/var/www/discourse/app/models/upload.rb:53:in `block in destroy' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/transaction.rb:230:in `block in within_new_transaction' /usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/transaction.rb:227:in `within_new_transaction' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/transactions.rb:212:in `transaction' 
/var/www/discourse/app/models/upload.rb:52:in `destroy' 
/var/www/discourse/app/jobs/scheduled/clean_up_uploads.rb:68:in `block in execute' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:70:in `each' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:70:in `block in find_each' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:136:in `block in find_in_batches' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:238:in `block in in_batches' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:222:in `loop' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:222:in `in_batches' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:135:in `find_in_batches' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:69:in `find_each' 
/var/www/discourse/app/jobs/scheduled/clean_up_uploads.rb:63:in `execute' 
/var/www/discourse/app/jobs/base.rb:134:in `block (2 levels) in perform' 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.5/lib/rails_multisite/connection_management.rb:63:in `with_connection' 
/var/www/discourse/app/jobs/base.rb:129:in `block in perform' 
/var/www/discourse/app/jobs/base.rb:125:in `each' 
/var/www/discourse/app/jobs/base.rb:125:in `perform' 
/var/www/discourse/app/jobs/base.rb:180:in `perform' 
/var/www/discourse/lib/scheduler/manager.rb:91:in `process_queue' 
/var/www/discourse/lib/scheduler/manager.rb:37:in `block in initialize'

It seems to be trying to tombstone files that do not exist in Azure Blob Storage.

I can’t seem to get a handle on what those files are yet… There are a bunch of posts that come in via email in that are put in queue and that I rejected. Perhaps the attached images inside those images are not yet copied into Azure? Just a thought…

EDIT: Tracing thru the code a bit, it also can be a single upload that exists in uploads but no on the Azure Blob Storage. Therefore the task can be constantly trying to move it to tombstone, but constantly failing at the remove_file call, without actually deleting the uploads record.

So in other words, it can be caused by a single missing upload file over and over again… but how can I find out WHICH upload it is?


(Stephen Chung) #54

I am wondering, is there a way to trap the error and output more meaningful error messages?

For example:

These calls on blob_service will throw if the blob doesn’t exist. However, the exception only returns a 403 (BlobNotFound) error, without any indication on what the non-existing path is.

Would it be a good idea to put try/catch around all blob_service calls so at least we get the context of the call in the error messages?

Alternatively, wouldn’t it be a better idea to skip the tombstoning if the blob no longer exists instead of failing?


(Maja) #55

I’m adding improvements to the plugin this week, I’ll fix the behavior for this case as well, thanks for reporting!


(Heidi V) #56

Hey, I wanted to update one of the instructions:
in

Azure account

  1. If you don’t have an Azure account yet, create one for free
  2. In your Microsoft Azure portal, go to Storage account and add a new account - make sure to select Blob storage under Account kind
  3. Go over to your newly created storage account and scroll down to find BLOB SERVICE. Click on Blobs, and on the upper left corner, click on + Container . Add a container into which uploads should be stored and set the public access level to Blob .
  4. Find Access keys in the storage account settings to fill in later when configuring Discourse.

#57

What happens when you rebake without uploading the files yet? Will the posts rebake any differently compared to when the files are there?