After updating to the latest, it seems to be working now. The correct post_uploads record seems to be created.
5 Likes
Now that Azure Blob seems to be working fine, any chance to upload backups to it?
Am I too greedy?
Our first priority is sorting out asset hosting like we have for s3, we will eventually get to backups
3 Likes
The update to S3 seems to break Azure Blob. Iâm getting Oops after the recent upgrade.
ActionView::Template::Error (undefined method `split' for nil:NilClass) /var/www/discourse/app/models/site_setting.rb:144:in `s3_base_url'
This commit:
changes s3_base_url:
https://github.com/discourse/discourse/blob/master/app/models/site_setting.rb#L143-L146
s3_upload_bucket appears to be null when using Azure Blob.
I believe the Azure Blob plugin needs to override s3_base_url or s3_upload_bucket.
But it is strange because according to this:
https://github.com/discourse/discourse-azure-blob-storage/blob/master/plugin.rb#L42-L45
s3_base_url should return the azure URL instead of falling into the base implementation â which it does by getting the error for split.
Yes, with recent changes this issue should be fixed. Can you make sure youâre using the plugin from this repository https://github.com/discourse/discourse-azure-blob-storage and the last version of it?
6 Likes
OK, I find that I was not using the official one.
Iâll switch to the official one and try again.
EDIT: The official plugin works fine! Thanks!
5 Likes
Just realize that this plugin keeps generating some strange error messages in the logs, but doesnât seem to be causing any ill effects:
Job exception: BlobNotFound (404): The specified blob does not exist.
RequestId:deb4165d-401e-0133-49b4-27e5ab000000
Time:2018-07-30T03:19:56.6174455Z
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/http/http_request.rb:153:in `call'
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/http/signer_filter.rb:28:in `call'
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/http/signer_filter.rb:28:in `call'
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/http/http_request.rb:110:in `block in with_filter'
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/service.rb:36:in `call'
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/filtered_service.rb:34:in `call'
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-core-0.1.13/lib/azure/core/signed_service.rb:41:in `call'
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-storage-common-1.0.1/lib/azure/storage/common/service/storage_service.rb:60:in `call' /var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-storage-blob-1.0.1/lib/azure/storage/blob/blob_service.rb:176:in `call'
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-storage-blob-1.0.1/lib/azure/storage/blob/blob.rb:761:in `copy_blob_from_uri'
/var/www/discourse/plugins/discourse-azure-blob-storage/gems/2.5.1/gems/azure-storage-blob-1.0.1/lib/azure/storage/blob/blob.rb:831:in `copy_blob'
/var/www/discourse/plugins/discourse-azure-blob-storage/lib/azure_blob_store.rb:30:in `remove_file'
/var/www/discourse/lib/file_store/base_store.rb:20:in `remove_upload'
/var/www/discourse/app/models/upload.rb:53:in `block in destroy'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/transaction.rb:230:in `block in within_new_transaction' /usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/transaction.rb:227:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/transactions.rb:212:in `transaction'
/var/www/discourse/app/models/upload.rb:52:in `destroy'
/var/www/discourse/app/jobs/scheduled/clean_up_uploads.rb:68:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:70:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:70:in `block in find_each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:222:in `loop'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:222:in `in_batches'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/batches.rb:69:in `find_each'
/var/www/discourse/app/jobs/scheduled/clean_up_uploads.rb:63:in `execute'
/var/www/discourse/app/jobs/base.rb:134:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.5/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:129:in `block in perform'
/var/www/discourse/app/jobs/base.rb:125:in `each'
/var/www/discourse/app/jobs/base.rb:125:in `perform'
/var/www/discourse/app/jobs/base.rb:180:in `perform'
/var/www/discourse/lib/scheduler/manager.rb:91:in `process_queue'
/var/www/discourse/lib/scheduler/manager.rb:37:in `block in initialize'
It seems to be trying to tombstone files that do not exist in Azure Blob Storage.
I canât seem to get a handle on what those files are yet⌠There are a bunch of posts that come in via email in that are put in queue and that I rejected. Perhaps the attached images inside those images are not yet copied into Azure? Just a thoughtâŚ
EDIT: Tracing thru the code a bit, it also can be a single upload that exists in uploads but no on the Azure Blob Storage. Therefore the task can be constantly trying to move it to tombstone, but constantly failing at the remove_file call, without actually deleting the uploads record.
So in other words, it can be caused by a single missing upload file over and over again⌠but how can I find out WHICH upload it is?
6 Likes
I am wondering, is there a way to trap the error and output more meaningful error messages?
For example:
These calls on blob_service will throw if the blob doesnât exist. However, the exception only returns a 403 (BlobNotFound) error, without any indication on what the non-existing path is.
Would it be a good idea to put try/catch around all blob_service calls so at least we get the context of the call in the error messages?
Alternatively, wouldnât it be a better idea to skip the tombstoning if the blob no longer exists instead of failing?
1 Like
Iâm adding improvements to the plugin this week, Iâll fix the behavior for this case as well, thanks for reporting!
7 Likes
Hey, I wanted to update one of the instructions:
in
Azure account
- If you donât have an Azure account yet, create one for free
- In your Microsoft Azure portal, go to Storage account and add a new account - make sure to select Blob storage under Account kind
- Go over to your newly created storage account and scroll down to find BLOB SERVICE. Click on Blobs, and on the upper left corner, click on + Container . Add a container into which uploads should be stored and set the public access level to Blob .
- Find Access keys in the storage account settings to fill in later when configuring Discourse.
3 Likes
What happens when you rebake without uploading the files yet? Will the posts rebake any differently compared to when the files are there?
Recently I notice that all uploads are failing with the following error:
undefined method `is_image?â for FileHelper:Class
All uploads are failing at this point. Any ideas?
This started happening after a recent rebuild so must be certain recent changes in core.
3 Likes
I have updated the plugin to apply the change from the core. Can you please try to rebuild again?
8 Likes
Seems fixed now. Thanks!
3 Likes
There is an annoying issue I would like to suggest a change for.
As I review my error logs, I keep getting a whole bunch of
Job exception: BlobNotFound (404): The specified blob does not exist.
RequestId:891fdac2-801e-0080-20b2-d25979000000
Time:2019-03-04T17:47:07.8968878Z
It is usually from here:
/var/www/discourse/plugins/discourse-azure-blob-storage/lib/azure_blob_store.rb:30:in `remove_file'
/var/www/discourse/lib/file_store/base_store.rb:20:in `remove_upload'
/var/www/discourse/app/models/upload.rb:105:in `block in destroy'
I believe that there is a blob that doesnât exist (a mismatch between database records and the actual blob store).
However, there is no way to know which blob is causing the problem. The RequestId looks like the unique id for Sidekiq.
I suggest trapping errors in Azure blob operations then appending the blob path to the error messages to facilitate trouble-shooting.
3 Likes
Sure sounds like a good change, care to try a PR. ATM we have no customers using the Azure storage plugin so it is hard to justify shifting resources to it.
5 Likes
OK. Let me give it a shot.
1 Like
Did one. Hopefully I get Ruby rightâŚ
https://github.com/discourse/discourse-azure-blob-storage/pull/10
4 Likes