Awesome, @vinothkannans can you review and merge ?
Hello @maja
I have some issue, which could be related with this plugin.
There is more info about this issue:
Could you check if the problem is with this plugin and try to fix them?
And maybe this one is related too:
Anything uploaded with short-url
automatically gets a 500.
Logs show this:
NoMethodError (undefined method `url_for' for #<FileStore::AzureStore:0x00007f4553fc2ae8>)
/var/www/discourse/app/controllers/uploads_controller.rb:107:in `show_short'
/var/www/discourse/app/controllers/uploads_controller.rb:107:in `show_short'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:194:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:132:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:134:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionview-5.2.3/lib/action_view/rendering.rb:32:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.0.2/lib/mini_profiler/profiling_methods.rb:78:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:191:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:252:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-protection-2.0.5/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:68:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:25:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/var/www/discourse/lib/content_security_policy/middleware.rb:12:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/logster-2.3.2/lib/logster/middleware/reporter.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:28:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:18:in `call'
/var/www/discourse/config/initializers/100-silence_logger.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/www/discourse/lib/middleware/enforce_hostname.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.0.2/lib/mini_profiler/profiler.rb:171:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-2.2.2/lib/message_bus/rack/middleware.rb:57:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:169:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:605:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:700:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:548:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:144:in `start'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<main>'
Iâm quite sure https://github.com/discourse/discourse-azure-blob-storage/blob/master/lib/azure_blob_store.rb
is missing an implementation for url_for
âŚ
We had a PR for that which unfortunately wasnât merged, but there should be some great feedback in there in case anyone wants to start working on it.
https://github.com/discourse/discourse-azure-blob-storage/pull/11
As I mentioned in https://github.com/discourse/discourse-azure-blob-storage/pull/11#discussion_r312018993, as long as private uploads are not needed, the 3 new lines of the PR should actually be enough to fix the problem.
So⌠er⌠how should I pull this version with the PR instead?
Alternatively, how can I get the actual URL from the short-url
? Is there a mapping table somewhere?
Because right now no attachment can be downloaded and it is frustratingâŚ
OK, pulling from
git clone --single-branch --branch implement-url-for https://github.com/localjo/discourse-azure-blob-storage
works fine now. Will report after more testing.
This thing keeps coming up for the past few days, which I believe may be related to this issue:
/var/www/discourse/lib/file_store/base_store.rb:148:in `not_implemented'
/var/www/discourse/lib/file_store/base_store.rb:42:in `download_url'
/var/www/discourse/lib/cooked_post_processor.rb:386:in `add_lightbox!'
/var/www/discourse/lib/cooked_post_processor.rb:350:in `convert_to_link!'
/var/www/discourse/lib/cooked_post_processor.rb:682:in `block in post_process_images'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.4/lib/nokogiri/xml/node_set.rb:238:in `block in each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.4/lib/nokogiri/xml/node_set.rb:237:in `upto'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.4/lib/nokogiri/xml/node_set.rb:237:in `each'
/var/www/discourse/lib/cooked_post_processor.rb:679:in `post_process_images'
/var/www/discourse/lib/cooked_post_processor.rb:42:in `block in post_process'
/var/www/discourse/lib/distributed_mutex.rb:31:in `block in synchronize'
/var/www/discourse/lib/distributed_mutex.rb:27:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:27:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:12:in `synchronize'
/var/www/discourse/lib/cooked_post_processor.rb:38:in `post_process'
/var/www/discourse/app/jobs/regular/process_post.rb:26:in `execute'
/var/www/discourse/app/jobs/base.rb:232:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.0.7/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:221:in `block in perform'
/var/www/discourse/app/jobs/base.rb:217:in `each'
/var/www/discourse/app/jobs/base.rb:217:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/var/www/discourse/lib/sidekiq/pausable.rb:138:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
download_url
must be another function not implemented by the Azure Blob plugin.
EDIT:
OK, the impacts of missing download_url
are making themselves known.
- The image post process job keeps failing
- No images are optimized
- No images are lightboxed
- Topic List Preview canât pick up the preview image
Can some kind soul (@gerhard?) please implement download_url
?
EDIT 2:
@gerhard, can this be as simple as putting in:
def download_url(upload)
return unless upload
"#{upload.short_path}"
end
If would love to clone the repo to test it out, but Discourse has become such a huge monster now that I am not sure if anything is going to break!
EDIT 3:
Iâve sent the following PR to localjo/discourse-azure-blob-storage
:
https://github.com/localjo/discourse-azure-blob-storage/pull/1
It seems to resolve most of the visible problems without any visible adverse effectsâŚ
Is it possible to make some changes to allow Discourse to use Azure blob storage for backups?
Iâm running into this same error. Was your PR incorporated? If not, can you share the changes made so I can see if I can implement on our site? I canât access the github url shown.
Thanks!
The base repo seems to have gone away.
You can try pulling from this:
git clone --single-branch --branch implement-url-for https://github.com/schungx/discourse-azure-blob-storage.git
This has my changes. However, I believe something else broke, which I havenât fixed yet.
A PR I made to the official repo:
There appear to be massive changes recently on the S3 storage engine (which azure-blob-storage
steals from) to handle secured uploads.
On the way, certain handling on cooking posts breaks. It is generating HTML like this:
<img
src="https://xxx.com/t/topic/3493"
srcset=", 1.5x, 2x"
style="position: absolute; top: 0px; left: 0px; width: 678px; height: 388px;"
class="d-lazyload d-lazyload-hidden"
>
Notice that:
-
The
src
is set to the link of the topic instead of the URL of the image. -
The
srcset
URLâs all disappear (returning blanks probably). -
None of the optimized images are actually generated (probably due to the URLâs being blank).
This is some HTML that works fine:
<img
src="https://xxx.blob.core.windows.net/support/optimized/2X/2/2332e15b99ee86620e2f890d38aaceb16954cc8c_2_375x500.jpeg"
alt="20200113_163717" data-base62-sha1="51nILqcSQo5kGA4iz67LGoQmSpm"
class="d-lazyload"
srcset="//xxx.blob.core.windows.net/support/optimized/2X/2/2332e15b99ee86620e2f890d38aaceb16954cc8c_2_375x500.jpeg, //xxx.blob.core.windows.net/support/optimized/2X/2/2332e15b99ee86620e2f890d38aaceb16954cc8c_2_562x750.jpeg 1.5x, //xxx.blob.core.windows.net/support/optimized/2X/2/2332e15b99ee86620e2f890d38aaceb16954cc8c_2_750x1000.jpeg 2x"
width="375" height="500"
>
For this image, everything worked fine.
The code to generate the optimized images lay in UrlHelper.cook_url
which underwent some serious changes to handle secured uploads to S3.
Thanks for the info, I ended up fixing it based on your comments.
For the recent changes, I discovered that too. That was an easy fix, you just have to add the secure (4th) parameter to the optimize function in the plugin. Not sure if other changes are needed but this fixed the immediate issue you noted.
Ever since upgrading about four days ago, all optimized_images
has the url
field set to null
instead of the correct image url.
This causes Discourse to fail in horrible ways, because upload.thumbnail
will return a valid record, but the url
is null. Shouldnât there be a guard somewhere to check against this?
Which means that this line:
sets img["src"]
to null. I am sure that wonât make anything happy.
All the images in srcset
are set to empty because:
t.url
ends up being null.
Then everything ends up being exactly the way it is. Iâm quite sure that, somewhere down the line, if img["src"]
is null, then it gets set to the URL of the topic itself.
Therefore, the culprit is the fact that optimized images are NOT being created (but the records are, with null url
). There are no error messages.
Any post with images that have valid optimized images (with valid url
continue to work just fine, even when rebuilding HTML).
Any image that has been uploaded before (with valid optimized images), when uploaded again, continue to work just fine.
EDIT: OK, I added secure: false
to store_optimized_image
and everything seems to work fine now.
I still believe something must be done to handle situations where OptimizedImage
has url
= ""
(not null
).
This will STILL happen, but it seems only for some images and not others. For certain images of those kinds, small versions created when using the Topics List Preview
plugin will fail to create and thus leave records with url=""
.
These typically wonât show up because responsive images only start from 1x of minimum size and goes up (by default) to 2x. Therefore the small optimized images are never reached. But still, it is a worry that some day itâll trip upâŚ
Not to say that a preview image wonât show up on the topics list when Topics List Preview
plugin is installed.
Got an error upgrading Discourse due to a faraday
conflict:
I, [2020-05-22T05:46:54.081847 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
rake aborted!
Gem::ConflictError: Unable to activate faraday_middleware-0.11.0, because faraday-1.0.1 conflicts with faraday (>= 0.7.4, < 1.0)
/var/www/discourse/lib/plugin_gem.rb:20:in `load'
/var/www/discourse/lib/plugin/instance.rb:635:in `gem'
/var/www/discourse/plugins/discourse-azure-blob-storage/plugin.rb:10:in `activate!'
/var/www/discourse/lib/plugin/instance.rb:541:in `instance_eval'
/var/www/discourse/lib/plugin/instance.rb:541:in `activate!'
lib/discourse.rb:224:in `block in activate_plugins!'
lib/discourse.rb:221:in `each'
lib/discourse.rb:221:in `activate_plugins!'
/var/www/discourse/config/application.rb:297:in `block in <class:Application>'
/var/www/discourse/lib/plugin_initialization_guard.rb:5:in `plugin_initialization_guard'
/var/www/discourse/config/application.rb:296:in `<class:Application>'
/var/www/discourse/config/application.rb:65:in `<module:Discourse>'
/var/www/discourse/config/application.rb:64:in `<top (required)>'
/var/www/discourse/Rakefile:7:in `require'
/var/www/discourse/Rakefile:7:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Does discourse-azure-blob-storage
use faraday
?
Yes, indeed it does
https://github.com/discourse/discourse-azure-blob-storage/blob/67d13a285c17b6c4ca2d9154a8bed0a4f0fd5733/plugin.rb#L10
Is this plugin still broken?
Anyone currently using this plugin for uploads to Azure Blob Storage? If yes, is it stable? Please share your experience.