Awesome, @vinothkannans can you review and merge ?
Hello @maja
I have some issue, which could be related with this plugin.
There is more info about this issue:
Could you check if the problem is with this plugin and try to fix them?
And maybe this one is related too:
Anything uploaded with short-url automatically gets a 500.
Logs show this:
NoMethodError (undefined method `url_for' for #<FileStore::AzureStore:0x00007f4553fc2ae8>)
/var/www/discourse/app/controllers/uploads_controller.rb:107:in `show_short'
/var/www/discourse/app/controllers/uploads_controller.rb:107:in `show_short'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:194:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:132:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:134:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionview-5.2.3/lib/action_view/rendering.rb:32:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.0.2/lib/mini_profiler/profiling_methods.rb:78:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:191:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:252:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-protection-2.0.5/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:68:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:25:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/var/www/discourse/lib/content_security_policy/middleware.rb:12:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/logster-2.3.2/lib/logster/middleware/reporter.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:28:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:18:in `call'
/var/www/discourse/config/initializers/100-silence_logger.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/www/discourse/lib/middleware/enforce_hostname.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.0.2/lib/mini_profiler/profiler.rb:171:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-2.2.2/lib/message_bus/rack/middleware.rb:57:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:169:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:605:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:700:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:548:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:144:in `start'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<main>'
I’m quite sure https://github.com/discourse/discourse-azure-blob-storage/blob/master/lib/azure_blob_store.rb is missing an implementation for url_for…
We had a PR for that which unfortunately wasn’t merged, but there should be some great feedback in there in case anyone wants to start working on it.
As I mentioned in FIX: Implement url_for by localjo · Pull Request #11 · discourse/discourse-azure-blob-storage · GitHub, as long as private uploads are not needed, the 3 new lines of the PR should actually be enough to fix the problem.
So… er… how should I pull this version with the PR instead?
Alternatively, how can I get the actual URL from the short-url? Is there a mapping table somewhere?
Because right now no attachment can be downloaded and it is frustrating…
OK, pulling from
git clone --single-branch --branch implement-url-for https://github.com/localjo/discourse-azure-blob-storage
works fine now. Will report after more testing.
This thing keeps coming up for the past few days, which I believe may be related to this issue:
/var/www/discourse/lib/file_store/base_store.rb:148:in `not_implemented'
/var/www/discourse/lib/file_store/base_store.rb:42:in `download_url'
/var/www/discourse/lib/cooked_post_processor.rb:386:in `add_lightbox!'
/var/www/discourse/lib/cooked_post_processor.rb:350:in `convert_to_link!'
/var/www/discourse/lib/cooked_post_processor.rb:682:in `block in post_process_images'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.4/lib/nokogiri/xml/node_set.rb:238:in `block in each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.4/lib/nokogiri/xml/node_set.rb:237:in `upto'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.4/lib/nokogiri/xml/node_set.rb:237:in `each'
/var/www/discourse/lib/cooked_post_processor.rb:679:in `post_process_images'
/var/www/discourse/lib/cooked_post_processor.rb:42:in `block in post_process'
/var/www/discourse/lib/distributed_mutex.rb:31:in `block in synchronize'
/var/www/discourse/lib/distributed_mutex.rb:27:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:27:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:12:in `synchronize'
/var/www/discourse/lib/cooked_post_processor.rb:38:in `post_process'
/var/www/discourse/app/jobs/regular/process_post.rb:26:in `execute'
/var/www/discourse/app/jobs/base.rb:232:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.0.7/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:221:in `block in perform'
/var/www/discourse/app/jobs/base.rb:217:in `each'
/var/www/discourse/app/jobs/base.rb:217:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/var/www/discourse/lib/sidekiq/pausable.rb:138:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
download_url must be another function not implemented by the Azure Blob plugin.
EDIT:
OK, the impacts of missing download_url are making themselves known.
- The image post process job keeps failing
- No images are optimized
- No images are lightboxed
- Topic List Preview can’t pick up the preview image
Can some kind soul (@gerhard?) please implement download_url?
EDIT 2:
@gerhard, can this be as simple as putting in:
def download_url(upload)
return unless upload
"#{upload.short_path}"
end
If would love to clone the repo to test it out, but Discourse has become such a huge monster now that I am not sure if anything is going to break!
EDIT 3:
I’ve sent the following PR to localjo/discourse-azure-blob-storage:
https://github.com/localjo/discourse-azure-blob-storage/pull/1
It seems to resolve most of the visible problems without any visible adverse effects…
是否可以对 Discourse 进行一些修改,使其能够使用 Azure Blob 存储进行备份?
我遇到了同样的错误。你的 PR 被合并了吗?如果没有,能否分享所做的更改,以便我看看是否能在我们的网站上实现?我无法访问显示的 GitHub 链接。
谢谢!
基础仓库似乎已经不存在了。
你可以尝试从以下地址拉取:
git clone --single-branch --branch implement-url-for https://github.com/schungx/discourse-azure-blob-storage.git
这里包含我的修改。不过,我认为还有其他问题尚未修复。
我提交给官方仓库的 PR:
最近,S3 存储引擎(azure-blob-storage 从中“借用”)似乎发生了重大变更,以处理安全上传。
在此过程中,帖子处理中的某些环节出现了问题。它生成的 HTML 如下:
<img
src="https://xxx.com/t/topic/3493"
srcset=", 1.5x, 2x"
style="position: absolute; top: 0px; left: 0px; width: 678px; height: 388px;"
class="d-lazyload d-lazyload-hidden"
>
请注意以下几点:
-
src被设置为帖子的链接,而不是图片的 URL。 -
srcset中的所有 URL 都消失了(可能返回了空白)。 -
实际上没有生成任何优化后的图片(很可能是因为 URL 为空)。
下面这段 HTML 可以正常工作:
<img
src="https://xxx.blob.core.windows.net/support/optimized/2X/2/2332e15b99ee86620e2f890d38aaceb16954cc8c_2_375x500.jpeg"
alt="20200113_163717" data-base62-sha1="51nILqcSQo5kGA4iz67LGoQmSpm"
class="d-lazyload"
srcset="//xxx.blob.core.windows.net/support/optimized/2X/2/2332e15b99ee86620e2f890d38aaceb16954cc8c_2_375x500.jpeg, //xxx.blob.core.windows.net/support/optimized/2X/2/2332e15b99ee86620e2f890d38aaceb16954cc8c_2_562x750.jpeg 1.5x, //xxx.blob.core.windows.net/support/optimized/2X/2/2332e15b99ee86620e2f890d38aaceb16954cc8c_2_750x1000.jpeg 2x"
width="375" height="500"
>
对于这张图片,一切正常。
生成优化图片的代码位于 UrlHelper.cook_url 中,该代码为了处理 S3 的安全上传经历了重大变更。
感谢提供信息,我根据您的评论最终修复了问题。
关于最近的更改,我也发现了这一点。修复起来很简单,只需在插件的 optimize 函数中添加安全(第 4 个)参数即可。不确定是否还需要其他更改,但这已解决了您指出的直接问题。
自从大约四天前升级以来,所有 optimized_images 的 url 字段都被设置为 null,而不是正确的图片 URL。
这导致 Discourse 以非常糟糕的方式失败,因为 upload.thumbnail 会返回一条有效记录,但 url 为 null。难道不应该在某处添加一个检查机制来应对这种情况吗?
这意味着这一行:
会将 img["src"] 设置为 null。我敢肯定这不会让任何地方感到满意。
所有 srcset 中的图片都被设置为空,原因如下:
t.url 最终变成了 null。
然后一切最终都变成了现在的样子。我非常确定,在某个环节,如果 img["src"] 为 null,它就会被设置为该主题本身的 URL。
因此,罪魁祸首是优化后的图片并未被创建(但记录确实被创建了,只是 url 为 null)。没有任何错误信息。
任何包含具有有效优化图片(即具有有效 url)的帖子都能正常工作,即使在重新构建 HTML 时也是如此。
任何之前上传过(具有有效优化图片)的图片,在再次上传时也能正常工作。
编辑:好的,我已在 store_optimized_image 中添加了 secure: false,现在一切似乎都正常工作了。
我仍然认为必须采取措施来处理 OptimizedImage 的 url 为空字符串 ""(而非 null)的情况。
这种情况仍会发生,但似乎仅影响部分图片而非全部。对于某些此类图片,在使用 Topics List Preview 插件时生成的小尺寸版本会创建失败,从而导致留下 url="" 的记录。
这些记录通常不会显示,因为响应式图片默认从最小尺寸的 1 倍开始,最高到 2 倍。因此,那些过小的优化图片永远不会被用到。但即便如此,我仍担心未来某天会引发问题……
这并不意味着安装了 Topics List Preview 插件后,主题列表中就完全不会显示预览图片。
由于 faraday 冲突导致 Discourse 升级时出错:
I, [2020-05-22T05:46:54.081847 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
rake aborted!
Gem::ConflictError: Unable to activate faraday_middleware-0.11.0, because faraday-1.0.1 conflicts with faraday (>= 0.7.4, < 1.0)
/var/www/discourse/lib/plugin_gem.rb:20:in `load'
/var/www/discourse/lib/plugin/instance.rb:635:in `gem'
/var/www/discourse/plugins/discourse-azure-blob-storage/plugin.rb:10:in `activate!'
/var/www/discourse/lib/plugin/instance.rb:541:in `instance_eval'
/var/www/discourse/lib/plugin/instance.rb:541:in `activate!'
lib/discourse.rb:224:in `block in activate_plugins!'
lib/discourse.rb:221:in `each'
lib/discourse.rb:221:in `activate_plugins!'
/var/www/discourse/config/application.rb:297:in `block in <class:Application>'
/var/www/discourse/lib/plugin_initialization_guard.rb:5:in `plugin_initialization_guard'
/var/www/discourse/config/application.rb:296:in `<class:Application>'
/var/www/discourse/config/application.rb:65:in `<module:Discourse>'
/var/www/discourse/config/application.rb:64:in `<top (required)>'
/var/www/discourse/Rakefile:7:in `require'
/var/www/discourse/Rakefile:7:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
discourse-azure-blob-storage 是否使用了 faraday?
是的,确实如此
这个插件仍然有问题吗?
有人目前正在使用这个插件将文件上传到 Azure Blob Storage 吗?如果有,它稳定吗?请分享您的使用体验。