Digest send failures since update (NoMethodError)

Just updated to 2.6.0.beta5 ( 7fa6aca9db ) this afternoon, and my dashboard alerts me to failed jobs:

I see no emails(*) have gone out, and I have sidekiq jobs in retry status:

Jobs::UserEmail	
{"type"=>"digest", "user_id"=>206, "current_site_id"=>"default"}
Jobs::HandledExceptionWrapper: Wrapped NoMethodError: undefined method `value' for nil:NilClass

I tried to test email using Preview Summary and that errors out with

Server Error
while trying to load /admin/email/preview-digest.json
Error code: 500 error

My app.yml hasn’t changed. I use Mailgun, and all was well before the update.

(*) Edit: some 12 digests have gone out, but “There are 265 email jobs that failed.” Evidently the failure is sensitive to the content of the digest.

3 Likes

Do you have any third party plugins installed?

1 Like

No: only plugin is discourse-data-explorer.

1 Like

I think some data value seems to be nil or empty when it shouldn’t be - or it shouldn’t matter. Two tracebacks from the logs, one for preview and one for the sidekiq job, I think:

/var/www/discourse/lib/pretty_text.rb:440:in `block in strip_secure_media'
nokogiri-1.10.10/lib/nokogiri/xml/node_set.rb:238:in `block in each'
nokogiri-1.10.10/lib/nokogiri/xml/node_set.rb:237:in `upto'
nokogiri-1.10.10/lib/nokogiri/xml/node_set.rb:237:in `each'
/var/www/discourse/lib/pretty_text.rb:434:in `strip_secure_media'
/var/www/discourse/lib/email/styles.rb:347:in `replace_secure_media_urls'
/var/www/discourse/lib/email/styles.rb:281:in `to_html'
/var/www/discourse/lib/email/renderer.rb:31:in `html'
/var/www/discourse/lib/email/sender.rb:70:in `send'
/var/www/discourse/app/jobs/regular/user_email.rb:70:in `send_user_email'
/var/www/discourse/app/jobs/regular/user_email.rb:25:in `execute'
/var/www/discourse/app/jobs/base.rb:232:in `block (2 levels) in perform'
rails_multisite-2.5.0/lib/rails_multisite/connection_management.rb:76:in `with_connection'
/var/www/discourse/app/jobs/base.rb:221:in `block in perform'
/var/www/discourse/app/jobs/base.rb:217:in `each'
/var/www/discourse/app/jobs/base.rb:217:in `perform'
sidekiq-6.1.2/lib/sidekiq/processor.rb:196:in `execute_job'
sidekiq-6.1.2/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
sidekiq-6.1.2/lib/sidekiq/middleware/chain.rb:138:in `block in invoke'
/var/www/discourse/lib/sidekiq/pausable.rb:138:in `call'
sidekiq-6.1.2/lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
sidekiq-6.1.2/lib/sidekiq/middleware/chain.rb:143:in `invoke'
sidekiq-6.1.2/lib/sidekiq/processor.rb:163:in `block in process'
sidekiq-6.1.2/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
sidekiq-6.1.2/lib/sidekiq/job_retry.rb:111:in `local'
sidekiq-6.1.2/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
sidekiq-6.1.2/lib/sidekiq.rb:38:in `block in <module:Sidekiq>'
sidekiq-6.1.2/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
sidekiq-6.1.2/lib/sidekiq/processor.rb:257:in `stats'
sidekiq-6.1.2/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
sidekiq-6.1.2/lib/sidekiq/job_logger.rb:13:in `call'
sidekiq-6.1.2/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
sidekiq-6.1.2/lib/sidekiq/job_retry.rb:78:in `global'
sidekiq-6.1.2/lib/sidekiq/processor.rb:124:in `block in dispatch'
sidekiq-6.1.2/lib/sidekiq/logger.rb:10:in `with'
sidekiq-6.1.2/lib/sidekiq/job_logger.rb:33:in `prepare'
sidekiq-6.1.2/lib/sidekiq/processor.rb:123:in `dispatch'
sidekiq-6.1.2/lib/sidekiq/processor.rb:162:in `process'
sidekiq-6.1.2/lib/sidekiq/processor.rb:78:in `process_one'
sidekiq-6.1.2/lib/sidekiq/processor.rb:68:in `run'
sidekiq-6.1.2/lib/sidekiq/util.rb:15:in `watchdog'
sidekiq-6.1.2/lib/sidekiq/util.rb:24:in `block in safe_thread'

and

lib/pretty_text.rb:440:in `block in strip_secure_media'
nokogiri (1.10.10) lib/nokogiri/xml/node_set.rb:238:in `block in each'
nokogiri (1.10.10) lib/nokogiri/xml/node_set.rb:237:in `upto'
nokogiri (1.10.10) lib/nokogiri/xml/node_set.rb:237:in `each'
lib/pretty_text.rb:434:in `strip_secure_media'
lib/email/styles.rb:347:in `replace_secure_media_urls'
lib/email/styles.rb:281:in `to_html'
lib/email/renderer.rb:31:in `html'
app/controllers/admin/email_controller.rb:93:in `preview_digest'
actionpack (6.0.3.3) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
actionpack (6.0.3.3) lib/abstract_controller/base.rb:195:in `process_action'
actionpack (6.0.3.3) lib/action_controller/metal/rendering.rb:30:in `process_action'
actionpack (6.0.3.3) lib/abstract_controller/callbacks.rb:42:in `block in process_action'
activesupport (6.0.3.3) lib/active_support/callbacks.rb:112:in `block in run_callbacks'
app/controllers/application_controller.rb:357:in `block in with_resolved_locale'
i18n (1.8.5) lib/i18n.rb:313:in `with_locale'
app/controllers/application_controller.rb:357:in `with_resolved_locale'
activesupport (6.0.3.3) lib/active_support/callbacks.rb:121:in `block in run_callbacks'
activesupport (6.0.3.3) lib/active_support/callbacks.rb:139:in `run_callbacks'
actionpack (6.0.3.3) lib/abstract_controller/callbacks.rb:41:in `process_action'
actionpack (6.0.3.3) lib/action_controller/metal/rescue.rb:22:in `process_action'
actionpack (6.0.3.3) lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
activesupport (6.0.3.3) lib/active_support/notifications.rb:180:in `block in instrument'
activesupport (6.0.3.3) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.0.3.3) lib/active_support/notifications.rb:180:in `instrument'
actionpack (6.0.3.3) lib/action_controller/metal/instrumentation.rb:32:in `process_action'
actionpack (6.0.3.3) lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
activerecord (6.0.3.3) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
actionpack (6.0.3.3) lib/abstract_controller/base.rb:136:in `process'
actionview (6.0.3.3) lib/action_view/rendering.rb:39:in `process'
rack-mini-profiler (2.2.0) lib/mini_profiler/profiling_methods.rb:111:in `block in profile_method'
actionpack (6.0.3.3) lib/action_controller/metal.rb:190:in `dispatch'
actionpack (6.0.3.3) lib/action_controller/metal.rb:254:in `dispatch'
actionpack (6.0.3.3) lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
actionpack (6.0.3.3) lib/action_dispatch/routing/route_set.rb:33:in `serve'
actionpack (6.0.3.3) lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
actionpack (6.0.3.3) lib/action_dispatch/routing/mapper.rb:48:in `serve'
actionpack (6.0.3.3) lib/action_dispatch/journey/router.rb:49:in `block in serve'
actionpack (6.0.3.3) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (6.0.3.3) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (6.0.3.3) lib/action_dispatch/routing/route_set.rb:834:in `call'
lib/middleware/omniauth_bypass_middleware.rb:68:in `call'
rack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'
rack (2.2.3) lib/rack/conditional_get.rb:27:in `call'
rack (2.2.3) lib/rack/head.rb:12:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:354:in `call'
rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
actionpack (6.0.3.3) lib/action_dispatch/middleware/cookies.rb:648:in `call'
actionpack (6.0.3.3) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
activesupport (6.0.3.3) lib/active_support/callbacks.rb:101:in `run_callbacks'
actionpack (6.0.3.3) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
actionpack (6.0.3.3) lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
actionpack (6.0.3.3) lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
actionpack (6.0.3.3) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
logster (2.9.4) lib/logster/middleware/reporter.rb:43:in `call'
railties (6.0.3.3) lib/rails/rack/logger.rb:37:in `call_app'
railties (6.0.3.3) lib/rails/rack/logger.rb:28:in `call'
config/initializers/100-quiet_logger.rb:23:in `call'
config/initializers/100-silence_logger.rb:31:in `call'
actionpack (6.0.3.3) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
actionpack (6.0.3.3) lib/action_dispatch/middleware/request_id.rb:27:in `call'
lib/middleware/enforce_hostname.rb:22:in `call'
rack (2.2.3) lib/rack/method_override.rb:24:in `call'
actionpack (6.0.3.3) lib/action_dispatch/middleware/executor.rb:14:in `call'
rack (2.2.3) lib/rack/sendfile.rb:110:in `call'
actionpack (6.0.3.3) lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
rack-mini-profiler (2.2.0) lib/mini_profiler/profiler.rb:368:in `call'
message_bus (3.3.4) lib/message_bus/rack/middleware.rb:61:in `call'
lib/middleware/request_tracker.rb:176:in `call'
railties (6.0.3.3) lib/rails/engine.rb:527:in `call'
railties (6.0.3.3) lib/rails/railtie.rb:190:in `public_send'
railties (6.0.3.3) lib/rails/railtie.rb:190:in `method_missing'
rack (2.2.3) lib/rack/urlmap.rb:74:in `block in call'
rack (2.2.3) lib/rack/urlmap.rb:58:in `each'
rack (2.2.3) lib/rack/urlmap.rb:58:in `call'
unicorn (5.7.0) lib/unicorn/http_server.rb:632:in `process_client'
unicorn (5.7.0) lib/unicorn/http_server.rb:728:in `worker_loop'
unicorn (5.7.0) lib/unicorn/http_server.rb:548:in `spawn_missing_workers'
unicorn (5.7.0) lib/unicorn/http_server.rb:144:in `start'
unicorn (5.7.0) bin/unicorn:128:in `<top (required)>'
vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load'
vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<main>'

Do you have secure media enabled in the site settings?

No, secure media is disabled (which is the default).

FYI, I see secure media allow embed images in emails is enabled, but that’s also the default.

https://github.com/discourse/discourse/blob/d824a5edd2a4e35c273ede3f9d2cd7696851f260/lib/email/styles.rb#L281

@martin why do we call replace_secure_media_urls unconditionally? Can we add a guard that checks against secure media ?

2 Likes

Just to note: my update failed when run from the browser, so I did a rebuild from the command line.

(Just in case that means this installation will somehow have got to a different place compared to one which was successfully updated. I have another forum on a different host, which was successfully updated, which isn’t failing to send emails or prepare a Preview Summary.)

As I previously updated on the 19th, we’re seeing something introduced since then.

This is the line from the stack traces: attributes["srcset"] is perhaps a novelty? Or at any rate, value is not there.

https://github.com/discourse/discourse/blob/3655062c6061889c521120097e6271c47d72af50/lib/pretty_text.rb#L440

A commit from yesterday might explain why not many people are yet seeing this.
https://github.com/discourse/discourse/commit/3655062c6061889c521120097e6271c47d72af50#diff-92f9b2598b8d5259f3a041bc92f51317925dcf3660ad5f78710cdc01f2c02d82R440

1 Like

Same problem, what I have to do?

1 Like

I’m hoping @martin (or someone) will revert one or other of the changes noted upthread, or add the guard as @Falco suggests.

I have the same issue.

3 Likes

Same issue here too…

1 Like

I wonder if this could be a workaround: if the problem is the handling of (some) images in the digest emails, and if we’re happy to send out digests temporarily without images, we can set short email length to a very large number, so strip images from short emails will strip out all images.

Edit: no, I tried it and still the sidekiq jobs are piling up. No emails went out during the experiment, despite many jobs rerunning themselves.

PR with proposed fix open

https://github.com/discourse/discourse/pull/11110?v=2

6 Likes

same problem here, hope that the PR will implemented soon.

2 Likes

Can confirm…also having the same issue.

PR is merged into latest, please update and report back.

5 Likes

Working for me now, thanks!

2 Likes

Looking good for me so far, thanks a lot.

3 Likes