### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 3, 2022, 1:08am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/2 "2022-04-03T01:08:22Z")
Do you have any plugins installed? If you have non standard plugins you should remove them.
---
### Author:  [@wusel](https://meta.discourse.org/u/wusel)
#### Post date: [April 3, 2022, 1:25am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/3 "2022-04-03T01:25:11Z")
Tried this already, the issue prevails ☹
```plaintext
## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
after_code:
- exec:
cd: $home/plugins
cmd:
- git clone https://github.com/discourse/docker_manager.git
## Any custom commands to run after building
```
EDIT: FTR:
Intended Plugins:
```plaintext
root@discourse:/var/discourse# grep -B5 "git clone" containers/app.yml-with-plugins
hooks:
after_code:
- exec:
cd: $home/plugins
cmd:
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-spoiler-alert.git
- git clone https://github.com/discourse/discourse-prometheus.git
- git clone https://github.com/discourse/discourse-solved.git
- git clone https://github.com/discourse/discourse-chat-integration.git
- git clone https://github.com/discourse/discourse-voting.git
- git clone https://github.com/discourse/discourse-checklist.git
- git clone https://github.com/discourse/discourse-whos-online.git
- git clone https://github.com/discourse/discourse-calendar.git
- git clone https://github.com/discourse/discourse-affiliate.git
- git clone https://github.com/discourse/discourse-reactions.git
- git clone https://github.com/discourse/discourse-surveys.git
```
The old installation has:
```plaintext
root@discourse-old:/var/discourse# grep -B5 "git clone" containers/app.yml
hooks:
after_code:
- exec:
cd: $home/plugins
cmd:
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-spoiler-alert.git
- git clone https://github.com/discourse/discourse-prometheus.git
- git clone https://github.com/discourse/discourse-solved.git
- git clone https://github.com/discourse/discourse-chat-integration.git
- git clone https://github.com/discourse/discourse-voting.git
- git clone https://github.com/discourse/discourse-checklist.git
- git clone https://github.com/davidtaylorhq/discourse-whos-online.git
```
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 3, 2022, 8:55am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/4 "2022-04-03T08:55:59Z")
This is a [standard install](https://meta.discourse.org/t/142537?silent=true)? If you have separate data container has it been rebuilt? If postgres up to date? (Though I think that there is still a check for this on the code)
---
### Author:  [@wusel](https://meta.discourse.org/u/wusel)
#### Post date: [April 3, 2022, 9:42am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/5 "2022-04-03T09:42:58Z")
Yes, it is a [standard install](https://meta.discourse.org/t/142537?silent=true); AFAICS there is only one container running. (Bring up a new VM, repoint DNS, apt update, apt dist-upgrade, reboot, git pull, ./discourse-setup, Web-based setup, upload backup, restore backup, re-enable mail, see the errors again.) Note that the old install was able to email the link to the backup, and sending test emails still works in the new install — it’s seemingly only posting-related mails that fail.
---
### Author:  [@wusel](https://meta.discourse.org/u/wusel)
#### Post date: [April 3, 2022, 10:04am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/6 "2022-04-03T10:04:50Z")
To be more precise, it’s a minimal Debian 10 installation, on which I put discourse:
```plaintext
root@discourse:~# history
1 apt update
2 apt dist-upgrade
3 reboot ; exit
4 git clone https://github.com/discourse/discourse_docker.git /var/discourse
5 apt install git rsync
6 git clone https://github.com/discourse/discourse_docker.git /var/discourse
7 cd /var/discourse
[attach /dev/vdb, fdisk, mkfs.ext4, mount as /var/lib/docker]
18 apt-get install git apt-transport-https ca-certificates curl gnupg2 software-properties-common -y
19 df -h
20 curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | apt-key add -
21 add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
22 apt-get update -y
23 apt-get install docker-ce -y
24 ./discourse-setup
```
And one container:
```plaintext
root@discourse:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ac408a70305d local_discourse/app "/sbin/boot" 12 hours ago Up 12 hours 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp app
```
Edit:
> [@pfaffman](#):
>
> If postgres up to date?
```plaintext
root@discourse:~# apt update
Hit:1 https://download.docker.com/linux/debian buster InRelease
Hit:2 http://deb.debian.org/debian buster InRelease
Get:3 http://deb.debian.org/debian buster-updates InRelease [51,9 kB]
Get:4 http://security.debian.org/debian-security buster/updates InRelease [65,4 kB]
Fetched 117 kB in 2s (60,5 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
```
The first restores actually failed, though. `database_restorer.rb` failed in `restore_dump` due to duplicated links in post\_id 3841. I ended up replacing these links in a post from 2017 by a screenshot of them on the old install and doing another backup; only then I was able to restore the backup on the new install. As you mentioned postgres, this happened during `CREATE INDEX` with `ERROR: could not create unique index "unique_post_links"`. Further info: `EXCEPTION: psql failed: DETAIL: Key (topic_id, post_id, url)=(1300, 3841, [redacted]) is duplicated`.
While I don’t think this is directly related, I though I should mention it.
---
### Author:  [@wusel](https://meta.discourse.org/u/wusel)
#### Post date: [April 10, 2022, 11:53pm UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/7 "2022-04-10T23:53:31Z")
So, if nobody knows an easy way how to fix it, let’s debug this properly. But I need your help, as Discourse is a rather complex Application using a bunch of technologies I’m not familiar with. So: How are Mails “send to” Sidekiq?
> [@](#):
>
> Error in Sidekiq:
>
> ```plaintext
> Wrapped NoMethodError: undefined method `value' for # Did you mean? values_at
>
> ```
Which component of Discourse is giving Sitekiq a method “value”?
With email notifications stopping to work after an Upgrade, Discourse is unfortunately rather useless now. Instead of immediate attention, topics now take days to get attention, if any, as people don’t do active polling in 2022. No notification ⇒ nothing happened ⇒ no need to check the site ☹
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 11, 2022, 1:03am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/8 "2022-04-11T01:03:17Z")
> [@wusel](#):
>
> `EXCEPTION: psql failed: DETAIL: Key (topic_id, post_id, url)=(1300, 3841, [redacted]) is duplicated` .
This looks like a corrupt index. I don’t think I’ve seen this on postgres 13. It sounds like you fixed those on an old site and the upgraded by backing up and restoring to s new site?
It looks like the issue has to do with something in the code that’s sending notifications, but they it’s an issue with sidekiq.
---
### Author:  [@wusel](https://meta.discourse.org/u/wusel)
#### Post date: [April 11, 2022, 4:17pm UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/9 "2022-04-11T16:17:36Z")
> [@pfaffman](#):
>
> This looks like a corrupt index. I don’t think I’ve seen this on postgres 13. It sounds like you fixed those on an old site and the upgraded by backing up and restoring to s new site?
As I said, old site is running, I did a backup and put that onto a fresh install, restore failed. I modified the post noted as the culprit until restore on the fresh install worked. Only to see having the issue with Sitekiq again/still.
The old site runs postgres 13 as well (but goes back several years, so it most likely didn’t start with that version 🙂)
```plaintext
root@discourse-old:/var/discourse# ./launcher enter app
x86_64 arch detected.
root@discourse-app:/var/www/discourse# psql --version
psql (PostgreSQL) 13.5 (Debian 13.5-1.pgdg110+1)
```
So, according to this [post’s closing comments](https://meta.discourse.org/t/sidekiq-jobs-are-failing-for-1-of-my-users/51143/12), Discourse’s Database can get corrupted — and get fixed.
Tried with a new user, it get its registration email properly. But notifications on replies to its posts, no; Sidekiq errors out.
So, to me this means that Discourse is giving some fautly information to Sidekiq when it instructs it to send out Notifications (in contrast to registration mail). How to debug further?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 11, 2022, 4:26pm UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/10 "2022-04-11T16:26:01Z")
> [@wusel](#):
>
> ````plaintext
> > Wrapped NoMethodError: undefined method `value' for # Did you mean? values_at
> > ```
>
> ````
OK, so if the indexes are repaired then this suggests to me that something is getting called with an array rather than a model that has `value`. The problem isnt’ with sidekiq, per se, but with the function that sidekiq is causing to get called.
So it sounds like something is getting called that’s returning an array rather than a single item, but I can’t guess just what. I think you’ll need to look in `/var/discourse/shared/standalone/logs/rails/production.log` (or something very much like that if my fingers or memory are failing me). Then you can look in those logs for that error (or cause it to happen again so it’ll be at the end of the file). You should get more information about what is failing there.
---
### Author:  [@wusel](https://meta.discourse.org/u/wusel)
#### Post date: [April 11, 2022, 4:33pm UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/11 "2022-04-11T16:33:15Z")
Thanks, but it’s not really telling much:
```plaintext
Started POST "/sidekiq/retries" for 185.39.142.187 at 2022-04-11 16:31:35 +0000
start
Started GET "/sidekiq/retries" for 185.39.142.187 at 2022-04-11 16:31:35 +0000
Rendered email/notification.html.erb (Duration: 42.8ms | Allocations: 4323)
Rendered layouts/email_template.html.erb (Duration: 0.3ms | Allocations: 29)
Job exception: undefined method `value' for #
Did you mean? values_at
fail
```
`shared/standalone/log/rails/production_errors.log` is empty.
---
### Author:  [@simonk](https://meta.discourse.org/u/simonk)
#### Post date: [April 12, 2022, 8:49am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/12 "2022-04-12T08:49:10Z")
Does the error show up in **Admin → Logs → Error Logs**? If so, you can get a full stack trace which might help.
---
### Author:  [@wusel](https://meta.discourse.org/u/wusel)
#### Post date: [April 12, 2022, 10:08am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/13 "2022-04-12T10:08:00Z")
Oh, nice — yes, it does:
So, if I understand this correctly,
```plaintext
433 def header_value(name)
434 header = @message.header[name]
435 return nil unless header
*436 header.value
437 end
```
is – in the Discource code? – where Sidekiq bails out?
This is called from …
```plaintext
228 MessageBuilder.custom_headers(SiteSetting.email_custom_headers).each do |key, _|
*229 value = header_value(key)
230
231 # Remove Auto-Submitted header for group private message emails, it does
232 # not make sense there and may hurt deliverability.
```
So it could be a custom header?
I indeed do have an entry there:
I’ve reset this to defaults, and … actually it seems that email notifications are indeed been sent out again, as verified by readers.
Thanks!
One question remains, though: Why is »Auto-Submitted: auto-generated|Precedence: bulk« causing this failure? It states that custom headers shall be seperated by »|«.
---
### Author:  [@simonk](https://meta.discourse.org/u/simonk)
#### Post date: [April 12, 2022, 10:37am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/14 "2022-04-12T10:37:18Z")
(Disclaimer: not a ruby programmer)
I think this is particularly nasty behaviour in the mail library that discourse is using. Here’s the `header_value` function:
>
As far as I can tell, `@message.header[name]` is calling this method:
[https://www.rubydoc.info/github/mikel/mail/Mail%2FHeader:[]](https://www.rubydoc.info/github/mikel/mail/Mail%2FHeader:%5B%5D)
> As per RFC, many fields can appear more than once, we will return a string of the value if there is only one header, or if there is more than one matching header, will return an array of values in order that they appear in the header ordered from top to bottom.
Discourse [automatically sets a `Precedence` header](https://github.com/discourse/discourse/blob/9c33f6de05703a42465139fda54773e5fa26c617/lib/email/sender.rb#L194), so because you’re adding one as well via the `email_custom_headers` setting, there are now **two** `Precedence` headers, and `@message.header["Precedence"]` is returning an array instead of a string.
I think this bug will be triggered any time `email_custom_headers` contains a header that already exists on the message object.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 12, 2022, 10:52am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/15 "2022-04-12T10:52:26Z")
This looks to me like what’s happening (I suggested they something was an array instead of a single item, but couldn’t imagine how that could be true) and is a bug.
I’ll change the title and category of this topic.
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [May 11, 2022, 3:47am UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/25 "2022-05-11T03:47:39Z")
I merged a fix for this today, if we detect a duplicate header we just use the one defined in Discourse core rather than the custom one from the site setting:
[https://github.com/discourse/discourse/pull/16711/](https://github.com/discourse/discourse/pull/16711/)
---
### Author:  [@pmusaraj](https://meta.discourse.org/u/pmusaraj)
#### Post date: [May 14, 2022, 12:00pm UTC](https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/26 "2022-05-14T12:00:18Z")
This topic was automatically closed after 2 days. New replies are no longer allowed.