This causes mail delivery to fail, and discourse-doctor confirms the mangled password.
Notes
• Quoting the password explicitly at the prompt does not change the outcome.
• Password contains special characters (@, !), but the problem is not just YAML quoting: the username string is literally concatenated at the start of the password.
• Reproducible across multiple runs of ./discourse-setup.
Yeah. That’s the issue. There are multiple levels of escaping that need to happen, like when bash reads the value, when bash hands the value to sed, when sed replaces it, and then, maybe, when the yml file gets it. It’s a known issue:
edits containers/app.yml via Ruby’s YAML (Psych), so YAML quoting/escaping is handled by a real parser
deletes the per-key SMTP vars to avoid contradictions
apply with git apply -p0 in the discourse_docker repo.
Patch 1 -
discourse-setup
(write SMTP using Ruby YAML, not sed)
--- a/discourse-setup
+++ b/discourse-setup
@@ -867,6 +867,77 @@ write_smtp_settings() {
local app_yml="containers/app.yml"
[[ -f "$app_yml" ]] || die "Cannot find $app_yml. Did you run bootstrap?"
+ # Build a URL-encoded SMTP_URL using Python stdlib (no shell escaping games)
+ urlencode() {
+ python3 - <<'PY'
+import sys, urllib.parse
+print(urllib.parse.quote(sys.stdin.read().strip(), safe='._~-'))
+PY
+ }
+
+ # IMPORTANT: read vars without backslash mangling
+ # (these come from earlier prompts; just ensure -r is used at prompt time)
+ local addr="$smtp_address"
+ local port="$smtp_port"
+ local user_enc pass_enc
+ user_enc="$(printf '%s' "$smtp_user" | urlencode)"
+ pass_enc="$(printf '%s' "$smtp_password" | urlencode)"
+ local smtp_url="smtp://${user_enc}:${pass_enc}@${addr}:${port}"
+
+ # Use Ruby to safely load/modify/dump YAML (kills 3 layers of escaping)
+ ruby - <<'RUBY' "$app_yml" "$smtp_url"
+require "yaml"
+require "psych"
+path, url = ARGV
+doc = YAML.safe_load(File.read(path), permitted_classes: [], aliases: true) || {}
+
+# Ensure top-level structure is a Hash and has env
+unless doc.is_a?(Hash)
+ abort "containers/app.yml does not parse to a Hash"
+end
+doc["env"] ||= {}
+env = doc["env"]
+
+# Write single-line SMTP_URL; remove per-key vars to avoid conflicts
+env["SMTP_URL"] = url
+%w[DISCOURSE_SMTP_ADDRESS DISCOURSE_SMTP_PORT DISCOURSE_SMTP_USER_NAME DISCOURSE_SMTP_PASSWORD].each { |k| env.delete(k) }
+
+# Dump back. (Psych preserves strings safely quoted as needed.)
+File.write(path, Psych.dump(doc))
+RUBY
+
+ # quick sanity check for the classic "password prefixed by username" failure
+ python3 - <<'PY'
+import re, sys
+y = open("containers/app.yml","r",encoding="utf-8").read()
+m = re.search(r'^\s*SMTP_URL:\s*(?:"|\')?([^\r\n"\']+)', y, re.M)
+assert m, "SMTP_URL missing after write"
+creds = m.group(1).split('@',1)[0].split('://',1)[-1]
+assert ":" in creds, "SMTP_URL creds missing ':'"
+u, p = creds.split(':',1)
+assert not p.startswith(u), "Password appears prefixed by username"
+print("SMTP_URL looks sane.")
+PY
+}
+
- # Write per-key SMTP entries (address/port/username/password)
- # (legacy: performed via sed substitutions)
- # NOTE: historically fragile with special chars
- update_setting_yaml "DISCOURSE_SMTP_ADDRESS" "$smtp_address"
- update_setting_yaml "DISCOURSE_SMTP_PORT" "$smtp_port"
- update_setting_yaml "DISCOURSE_SMTP_USER_NAME" "$smtp_user"
- update_setting_yaml "DISCOURSE_SMTP_PASSWORD" "$smtp_password"
-}
+ # (legacy per-key writes removed in favor of SMTP_URL via YAML)
+}
then Patch 2 -
templates/web.template.yml
(to document the safer path)
--- a/templates/web.template.yml
+++ b/templates/web.template.yml
@@ -68,6 +68,14 @@ params:
DISCOURSE_SMTP_ENABLE_START_TLS: true
#DISCOURSE_NOTIFICATION_EMAIL: noreply@example.com
+ ## Preferred single-line SMTP configuration (set by discourse-setup):
+ ## URL-encode username & password; example:
+ ## SMTP_URL: "smtp://user%40example.com:p%40ss%3Aword@smtp.example.com:587"
+ ##
+ #SMTP_URL:
+
## If you cannot use SMTP_URL, you may set per-key variables instead.
## Beware that editing those lines with shell tools can be fragile if values include
## characters like @, :, /, ", \, or newlines.
why this works (and what it avoids)
bash layer: we only interpolate simple variables; secrets are passed into Python/Ruby via stdin/argv, not through sed regexes or shell evals.
sed layer: removed entirely.
YAML layer: Ruby/Psych handles quoting and escaping properly; no hand-rolled quoting.
SMTP creds: %-encoding in SMTP_URL is the right place to encode special characters for auth.
if you prefer to keep per-key vars, I can give you a sister patch that uses the same Ruby-YAML approach to set DISCOURSE_SMTP_* directly (still no sed), but the SMTP_URL route is the cleanest because it’s one key, one write, one encoding step.
We unfortunately can’t rely on Ruby (or Python) being installed on the host system in discourse installs - it’d be relatively simple to have the entire script be a ruby one if we could, but there’s no guarantee.
Got it — no Ruby/Python on the host. Here’s a pure Bash + awk patch that:
• builds a percent-encoded SMTP_URL using only Bash (byte-wise loop; no Python),
• inserts or replaces SMTP_URL under the env: block using awk (no sed),
• removes the per-key DISCOURSE_SMTP_* lines (safe anchored deletes),
• adds a tiny sanity check using grep/awk only.
Apply with git apply -p0 in the discourse_docker repo.
--- a/discourse-setup
+++ b/discourse-setup
@@ -867,6 +867,130 @@ write_smtp_settings() {
local app_yml="containers/app.yml"
[[ -f "$app_yml" ]] || die "Cannot find $app_yml. Did you run bootstrap?"
+ ##############################################
+ # Pure-Bash URL encoder for SMTP credentials #
+ ##############################################
+ # Encodes everything except A-Z a-z 0-9 . _ ~ -
+ # Works byte-by-byte; requires bash and printf.
+ urlencode_cred() {
+ local s="$1" out= i ch o
+ # set C locale to get byte semantics
+ LC_ALL=C
+ for ((i=0; i<${#s}; i++)); do
+ ch="${s:i:1}"
+ case "$ch" in
+ [A-Za-z0-9._~-])
+ out+="$ch"
+ ;;
+ *)
+ # Get byte value: print char, read with od, then format %HH
+ # Avoid external heavy deps; od is in coreutils / busybox.
+ o=$(printf '%s' "$ch" | od -An -tu1 | awk '{$1=$1;print $1}')
+ # If od somehow failed (empty), fall back to hex via printf %02X of first byte
+ if [ -z "$o" ]; then
+ o=$(printf '%s' "$ch" | head -c1 | od -An -tu1 | awk '{$1=$1;print $1}')
+ fi
+ printf -v o '%%%02X' "$o"
+ out+="$o"
+ ;;
+ esac
+ done
+ printf '%s' "$out"
+ }
+
+ # Build SMTP_URL (single line) from collected answers
+ # These vars are gathered earlier in discourse-setup:
+ # $smtp_address $smtp_port $smtp_user $smtp_password
+ local addr="$smtp_address"
+ local port="$smtp_port"
+ local user_enc pass_enc
+ user_enc="$(urlencode_cred "$smtp_user")"
+ pass_enc="$(urlencode_cred "$smtp_password")"
+ local smtp_url="smtp://${user_enc}:${pass_enc}@${addr}:${port}"
+
+ ########################################################
+ # YAML-safe edit via awk (no sed / no external runtimes)
+ # - ensure env: exists
+ # - insert or replace SMTP_URL under env:
+ # - remove DISCOURSE_SMTP_* keys
+ ########################################################
+ awk -v NEWVAL="$smtp_url" '
+ BEGIN{
+ have_env=0; in_env=0; inserted=0
+ }
+ # detect env: line at top level (start of line, possibly indented 0..)
+ # we’ll consider two-space indentation for children.
+ /^[[:space:]]*env:[[:space:]]*$/ {
+ print; have_env=1; in_env=1; next
+ }
+ # leaving env: block when indentation returns to 0 or next top-level key
+ in_env && /^[^[:space:]]/ {
+ if (!inserted) {
+ print " SMTP_URL: \"" NEWVAL "\""
+ inserted=1
+ }
+ in_env=0
+ }
+ # while in env:, handle replacements and deletions
+ in_env {
+ # drop per-key DISCOURSE_SMTP_* lines entirely
+ if ($0 ~ /^[[:space:]]*DISCOURSE_SMTP_(ADDRESS|PORT|USER_NAME|PASSWORD):/) next
+ # replace existing SMTP_URL line
+ if ($0 ~ /^[[:space:]]*SMTP_URL:[[:space:]]*/) {
+ print " SMTP_URL: \"" NEWVAL "\""
+ inserted=1
+ next
+ }
+ print
+ next
+ }
+ { print }
+ END{
+ # If env: never existed, append it with the key
+ if (!have_env) {
+ print ""
+ print "env:"
+ print " SMTP_URL: \"" NEWVAL "\""
+ } else if (in_env && !inserted) {
+ # env: existed and we were still in it at EOF
+ print " SMTP_URL: \"" NEWVAL "\""
+ }
+ }
+ ' "$app_yml" > "$app_yml.tmp.$$" && mv "$app_yml.tmp.$$" "$app_yml"
+
+ ##############################################
+ # Sanity check: basic guard against mangling #
+ ##############################################
+ # 1) SMTP_URL present
+ grep -q '^[[:space:]]*SMTP_URL:' "$app_yml" || die "SMTP_URL not written to $app_yml"
+ # 2) password not prefixed by username (classic failure signature)
+ awk '
+ BEGIN{ok=1}
+ /^[[:space:]]*SMTP_URL:[[:space:]]*"/{
+ line=$0
+ gsub(/^[[:space:]]*SMTP_URL:[[:space:]]*"/,"",line)
+ gsub(/".*$/,"",line)
+ # Extract creds before @ and after scheme
+ # e.g. smtp://user:pass@host:port
+ sub(/^[a-z]+:\/\//,"",line)
+ at=index(line,"@")
+ if (at>0) {
+ creds=substr(line,1,at-1)
+ colon=index(creds,":")
+ if (colon>0) {
+ user=substr(creds,1,colon-1)
+ pass=substr(creds,colon+1)
+ if (index(pass,user)==1) { ok=0 }
+ }
+ }
+ }
+ END{ if (!ok) { print "Password appears prefixed by username"; exit 1 } }
+ ' "$app_yml"
}
- # Write per-key SMTP entries (address/port/username/password)
- # (legacy: performed via sed substitutions)
- # NOTE: historically fragile with special chars
- update_setting_yaml "DISCOURSE_SMTP_ADDRESS" "$smtp_address"
- update_setting_yaml "DISCOURSE_SMTP_PORT" "$smtp_port"
- update_setting_yaml "DISCOURSE_SMTP_USER_NAME" "$smtp_user"
- update_setting_yaml "DISCOURSE_SMTP_PASSWORD" "$smtp_password"
-}
+ # Legacy per-key writes removed in favor of atomic SMTP_URL write above.
+}
Why this meets your constraint
• Zero Ruby/Python on the host: only Bash + awk + od/printf/grep (all standard).
• No sed replacement of secrets: we avoid the “Bash → sed → YAML” multi-escape trap.
• Atomic-ish write: edit into a temp file, then mv over containers/app.yml.
• Backwards-safe: if env: doesn’t exist, we create it minimally; if it does, we update in place and remove conflicting DISCOURSE_SMTP_* lines.
• Credentials are URL-encoded before insertion, so special chars in user/pass won’t break the URL parser in Rails/Net::SMTP.
If you’d prefer to keep the per-key variables instead of SMTP_URL, I can send a sibling patch that (still without sed) uses the same awk strategy to set:
Is it possible to compile the launcher into a binary executable file for release? In this way, the launcher(bash) can only download the corresponding executable file(binary) based on the system.
