A false flag: when mail.smtp.auth=false still sends AUTH LOGIN

protocolConnect login, host=<relay>, user=, password=<non-null>
Attempt to authenticate using mechanisms: LOGIN PLAIN DIGEST-MD5 NTLM XOAUTH2
Using mechanism LOGIN
AUTH LOGIN failed
Mail Send Fail email: ..., error: 334 VXNlcm5hbWU6

The symptom, and why it looks like a config error

The relay was deliberately configured to allow unauthenticated relaying, and the application’s config said not to authenticate. Everything about the setup read as “no auth involved”. Yet mail failed, and the error pointed at a login handshake nobody asked for.

The tell is in the log line:

protocolConnect login, host=<relay>, user=, password=<non-null>

• user= — the client is presenting an empty username. It is not using stale credentials; it has nothing to send.
• password=<non-null> — the password field is set (to something, possibly empty), which is exactly what makes the library think auth is wanted.

And the failure message:

error: 334 VXNlcm5hbWU6

334 is an SMTP continuation reply; VXNlcm5hbWU6 is base64 for Username:. The relay asked for a username, the client sent an empty one, and the exchange died. The relay was not rejecting the client — the client had walked into an auth conversation it could not finish.

Reproduction

A client-side auth bug needs a relay you control, one that offers AUTH without requiring it. MailDev is a good fit: a maintained SMTP catcher that accepts unauthenticated mail by default, with a JSON API and web UI for inspecting what it received. (MailHog is the older equivalent; a real Postfix would work but is far more setup for the same result.)

Start it plain — accepting mail with no auth, which is the “permissive relay” case:

docker run -d --name maildev --restart unless-stopped \
  -p 25:1025 -p 1080:1080 \
  maildev/maildev

SMTP lands on port 25 (forwarded to the container’s 1025), the web UI on 1080, and captured mail is available as JSON at /api/email. One curl gotcha: GET / returns {"error":"Not found"} because that route is API-only — the SPA only renders in a browser. Hit /api/email for the raw captured messages.

Configure the application for that relay and confirm it works. In our case:

mail.sender=no-reply@example.com
mail.host=10.0.0.10
#mail.username=
#mail.password=
mail.smtp.protocol=smtp
mail.port=25
mail.smtp.auth=false
mail.smtp.ssl.enable=false
mail.smtp.starttls.enable=false
mail.smtp.starttls.required=false
mail.smtp.debug=true

A quick smtplib script (EHLO → MAIL FROM → RCPT TO → DATA, never sending AUTH) confirms the catcher is working and queues a message.

Config gotcha worth stealing: the application also shipped a commented-out sample config using the key mailhost=. That key is not read — it loaded as blank with no error, and the startup log showed configure.mail.host : with nothing after the colon. The working key was mail.host. A silent blank is worth guarding against in any config loader: print resolved values at startup, and diff them against what you wrote.

Then recreate the catcher so it advertises AUTH — with credentials of its own, which the application will not have:

docker rm -f maildev
docker run -d --name maildev --restart unless-stopped \
  -p 25:1025 -p 1080:1080 \
  maildev/maildev --incoming-user testuser --incoming-pass testpass

Leave the application config exactly as it was — mail.smtp.auth=false, username and password still commented out — restart it, and trigger a send. The failure reproduces verbatim:

DEBUG SMTP: protocolConnect login, host=10.0.0.10, user=, password=<non-null>
DEBUG SMTP: Attempt to authenticate using mechanisms: LOGIN PLAIN DIGEST-MD5 NTLM XOAUTH2
DEBUG SMTP: Using mechanism LOGIN
DEBUG SMTP: AUTH LOGIN command trace suppressed
DEBUG SMTP: AUTH LOGIN failed
Mail Send Fail email: someone@example.com, error: 500 Error: missing username

The relay only had to advertise AUTH. It did not have to enforce it. That is the whole trigger.

Root cause

The bug is split across two pieces of code, and each one alone is harmless.

1. The config loader defaults blank fields to "", not null:

mail_smtp_auth = getBoolean("mail.smtp.auth", false);
mail_username  = getValue("mail.username", "");            // default "" not null
mail_password  = getValueOrDecrypted("mail.password", ""); // default "" not null

Unset keys therefore arrive as empty strings. The author almost certainly meant “no credentials”, but the value says “credentials, which happen to be empty”. A second, normally-unused branch of the same loader does null them out — so the intended representation existed, it just was not reachable on the normal path.

2. JavaMail’s auth decision does not consult that flag the way you would expect. Inside com.sun.mail.smtp.SMTPTransport.protocolConnect(), authentication is gated on:

useAuth || (user != null && password != null)

With mail.smtp.auth=false, useAuth is false — but the second half is true, because "" is not null. If the server’s EHLO response advertises AUTH or AUTH=LOGIN, the transport then calls authenticate(user, password) with the empty strings it was handed.

Chain, end to end:

mail.smtp.auth=false
  → username/password are "" (not null)
    → user != null && password != null  ⇒ true
      → relay advertises AUTH in EHLO
        → AUTH LOGIN with an empty username
          → relay: 334 VXNlcm5hbWU6
            → handshake fails, mail never sends

This is why the behaviour looks inconsistent in the field: against a relay that never mentions AUTH, mail.smtp.auth=false works perfectly. Against a relay that supports auth but does not require it — extremely common on internal relays — it breaks.

The fix

Workaround, if you are blocked today. The loader had a second branch that forces credentials to null. It is reachable through an oddly-named flag left over from an unused code path:

if (mail_auth_mechanism_gssapi) {
    mail_smtp_auth = false;
    mail_username  = null;
    mail_password  = null;
} else { /* the load-with-empty-string-defaults branch above */ }

Grepping the whole jar showed that field is read nowhere else — it wires up no actual Kerberos/GSSAPI behaviour. It is purely a way to force null credentials. Turn it on alongside your existing setting:

mail.smtp.auth=false
mail_auth_mechanism_gssapi=true

Verified both ways on a test host:

• Against a catcher that mandates auth: no protocolConnect login line at all — the client made no auth attempt, and only the relay’s own 530 Error: authentication Required came back. The spurious attempt was gone.
• Against a plain no-auth catcher: mail sent normally — useEhlo true, useAuth false, 250 Message queued, delivered. No auth negotiation.

Proper fix, for the library authors. In the loader’s normal branch, set the username and password to null (not "") when mail.username / mail.password are unset or blank — mirroring what the GSSAPI branch already does. Then user != null && password != null evaluates false and JavaMail correctly skips auth when mail.smtp.auth=false.

Caveats

• This workaround is only valid while the relay requires no authentication. If you later need authenticated SMTP, you cannot use the flag. That path is unaffected though: mail.smtp.auth=true with real credentials already works, because the failing condition needs auth=false and credentials that resolve to non-null empty strings.
• The bug needs all three: mail.smtp.auth=false, unset/blank credentials, and a relay that advertises AUTH. Miss any one and you will never see it — which is exactly why it survives in production until a specific customer’s relay hits it.
• If you patch the config file by hand, back it up first and know the rollback: restore the backup, restart the service. The fake SMTP catcher is disposable (docker rm -f maildev).

Takeaways

1. “Unset” and “empty” are different values. A "" default silently converts a missing credential into a present-but-blank one, and every null-check downstream now says “credentials supplied”. Default unset things to null.
2. A boolean flag is only as honest as the branch that reads it. Here mail.smtp.auth=false was loaded faithfully and then effectively overruled by an || in a third-party library. When debugging, read the condition, not the flag name.
3. Relays that advertise-but-do-not-require AUTH are the interesting test case. Testing only against a no-AUTH relay or an AUTH-mandatory relay would have missed this entirely. Make that middle case a test fixture — it caught a real customer bug here.

Derived from an internal WhaTap debugging session; client and environment details generalised. Full internal record: concepts/notihub-mail-smtp-auth-bug.md.