Delivery issues with SES
Incident Report for Buttondown
Resolved
Over this past weekend, SES (which is one of Buttondown's email service providers) paused sending from Buttondown's domain due to a spate of traffic.

While I was able to route all outgoing emails from SES to a failover provider, there was an unforeseen edge case — newsletters who have _custom domains_ that were set up to send from SES but not from one of the failover providers.

The problem is now resolved, but the symptom was that transactional emails — such as asking folks to confirm their subscription or informing them of any changes to their account — were not going out for a while.

Affected users have been informed and actioned (if you think you may have been impacted, email me to let me know!) but there are a number of things that can be handled better here.

- With regards to failover, I'm going to add a script that temporarily sets newsletters with custom domains into "non-custom-domain-sending mode" (clever name, I know!) in order to unblock such situations.
- There's insufficient retrying in the transactional emails at the moment; I don't think I will be surfacing that logic to authors, but it's important to be able to easily track which of such emails failed and make it easier for me to resend them.

As always, apologies for the hiccup — the privilege of sending your emails is an important one, and one I take seriously!
Posted Jan 15, 2023 - 21:00 PST