I’ve Got 99 Problems But SPF Ain’t One

One of the biggest challenges organizations face in getting email authentication to enforcement involves Sender Policy Framework, aka SPF.

Here’s why: If you can’t identify all the senders who should be able to send email messages using your domain, and then use SPF to authorize them, you can’t move your DMARC policy to p=quarantine or p=reject. Moving to enforcement without authorizing every service you actually use means that a few legitimate senders are going to get blocked.

Solving this problem takes a lot of ingenuity, due to limitations within the SPF standard.

A Quick Backgrounder on SPF

While the SPF standard dates to 2003 and is well-understood and widely accepted, many IT organizations find that it’s tricky to make it work in a modern, cloud-centric environment.

SPF is a DNS-based mechanism for defining a whitelist of mail servers that can send email on your domain’s behalf. The specification lets you authorize individual mail servers by IP address or by ‘including’ SPF records that are defined on a specified domain. When a domain name is listed in an SPF record, that tells receiving mail servers to go to the indicated address, where they will find additional rules: IP addresses, SPF macros, or additional domain names where more rules can be found. In other words, domain references can be nested, so one ruleset may contain several others.

Diagram: How SPF works by validating the Return-Path sender

For most services that send email on your behalf, you need to put something in SPF to specifically white list that sender: Either its IP address(es), or an SPF “include” mechanism that indicates where receiving mail servers can find the appropriate rulesets.

That was easy enough when companies primarily sent mail through servers they owned and operated themselves. It gets significantly more challenging when you have dozens of cloud services sending email on your behalf.

SPF lets you put a large number of IP addresses in your SPF record, but it strictly limits the number of domain lookups that receivers will do to just 10. That count includes not only domains explicitly listed in your SPF record, but also any domain lookups contained within the listed domains.

If a service, such as Gmail, contains multiple domain lookups in its SPF record, all of those lookups count towards the total. So you can easily exhaust the 10-lookup limit by putting just three or four services into your SPF record.

SPF ‘Flattening’ Doesn’t Work

The solution many IT organizations have hit upon is called SPF “flattening” — which means taking all of those domain lookups (and the domain lookups nested within them) and translating them into a single “flat” list of IP addresses.

In other words, instead of letting the receiving mail gateways do the lookups for each inbound message, you do the lookups yourself, by hand if necessary. Eventually each of those lookups will (usually) lead you to a list of authorized IP addresses that you can place into your SPF record instead of referencing one or more domains for each service.

Sounds simple, right? Here’s where things can go badly wrong, though:

  • Service providers frequently add and remove IP addresses from the list of sending IPs for their service.
  • It’s easy to make errors — either in the IPs themselves or in the SPF syntax — when you’re building these long lists.  Are you sure you got that IPv6 address right?
  • Transforming that list of IP addresses and netblocks into an SPF record may require you to split it into multiple SPF records, linking them together — and possibly running into that 10 domain lookup limit all over again.
  • Cloud service providers generally don’t notify their customers when they change the list of IP addresses from which they send email, so you’re going to have to track those changes yourself.

That means, if you’re the owner of a “flattened” SPF record, that you now have the unenviable job of monitoring all the services in use, making sure that the list of IPs for each is still current, and that the overall list is complete.

And you did take notes when you were assembling the list, so you can tell which IP belongs to which service, right? Because you — or future IT admins — won’t be able to tell which is which just by looking at a long list of IPs.

Finally, humans tend to be really bad at managing lists of digits. Typos, transpositions, dropped periods, and other kinds of errors crop up all the same.

For this reason, SPF flattening is fragile, brittle, error-prone, and winds up creating a significant maintenance overhead.

The Solution: Valimail Deliver™

Valimail’s solution is called Valimail Deliver™, and it solves the SPF 10-lookup limit without recourse to SPF flattening.

Valimail Deliver, backed by the Valimail IDEA platform, includes the company's unique, patented Instant SPF™ technology.

Built using Valimail's global, cloud-based infrastructure, Instant SPF dynamically generates a perfectly tailored SPF record, in milliseconds, in response to each mail server request. Instant SPF is scalable, fail-safe, and is already serving SPF records to process many billions of customer messages per month as part of the Valimail Enforce product.

Instead of relying on a single static SPF record, Instant SPF auto-generates an SPF record in real time as needed. You don’t need to worry about the “plumbing” or the details of the SPF standard. Instead, you simply designate the services you want to authorize by selecting them from a list. We take care of the rest.

Our approach is completely compliant with the SPF standard and is supported by every receiver that complies with the SPF specification, including all major ISPs and SEGs.

And it’s powerful enough that, in part because of Instant SPF, Valimail is able to offer the only enforcement guarantee of any email authentication provider. In fact, on average our customers get to DMARC enforcement within about 100 days.

See our website for more information on Valimail Deliver.

The CTO and co-founder of Valimail, Peter is an MIT- and Stanford-trained technologist who has worked in a variety of software verticals including security, enterprise, email, and video. He has built products and teams at a number of large technology companies such as RSA Security and Perot Systems, as well as at small startups like Tout, Securant, and Swapt.