As an email authentication company, we maintain the highest security standards, and maintaining a bulletproof infrastructure is extremely important. One of our most important requirements is multifactor authentication (MFA). Among many other requirements, our MFA implementation has helped our secure, multi-cloud AWS infrastructure achieve FedRAMP authorized status and SOC2 Type 2 certification.
Since we have a large and growing engineering team, many technical staff play a role in managing our secure environments, and this poses a special challenge. We designed Valimail’s infrastructure to be resilient and secure, as well as manageable by every member of our engineering team. However, we use multiple AWS accounts, which somewhat complicates the issues involved in adding or removing users. The usual methods of MFA within AWS were not a match for our requirements.
Supporting our requirements while using only services within AWS has been interesting, and we thought we’d share some of what we learned while making this happen.
Our environment
We follow best practices in both software and infrastructure design, keeping our systems behind secured and auditable bastion hosts, with a small, controlled group of users having access to those machines for maintenance activities and, if needed, debugging production issues.
We manage engineers’ access to these systems by following a pattern described by Cloudonaut, called the “bastion account,” where:
- A single AWS account manages users and roles, and
- We create and manage the actual EC2 (and other) resources within separate accounts.
The advantages of using this pattern are several:
- Easier user management
- Clear, auditable systems access
- No tokens are stored on our machines to enable user access
- We can use IAM to enable or disable access to any of our environments at any time, no rollouts or updates required.
Following AWS best practices, we also require users to set up MFA before using AWS services. MFA within AWS services, however, is not sufficient to support the security posture we wanted to create. We designed our security requirements to ensure we meet standards for FedRAMP, as one example. Using MFA for AWS services alone was not enough.
Our requirements for MFA
Multi-factor authentication within AWS only covers things such as console access and API access. Our requirements specified MFA as a prerequisite to login to specific systems, and we needed an audit trail across our multiple accounts. We wanted a solution that would allow us to manage access for a growing group of engineers, devops, and other users in a secure fashion, with full auditing of all aspects of the user authentication flow across a host infrastructure that could be replaced at any time.
AWS does not offer anything we could use as a service. As we researched the topic, we only found one instance of someone using the AWS MFA service to validate system access — and it required a hack we were not comfortable depending upon. Internal discussions around security requirements led us away from using third party vendors to secure access to our systems. Other research led us to an AWS Startups blog post where they describe setting up Google Authenticator (GA).
At first glance, this appeared to be a solution. But after walking through the user- and systems-management aspects, we found that it would not work for us. We manage a herd of machines that are intended to be replaced at any point in time. The suggested method for installing Google Authenticator would cause MFA tokens to be re-issued every time a host was replaced. The described method of generating and storing these tokens did not fit our desired security posture, as well.
How we implemented MFA
We needed a method that worked within our bastion account setup, where we provision users in IAM within one account, and grant them access by IAM roles to resources within other accounts. We also wanted to make it easy to securely manage the MFA token generated by Google Authenticator without having to distribute it across all of our infrastructure.
The solution we implemented is to manage the GA token within the AWS SSM Parameter Store. Using the Parameter Store gave us a single place to store an encrypted secret on a per-user basis. It fit well within our existing IAM permissions and roles setup, and did not add a significant user administration burden. We are able to see access requests for these tokens in our existing auditing systems, as well.
We then turned to how to retrieve and verify the MFA token using Linux Pluggable Authentication Modules (PAM). Google Authenticator supplies a PAM module that performs the authentication of the user’s token. We created a script that securely retrieves the token from the Parameter Store and makes it available to GA for verification. This script is run by pam_exec.so, after the user’s SSH key has successfully verified. Once the token has been passed to GA, it is removed from the host, whether the user was granted access or not.
This solution met our security and compliance requirements by:
- Securely storing user access tokens for MFA and SSH
- Enforcing MFA across our infrastructure for all users
- Managing users consistently across all our accounts
- Providing auditable records of all user access to our infrastructure
We don’t have to distribute any keys or tokens across our networks, nor do we have to clean them up should we change a user’s access. Logging of all access attempts is done in cloudtrail as well as in the Linux system logs, giving us an audit trail that is verifiable in a manner that meets our security requirements. And last but not least, this scheme doesn’t require us to set up and maintain complex infrastructure to support our growth.
We use many approaches to build and maintain a secure infrastructure at Valimail. With this method of securing systems access, we were able to address the requirements necessary to obtain our SOC2 Type 2 and FedRAMP certifications extremely quickly. It’s just one example of how we think about security in every aspect of our systems design.
Top photo: Fisherman’s Bastion Viewing Tower by Mark Gregory/Flickr