At Amazon, our culture, built on honest and transparent discussion of our growth opportunities, enables us to focus investment and innovation to continually raise the bar of our ability to deliver value for our customers. Earlier this month, we had the opportunity to share an example of this process at work in Mantle, our next-generation inference engine for Amazon Bedrock. As generic AI inference and fine-tuning workloads evolve, we need to evolve how we provide inference to our customers in an optimized way, leading to the growth of Mantle.
As we plan to reimagine the architecture of our next generation inference engine, we have made raising the security bar our top priority. AWS shares our customers’ unwavering focus on security and data privacy. This has been at the heart of our business from the beginning, and it was in particular focus from the early days of Amazon Bedrock. We have understood from the beginning that generative AI inference workloads provide an unprecedented opportunity for customers to harness the latent value of their data, but with that opportunity comes the need to ensure the highest standards in security, privacy and compliance as our customers build generative AI systems that process their most sensitive data and interact with their most critical systems.
As a baseline, Amazon Bedrock is designed with the same operational security standards you see in AWS. AWS has always used a least-privilege model for operations, where each AWS operator has access to only the minimum set of systems needed to perform their assigned task, limited to the times when that privilege is needed. Any access to systems that store or process customer data or metadata is logged, monitored for anomalies, and audited. AWS provides protection against any action that would disable or bypass these controls. Additionally, your data on Amazon Bedrock is never used to train any models. Model providers have no mechanism to access customer data, as the estimation is done only within an account owned by Amazon Bedrock, to which model providers do not have access. This strong security approach has been a key enabler for our customers in unlocking the potential of Generator AI applications for their sensitive data.
With Mantle, we raised the bar even higher. Following the approach of the AWS Nitro system, we designed Mantle as zero operator access (ZOA), where we intentionally excluded any technical means for AWS operators to access customer data. Instead, systems and services are administered using automation and secure APIs that protect customer data. With Mantle, there is no way for any AWS operator to sign in to the underlying compute system or access any customer data, such as estimate signals or completions. Interactive communication tools like Secure Shell (SSH), AWS System Manager Session Manager, and Serial Console are not installed anywhere in Mantle. Additionally, all estimation software updates need to be signed and verified before being deployed into service, ensuring that only approved code runs on Mantle.
Mantle uses the recently released EC2 instance verification capability to configure a hardened, constrained, and immutable compute environment for customer data processing. Mantle’s services that are responsible for handling model weights and performing inference operations on client signals are supported by high assurance cryptographically signed verification measurements from the Nitro Trusted Platform Module (NitroTPM).
When a client calls a mental endpoint (for example, bedrock-mantle.(regions).api.aws) such as those serving the Response API on Amazon Bedrock, client data (signals) leave the client environment via TLS, and are encrypted all the way to the Mantle service, which operates with ZOA. Throughout the flow and into Mantle, no operator, whether from AWS, the customer, or the model provider, can access customer data.
looking forward
Mantel’s ZOA design exemplifies AWS’s long-term commitment to the security and privacy of our customers’ data. It’s this focus that has enabled AWS teams to invest in further raising the bar on security. Additionally, we have made the core confidential compute capabilities we use internally at Amazon, such as NitroTPM Attestation, available to all customers to use on the Amazon Elastic Compute Cloud (Amazon EC2).
We are not stopping here; We are committed to continuing to invest in enhancing the security of your data and providing you with greater transparency and assurance on how to achieve this.
About the authors
Anthony Liguori AWS VP and Distinguished Engineer for Amazon Bedrock and principal engineer at Mantle.
