Where Does Amazon Bedrock Fit in the Enterprise AI Stack?
AWS is reporting strong demand for Bedrock, but where should enterprises actually place it in the AI stack, particularly those coming from mainframe environments?
AWS is reporting strong demand for Bedrock, but where should enterprises actually place it in the AI stack, particularly those coming from mainframe environments?
AWS has taken the position that enterprise AI adoption is moving beyond individual models and applications, toward a connected ecosystem of AI agents.
As part of that stance, Amazon is previewing AWS Agent Registry, available through Amazon Bedrock AgentCore, to enable enterprises to find, share and reuse AI agents, tools and agent skills across teams. “We’re building toward a future where the registry spans every AWS service where agents are built, including Amazon Quick, and Kiro,” the company stated.
The release comes as demand for Amazon Bedrock is accelerating. In its most recent quarterly earnings results, the company reported that “hundreds of thousands of customers now use Bedrock, more customers were added in the last six months than in the first two years after launch, and customers spent more in Q2 than all prior quarters combined.” AWS’s overall AI business exceeded a $25 billion annual revenue run rate, growing by a triple-digit percentage.
Bedrock is a managed infrastructure service on AWS that provides access to a catalog of hundreds of foundation AI models through a single API. Available models include Anthropic’s Claude, OpenAI’s ChatGPT, Meta’s Llama, Mistral, DeepSeek and Amazon’s Nova.
The platform offers evaluation tools to select the appropriate models for specific application needs and the ability to swap them in and out without rewriting code. This flexibility is one of the reasons Bedrock is growing quickly, Amazon CEO Andy Jassy said on the company’s earnings call:
“If you’re a company that’s building important AI applications, you want to make sure that you have the ability to use all the available models. They’re going to each leapfrog each other at different times. They’re going to have lots of different models that actually are comparable in capabilities.”
Bedrock AgentCore provides teams with building blocks to deploy and operate AI agents. But enterprises with significant mainframe and on-premises environments face the architectural challenge of connecting these workloads to their existing data, applications and infrastructure while addressing security, governance, identity and cost without creating another silo.
What Bedrock Brings to Enterprise AI Architecture
Bedrock sits in the AI services and model layer of the enterprise technology stack—above it are the applications that users interact with, such as AI assistants and agents, and below it are the data, knowledge bases, APIs and enterprise systems that provide the context for those applications. The underlying infrastructure includes AWS compute, networking and security.
Many enterprises will need to connect workloads based on Bedrock to on-premises environments. A typical setup might involve mainframe data being accessed through a secure connection to AWS for use by a Bedrock-based application.
“In a situation where you want to, or you must keep, data on your own existing infrastructure on the mainframe, then you would need to make sure that you secure the connection to Bedrock correctly,” Jerzy Kopaczewski, co-founder and CTO of Devopsity.io, tells TechChannel.
“It stays in your VPC (virtual private cloud), but you have a secure connection to your VPC from the mainframe. It’s a classic hybrid architecture where you have a secure channel between two data centers—one is yours and one is AWS,” Kopaczewski explains. “Then you need to make sure that you’re shifting enough, but not too much, data. You need to establish your guardrails, what elements of your data you’re willing to share, to flow to the external provider.”
The answer may not be to move large volumes of data to the cloud. Instead, enterprises need to determine what information the AI model requires for the workload.
“Bedrock specifically is quite elastic—it has a rich security tooling around it, so you can set up multiple gateways on an input and on an output,” Kopaczewski explains.
Securing the data that is sent to a model is only one part of the challenge, as enterprises also need to control what happens to the information it generates in response.
“It’s more important what the model is actually outputting rather than what it takes into input because the data stays in your VPC in AWS. You sign a contract with them that this should not leave your private space, but then you might unwillingly share outputs with people who might not have authorization to actually see it,” Kopaczewski warns.
An AI-generated response to a prompt could contain sensitive information the model retrieves from enterprise systems, particularly in instances where applications use retrieval-augmented generation (RAG). To limit exposure of sensitive data, a robust architecture sets authorization and authentication controls over the data flow between AWS and enterprise systems, determining who can access an AI application and its results, Kopaczewski says. “What form is it coming back to, and who can actually see the outputs and results?”
Bedrock Isn’t Necessarily the Answer
The convenience of a managed, cloud-based service does not mean Bedrock will be the right choice for every enterprise AI workload. As inference volumes increase, enterprises may need to reassess whether it would make economic and operational sense to run models themselves.
“Bedrock is a good solution for ideation and use until you reach a certain traffic threshold. If you cross it, then it might make sense for you to actually not use Bedrock, but decide on running your own LLMs,” Kopaczewski says.
Enterprise teams might determine that high consumption volumes or highly specialized workloads justify deploying models on dedicated AI infrastructure managed directly by the organization, using platforms like Kubernetes or AWS compute services such as Elastic Cloud Compete (EC2) and Elastic Container Service (ECS).
The decision should be based on factors such as inference volumes, latency, utilization requirements, model choice, security and regulatory considerations, available engineering expertise and the operational burden of managing models. Cost is another factor, as while a managed service can reduce the infrastructure and operational overhead of running AI models, a dedicated environment could prove to be cost-effective at a sufficient scale.
“AWS is best used when the architecture fits the cloud-native structures. The biggest thing to watch is opportunities to modernize the application stack and use the cloud … not just move, and then basically use the mainframe on AWS, which is quite a common thing,” Kopaczewski says.
Enterprises should avoid making decisions about managing or self-hosting AI solely on current workloads. Just as Amazon notes that preferences for specific models change, as adoption grows, application requirements could change significantly.
“You need to design this looking six, 12, 24 months in advance,” Kopaczewski advises. “Sometimes you’re better off using Bedrock. Sometimes you should be already running inference on your controlled, managed instances.”
Bedrock is well suited to enterprises looking to experiment with AI, access multiple foundation models, build applications and, increasingly, deploy AI agents, without the operational burden of running each model in-house. The platform also offers a way to scale AI workloads without immediately building dedicated infrastructure, particularly if the organization is already using AWS.
The strategic question for enterprises with significant mainframe infrastructure is where Bedrock provides the most value, and where the organization needs to retain control over the underlying infrastructure.