AI Agents Are Making AWS Networking a 3D Chess Game
AI models and applications create data traffic patterns that enterprises may not have designed for, increasing the complexity of AWS cloud management
Enterprises have spent years optimizing their cloud compute and storage costs, but now large language models (LLMs) and AI agents are adding another layer of traffic to already complex hybrid cloud environments.
Traditional applications already generate east-west traffic, but AI introduces additional communication. Data now moves between models, inference databases, data lakes applications and, increasingly, agents, multiplying east-west network traffic within a data center. On AWS, traffic between workloads is a billable resource.
Enterprises have not traditionally measured traffic in the same way as compute or storage, making these costs difficult to predict.
“You’re exponentially increasing your east-west traffic between these things that you already had east-west traffic in your normal operating model to begin with, but now you add AI and/or agents, and that’s going to increase easily by 10x what you already had,” Herb Hogue, CTO at Myriad360, tells TechChannel. “That’s an additive cost that most forget about, and it’s hard to quantify because you normally don’t track that traffic.”
The AI Bill Doesn’t Stop at the Token
Enterprises are becoming aware of the massive costs associated with burning through LLM tokens. But they may not fully account for the infrastructure and networking costs that AI applications incur.
“It is very expensive,” Hogue says, “You have your egress and ingress data costs. You have your AI token costs. You still have your LLM and data lakes or databases sitting around with data traversing.”
In addition to traffic moving between LLMs and the enterprise, data moves between AWS regions, as well as between AWS and on-premises environments, and the replication of AI-related data, inference databases, backups and logs adds further networking complexity.
Given that AI agents can act autonomously, the amount of data traffic they will generate once they are deployed is difficult to predict. “It’s hard to gauge what the LLM and/or agents are going to do in the wild,” Hogue notes.
While the traditional three-tier applications that enterprise teams are accustomed to have relatively understandable traffic patterns, AI introduces a multi-dimensional model encompassing various data sets, models, inference, agents and applications.
Agents can dynamically invoke other services and create sub-agents, models can be hosted in different locations and data may reside on-premises while inference happens in AWS.
“Your data propagation is imploding at a very rapid scale,” Hogue says. “And the other piece is for all of these processes to run, you need a lot of memory designated to them. … There’s a lot of ancillary costs that surprise some people.”
Legacy analysis tools were well suited to static on-premises operating environments, but “now there’s so many variations of the ecosystem that … make it very difficult to price out what you’re doing until you’re there, and once you’re there, it’s very difficult to get out,” Hogue says. “That’s a major expense/potential deterrent for cloud, which is why you’re also seeing some clients repatriating certain functions, particularly IP-based functions where their data is really important to them, and they’ll bring it on-prem.”
Hybrid Cloud Makes the Problem Three-Dimensional
AWS networking is sophisticated, and enterprises that opt for a hybrid approach with some data remaining on-prem and some in the cloud need to think about the multiple dimensions simultaneously.
“We’re now playing three-dimensional chess in the world that we live in,” Hogue says. “Although it’s gotten easier, it’s gotten far more complicated at the same time.”
Network design, workload placement, cost, performance and security are all interdependent elements of application infrastructure that are more complex than standing up an instance in a mainframe environment.
“AWS, in my view, is very networking-centric in how you access their services and construct, which is great, but the level of sophistication is high,” Hogue says. “So if you’re going to create and/or traverse a network on a global scale with some degree of hybrid ability, you’re going to require enhanced networking skills on your team to maintain and manage that.”
Teams should evaluate where compute should run, where data should be stored, where inference will happen and how much traffic will cross the hybrid boundary. Running locally requires large volumes of network traffic back and forth, and raises the question of how much latency is acceptable.
AI makes the old question of cloud or on-premises more complex. Keeping AI workloads close to enterprise data can reduce unnecessary traffic and simplify governance.
“That’s a major expense/potential deterrent for cloud, which is why you’re also seeing some clients repatriating certain functions, particularly IP-based functions where their data is really important to them, and they’ll bring it on-prem,” Hogue says.
Data Sovereignty Adds Another Layer
Growing data sovereignty requirements are also increasing network complexity, as data residency regulations can force enterprises to distribute workloads and data geographically.
Data may need to remain in particular regions, which could require enterprises to replicate LLMs and inference databases across multiple AWS regions, creating additional copies of backups and logs and further adding to network and storage costs. Different regional requirements may also constrain workload placement.
“Government access is becoming a major, major problem,” Hogue says, “and now you have multiple countries in the EU, the U.S., Asia Pac—everybody’s doing something slightly different, and there is a very strong concern as to what the local government can warrant you to provide access to. That was never something that you thought about as regularly as you do today, and so that is having a massive impact.”
Another element when it comes to AI applications is the data retention policies of the LLM providers, which has prompted some enterprise leaders to restrict the use of certain models. Leaders need to understand what data the enterprise retains from AI interactions and what the provider retains.
“We’ve also seen that’s now admissible, as governments can request that history,” Hogue says. “Do you want that history retained by a third party, and them providing your information without maybe you knowing, because they were required? There’s a lot of dynamics around tertiary data that need to be considered, which is why the repatriation element is occurring, and the locality issue is happening on a greater scale.”
As traffic becomes more dynamic, enterprises need visibility into how much network traffic is generated by AI workloads, which agents are communicating with which services, what data is moving and where, which identity authorized the traffic and what happens when an agent invokes another agent.
If enterprises lack a clear view of where AI traffic is going, they cannot optimize the costs or architecture. Availability is becoming a bigger concern, particularly given geopolitical factors.
“We used to take into effect power outages, acts of God, weather events, but now we also have to take into account wars and facilities being targeted in a war. We’ve seen that in the Middle East happen several times,” Hogue points out. “Does the location get attacked and wiped out? Is it in a combat zone, or a friendly zone?”
AWS recently confirmed that it is unable to recover data and resources stored in one of its three Availability Zones in the United Arab Emirates Region, which were hit by Iranian drone strikes in March. It is still working to recover the other two AZs.
Given the risk to cloud resources, “the other piece that has come back in vogue is having hard copies of your data again,” Hogue says, “and many companies are back to making on-prem backups with drives and/or tape and storing them offline, so that if something was to occur, they have a pristine state to start from.”
For hybrid environments in particular, network architecture is becoming much more tightly coupled with application architecture to accommodate AI. The questions that enterprises might have treated separately around cost, performance, security, sovereignty and workload placement increasingly have to be answered together, as deciding where to run workloads depends on understanding how much traffic that decision creates and what the traffic costs.