Pressure in the Pipeline: How AI is Reshaping the QA Sector
It’s become a cliché phrase that “AI is changing the game for (insert industry name here).” But trite as it may sound, nowhere is it more true than in the software development and quality assurance (QA) space. A 2023 GitHub survey of 500 U.S.-based software developers found that 92% now use AI in their work, and with good reason— AI can increase code production efficiency by almost 50%, according to a recent study by McKinsey.
And as AI becomes ubiquitous in the software development ecosystem, the amount of code in the pipeline is increasing dramatically. Over 25% of Google’s code is now generated by AI, and a recent GitHub experiment found that integrating AI into a JavaScript coding task increased productivity by 55%. AI assistants that can handle everything from code creation to debugging are being launched by everyone from OpenAI to Amazon.
QA’s Quantum Leap
These transformative changes are shaking up the quality assurance (QA) sector, says Katrina Collins, a product manager at TestRail who specializes in AI for the QA platform. “The QA sector has been relatively stagnant since automation emerged in the early 2000s,” says Collins. “But now, AI is impacting development and QA on a huge scale—it’s a quantum leap. It’s an incredibly exciting time to be involved, but anytime you have a major development like this, it creates anxiety for people.”
Some of that anxiety may be due to the sheer rapidity of the changes that QA professionals must integrate into their workflows. And while the AI-driven code productivity surge is a boon for the software development sector, it also leads to pressure on QA teams as the amount of code in the pipeline increases.
But QA teams are also leveraging AI to mitigate these issues. TestRail, which provides a centralized QA platform to optimize testing processes, recently conducted a survey of over 1,000 QA professionals, finding that most are using AI tools for test case design, automation or execution.
Code Quality, Cybersecurity and Data Privacy
While finding ways to cope with these rapidly shifting sands, QA professionals also must contend with new unknowns when working with AI-generated code. Among those questions is how often the code will need fixing, and while research is still sparse, early indicators show reason for concern.
A study presented at the 2024 International Conference on Software and System Engineering found that AI-generated code had approximately the same defect density as code written by humans, and a recent regression analysis by GitClear found that code churn—often an indicator of poor code quality—increased by close to 40% in 2023 when compared to the pre-AI baseline.
Collins also points to new concerns around cybersecurity and data privacy as more companies leverage AI assistants for code development. “AI code can also create vulnerability to attacks and fraud,” she says. “We need both AI and human agents to recognize those abnormalities and secure the code.”
Additionally, many “black box” large language models (LLMs) don’t disclose the source of their training data or make guarantees about how information fed to the LLM is safeguarded or will be used in the future, creating new risks. QA teams that leverage AI need to be aware of how sensitive data—say, API keys or financial information—will be consumed and utilized.
Other pitfalls also await QA teams as they adjust to the new age of AI. For example, outdated tech stacks can limit the ability of QA teams to effectively harness AI to manage their growing workload. “It’s like plugging a Lamborghini engine into an old Volkswagen and can create pressure on your development infrastructure,” Collins says.
New Horizons
But while the explosion of AI use in code development brings potential pitfalls, it also creates new opportunities. As the industry shifts, AI will play an ever-deeper role in QA. Collins envisions an increase in AI-driven predictive testing, with AI becoming better able to detect which segments are most fragile and prioritizing test cases accordingly. The TestRail study found that 24% of QA professionals are already leveraging AI to generate test cases, and 25% are using it to automate script generation.
Collins believes that leveraging AI agents will free up QA teams for deeper work. “If we can delegate these very repetitive tasks to AI tools, it means QA teams can spend more time doing exploratory testing, and approach systems from different perspectives and angles,” she says, noting that TestRail is developing a new test case generator powered by AI.
And as roles and needs shift in the QA sector, new needs—and the jobs to meet them—will surface. For example, UX interfaces that seem completely accurate to an AI agent are often unintuitive to human users. “We will need humans to test and analyze these things,” Collins says.
Collins also notes that, just as in other areas of development, AI-generated code quality depends on the expertise of the developer prompting the LLM, potentially creating new roles for AI experts in development and QA. “AI is a tool, so you have to know how to communicate with it correctly and give it the right instructions,” she says. “The context you provide must be rich and deep. With AI, the more food you give it, the heartier it is.”
QA’s Next Chapter
While AI’s current constraints and risks are real, its transformative potential is undeniable. A recent GitHub analysis estimates that by 2030, AI developer tools could provide the equivalent of an additional 15 million developers to global production capacity, with an economic impact upwards of $1.5 trillion.
Collins believes that QA professionals will experience both upheaval and new opportunities as the AI revolution unfolds, mirroring the automation surge of the 2000s. “Everyone thought that automation would take QA professionals’ development jobs, but instead, there was a new need for different types of QA automation engineers and sudden markets for new tools,” she says, predicting much the same for this current wave of AI innovation.
Given the ubiquitous presence of AI in the development pipeline, QA professionals must learn to harness the power of this new tool while contending with its risks and limitations. Those who meet this challenge will be best positioned to succeed in the brave new world of AI.