Avoiding the AI Cognitive Atrophy Trap
Data and AI expert Dr. Fern Halper goes on IT Social Hour to discuss what organizations need to be considering as machines take over human cognitive tasks
Listen to the audio-only version here.
The following transcript has been edited for clarity:
Andy Wig:
All right. Well, I think we’re going to explore some fascinating concepts here, at least for me. So I really appreciate you joining me here, Fern. So first off, I mean, this book that you wrote, Data Makes the World Go Round: The Data, Tech and Trust behind AI success. I mean, when I first heard about this, I was just curious, and it’s a full book. How do you write a book about … I assume it takes some time. How do you write a book about such a fast moving technology when it seems like every other week there’s some huge news that breaks? How do you go about that?
Dr. Fern Halper:
Yeah. I guess one of the things I’ve realized was that maybe I wasn’t really writing a book just about AI or I should say AI tools. I was really writing a book about the capabilities that organizations needed to put in place to succeed with AI because as you point out, obviously the technology is changing constantly and there’s the next model, et cetera. But the organizational challenges are remarkably stable. Organizations still need trusted data. They need governance, they need leadership support, they need operating models, they need skills, they need processes. So those are some of the same issues that I’ve been seeing in machine learning initiatives years ago. And so I tried to layer on top of those issues, some of the newer issues. And in many ways, I think AI exposes weaknesses that already existed, right? Organizations that have poor data quality, they have weak governance, they have siloed operations, they don’t have clear business objectives.
They struggled before AI and then AI magnifies those weaknesses. Yes, there’s new issues with data quality because we’re talking maybe about unstructured data and organizations have been dealing with data governance and now they have to deal with AI governance. I wrap some of those things in as well, but surprisingly, a lot of the issues still remain the same and I talk to organizations all the time and yeah, they’re saying, “Oh, it’s moving so fast. It’s moving so fast.” I think the leaders are also saying “It’s moving so fast, it’s moving so fast” because some of them don’t … They think of AI as a tool as opposed to a set of enterprise capabilities. So to them it’s moving very fast and yeah, that aspect of it is moving fast, but some of the other things have been going on for years and years. Anyway, I’ll stop there.
Andy:
No, no. Yeah. I hear what you’re saying. There are certain fundamentals it sounds like that have been true well forever, I guess. I think one thing you mentioned in your book is people maybe jumping past that foundational stage right into gen AI and trying to spin that up. Sounds like that’s one of the challenges people are facing.
Fern:
Yeah, for sure. And that’s what I noticed early on was one of the reasons I wrote the book was because I felt like organizations all of a sudden all of these AI experts started coming out of the woodwork saying, “I can write a prompt. I know about these off the shelf tools.” And I said, “Hold on there. There’s a lot more to AI than the latest tools.” So having worked as an industry analyst for many years and having been a data scientist at Bell Labs before that, I sort of said, “Just take a step back and bring together all of the pieces, all of those capabilities that are needed.” So that’s what I did. That’s not to say you didn’t write about Agentic AI and open table formats and line chain and everything else. I tried to put that in there also, but the idea was I wanted to help executives understand what was involved so that they could have more realistic expectations out of what they could get from their AI initiatives.
Yeah, like anyone can build, I’m not saying anyone, but if you have a really good idea, you can build an AI application and good for you, you could make a lot of money on that, but I’m talking about enterprises, organizations, companies that have data that should be using their data with their AI.
Andy:
There’s the demo and then there’s the actual task of scaling and implementing it. I think what we’re really here to talk about today is kind of what happens, really what happens when you have that success in implementing these systems and now your entire organization is running on AI in some capacity and then, well, what happens to the people and what happens to their brains basically when they’re offloading this cognitive load. And so I think that’s kind of maybe one of the underdiscussed things right now as people are kind of race to operationalize this stuff. But I mean, I think going back to that, setting the foundations, you probably need to be thinking about this too. And so I’m curious, I mean, you’ve been working with AI starting in the ’90s back in the machine learning days when it was just that, I guess, basically, right?
And so when did this whole idea of maybe cognitive atrophy, when did that kind of dawn upon you? Or when did you realize this is going to be something we’re going to need to be thinking about in addition to all the technical considerations?
Fern:
Yeah. I guess I certainly wasn’t concerned about cognitive atrophy when I was working on machine learning projects at Bell Labs. I mean, if anything, it was the opposite. My brain was about to explode, trying to understand the algorithms, et cetera.
But machine learning then wasn’t really replacing large portions of human thought, right? It was augmenting decisions. It could help to automate predictions or help identify patterns, but we were still doing most of the work and what changed with generative AI and agents is that we’re beginning to delegate those increasingly sophisticated cognitive tasks. I think I got interested thinking about when the term was coined about workslop, that really interested me. That was maybe about a year or so ago and I started to think more about that and human in the loop because now we’re not automating calculations, we’re automating, writing, coding, research, analysis. And I think that’s a very different level of cognitive delegation. And then at TDWI, where I can perform primary research, I started to look at this in some surveys that I was running. Do you think that AI is going to replace you?
That’s sort of one question, but what if you become a human in the loop? What does your day look like if you’re a human in the loop and what does it all look like five to 10 years from now? And what does that mean in terms of what you’re doing and how are you cognitively engaged?
Andy:
Right. And you brought up workslop and that’s a newer to me term than a year ago. I feel like I learned that more recently than that actually. And so you’re a little ahead of me, I think, but workslop, can you explain, what do you mean by workslop?
Fern:
Yeah. Well, it’s not my term, but I think it’s a great one. It was coined by, I think people at Stanford and BetterUp Labs, I think. And they came up with this idea that with generative AI and maybe give an example that they didn’t necessarily think of, but what I think of with generative AI, people were using it to create just even say marketing content, right? And then they would pass it to someone and the person who was the technical person would read it and say, “This makes no sense whatsoever.” And then they would have to go back and rewrite what that person wrote. And so they coined it as workslop because it was sloppy work and now it was going to take multiple hours for someone else. Yeah, it made you more productive, the marketing person, but it made the technical person less productive because now they had to spend three more hours going through what you had generated in three seconds basically.
So that’s the notion behind workslop.
Andy:
So it’s basically kind of AI generated work that maybe there’s not a whole lot behind it maybe causing more work down the line. That’s interesting. It’s kind of like how with a lot of AI outputs, they sound very confident and it sounds right, but then when you start to think about it and try to follow the logic, it doesn’t make sense. And then you think, “Well, maybe I’m just not smart enough to understand this.” And maybe the AI knows better than me maybe, so I’ll just go along with that. I think that’s a whole nother problem with this cognitive atrophy challenge, isn’t it?
Fern:
Yeah, automation bias. That’s the name of it. And there have been studies done even before AI came out and don’t ask me to name them at this point because I couldn’t tell you, but I’ve read them where they’re actually talking about that humans are more likely when they see something coming out of a machine, they just accept it. And think about it, right? If you’re the human who is reviewing, validating, that becomes very, at least to me, depending what you’re doing, it can become mind numbing. And so if your mind is numbed because you’re just seeing all this stuff, check this, check that, right? Then you’re just ultimately going to say, “Yeah, okay, that looks okay. That looks okay.” So that’s the automation bias I think that we’re being faced with here because there’s sort of good ways to interact with the AI and then there are more cognitively draining, I think, ways to act with AI.
And I’m not a neuroscientist or psychologist, but there have been studies done that show that different parts of your brain actually start to turn off a little bit when you’re using AI to generate in the case of the study that I’m thinking of, it was to generate essays, right? So you’re not thinking as much.
Andy:
Yeah. The whole thing where it’s like maybe the output it gives you maybe is right 99 out of a hundred times and because it’s right 99 of the hundred times, maybe you get more complacent and you’re not ready for that one time where it’s wrong. I can imagine that I don’t know if my thinking’s kind of on the right track there, but-
Fern:
And then think about if agents five to 10 years from now, if there are all of these multi-agent systems generating all sorts of decisions, little decisions, big decisions, you’re going to have to have some sort of tiered approach to what you’re overseeing. But even with that, the volume is going to be so big that it’s unclear to me if humans can even keep up with that depending on how quote unquote successful organizations are. And I’m not saying this is a bad thing, I’m just saying it’s something that organizations need to think about
Andy:
The transparency thing, like if you’ve got agents on top of agents, on top of agents, how far deep can you look maybe to really understand.
Fern:
What’s happening and even a simpler example, just in terms of cognitive atrophy, if I’m an analyst, I’m a data analyst and maybe I was writing SQL code, I did that a lot, but now I don’t have to do it because agents are generating the queries, right? If I’m a developer and I stop coding because agents write code or if people stop performing analysis because agents generate the reports, then my underlying skills are going to erode over time.
That’s another concern. So it’s not necessarily about individual cognitive atrophy, it’s about sort of organizational expertise atrophy because you gain skills through practice and experience. And if AI is increasingly performing those tasks that historically developed expertise, where do the future experts come from? So on the one hand, there’s the cognitive atrophy. On the other hand, there’s building of expertise. And then on the third hand, even if there are people who have the expertise and are supervising agents and have the human skills of judgment, et cetera, how many supervisors do you actually need? Maybe you had 50 analysts that were writing SQL queries, but how many supervisors do you actually need to supervise agents that are writing those SQL queries? I don’t know the answer to that at this point, but I think it’s something that leaders need to be thinking about what should people be doing in the organization and what does the redesign of their roles look like, et cetera.
Andy:
It kind of brings to mind, you hear the analogy sometimes like a car, vehicle. I don’t need to know how the internal combustion engine really works to drive it, but at the same time, I guess somewhere in the world there’s someone who knows how that works. So if something goes wrong, you can kind of tap that person and have them fix it. I mean, can you imagine a world where there’s like certain bodies of knowledge, certain skills that are just not human at all? That’s not a thing we can even comprehend or not a thing we have insight into at all. Is that something you foresee at all or am I getting too sci-fi?
Fern:
It’s sounding like a bad sci-fi show, right? Yeah. Where your ancestors built the machines that help your organization, you help your society thrive and all of a sudden something goes wrong with those machines and you didn’t really know, you don’t know how to fix them. So what happens to your society? Hopefully we’re not going to get to that point and certainly agents have a ways to go and that will I think give people time to think about what do we actually want to be and what type of work do we actually want to do? I’m not a philosopher. … You know, what’s happening here?
Andy:
You do have insight on like the technology itself and the data side of things and all that. So maybe you can answer this next question is like, how much time do you think we do have before we really do have to figure this out?
Fern:
I think we have 20 years, right? If you think about some of the problems with expertise and the fact that organizations aren’t necessarily hiring junior people to write SQL queries or do whatever else they’re doing and then because they have the agents, then the expertise erodes and then when you have the next group of people sort of coming in, people retire and then the next group comes in. So whatever that timeframe sort of looks like when we’ve moved to Agentic and we don’t have anyone to sort of supervise properly because maybe it’s 20 years, maybe it’s more than that or less than that sort of depending, but it definitely has something to do with how people are being trained now. Although you could argue that if organizations start thinking about it now and educators start thinking about it now and that we’ll have another generation that would be ready to deal with what we think AI is going to look like.
So I think it could go either way, but someone has to think about it.
Andy:
So it’s interesting because we talk about a TechChannel a lot in the enterprise computing space we cover, we talk a lot about how AI tools can kind of help bridge the skills gap and kind of that they can, if someone comes in and doesn’t know COBOL, that’s okay because the AI can translate Java to COBOL, whatever. But now in this conversation we’re talking about how maybe AI could actually create a skills gap where there’s in 20 years from now, people that know how to use SQL are gone and it’s only AI doing it. And so I don’t know, that’s just-
Fern:
If you didn’t evolve SQL from its present day form, then maybe that wouldn’t matter because the SQL is the same SQL. Then there’s the whole question about original thought, right? The people that are going to evolve and create and who are creative and those people are generally at the long tail of the normal distribution curve and AI is not necessarily about that, right? It’s about the center of the bell curve. We’ve all seen that. So that’s a whole nother aspect of this. I’ve been doing AI since the ’90s, right? I’m not anti AI so I don’t want to come across that way, but I just feel like we need to think about this and leaders need to think about this and think about what type of organization do you want? Sort of think past just the cost efficiencies and cost cutting and what AI can do to how do you want your people engaged?
What does your organization of the future look like? How are you going to train these people? How are you going to redesign their roles and have cognitive engagement, right? Good cognitive engagement is sort of like the person who is a medical researcher who’s working with AI to determine the causes of pancreatic cancer, right? That was in the news not that long ago. That’s great using both of … They’re cognitively engaged, they’re working.
The non-cognitively engaging AI I think is the call center, well, that could still be cognitively engaged if you have call center agents, machine agents who are dealing with people and only when an exception or someone’s really upset, does a call center person come on, well then they’re dealing with difficult problems and so maybe they are engaged, but then sort of fast forward 20 years and who’s overseeing the call center, someone who doesn’t know how to deal with people. I just think people need to think forward a little bit about that and think about what they … Are there AI free days? Are your analysts going to first come up with their hypothesis and then use the AI? I tend to try to say, “Don’t let your brain erode firm and try to think of what are you thinking about here? Don’t ask the AI to write it for you.
Try to think about what it means for yourself
First.” And certainly AI is going to be helpful with a lot of the repetitive tasks and there’s things that humans don’t need to get involved with necessarily, but it’s a matter of what you want the organization to look like and then are you going to upskill the people who were doing that other work? What does that all look like? So I think I was at a conference with a lot of CIOs and CTOs and they were very excited about the potential of AI and they were talking about human in the loop and they kept saying, “It doesn’t matter, it’s going to be fine, whatever they were talking about, whether that had been like agentic commerce or whatever, a human is going to be in the loop.” And that also got me thinking to them it was just the check box, a compliance sort of box.
Those people were necessarily thinking about what that human in the loop was actually doing. So then I went and said, “Well, Okay, at TDWI, we have a Teams and Salary survey every year. I’m going to put some questions in there about what people are doing now and if they’re a human in the loop and what type do they enjoy being a human in the loop, what are they doing as a human in the loop? So I asked so far, I’ve gotten about a hundred responses from, and these are data and analytics and AI professionals so they’re people that sort of know. And first overwhelmingly, no, they’re not human in the loop. A lot of them are not humans in the loop at this point, but those that are humans in the loop, they sort of enjoy it. They don’t mind validating and reviewing and because they think it’s making whatever sort of repetitive tasks they had to do easier for them and they’re okay just sort of validating.
And again, who knows if there’s going to be automation bias and they didn’t say I didn’t ask in this survey. But then when you ask them what’s going to happen five to 10 years from now with agents, they talk about how agents are going to be automating everything. But do they think that their job is going to be taken away by agents? Overwhelmingly they say no. And a lot of these people had 20 plus years of experience in the data and analytics space and they think that their job won’t be taken by agentic or any type of AI because they possess human qualities such as judgment and critical thinking and those sorts of things, which I think is true. But again, back to the point of how many people will the organization need that have judgment and critical thinking. So it brings up a lot of interesting questions, I think.
Andy:
The idea is you’re kind of staying on top of that AI stack maybe where like as the agents and AI gets more complex, takes on more tasks, you’re kind of leveling up and then maybe you’re orchestrating, you’re just kind of constantly leveling up, but how far can you level up? I mean, there’s only so much compute in the world to handle this complexity. So how far can you go? You know what I mean? And you bring up the question of how many people will be there.
Fern:
In my circles, they’re already talking about the billion-dollar company with one person at the top. But certainly people who have … I think judgment’s always going to be needed. People who can build the algorithms and be mathematical. They’re there, the strategic thinkers, they’re there.
I think that there’s lots of—the people who deal with the governance of this and having to oversee it, they’re there. So there are going to be roles, but again, what do the roles look like? And when I heard people talking about agent interns and now agents are part of my team, that’s how organizations are thinking about this and that’s fine. When I ask those hundred people, do you have agent interns as part of your team? Only two of them said yes at this point. So it’s not like this is like overwhelmingly happening at this point.
Andy:
What exactly is an agent intern?
Fern:
It’s an agent that performs a task and you supervise, you’re supervising that agent just like you would, if you with you for the summer, you’re helping that intern understand more. Some people call them agent interns. Then there’s also agent team members, right? So now it’s your team member who goes out and figures out who the best supplier is. And now that’s done by an agent who’s going out and gathering information and comparing prices and satisfaction levels, et cetera, and returning a decision. I think it’s XYZ company. So that’s now your procurement agent who’s part of your team.
Andy:
So these can almost, and we’re getting a little away from the core topic but this is just interesting. So these agents can maybe kind of evolve like an actual employee would, in theory, under one kind of model.
Fern:
Right, that’s how I’ve heard a number of organizations talking about this. Even if the agent ultimately isn’t accountable, they’re doing the work.
Andy:
So I mean, ultimately, is there a world where this is all kind of okay maybe that we don’t have experts in whatever SQL, name your discipline. I mean, there’s been a lot of technological revolutions where certain aspects of our work have been replaced and we found other jobs, other jobs have been created and there’s probably other jobs that we aren’t even imagining right now that will be created. But I don’t know. Is there a world where that’s possible where maybe this is all new and we’re all freaking out a little bit-
Fern:
I think with any revolution, you think about the industrial revolution, right, all these skilled craftsmen and whatnot were sort of put aside their judgment and expertise were still needed in certain areas, but there was certainly a transition period. So I guess we would be in this transition period, not right now, but like in the coming years, we’re probably going to end up being in some transition period that may be tough for some people and for other people it will be exciting. It just depends how business and our society sort of rise to the occasion. I mean, the industrial revolution ultimately created a lot of jobs and even you think about, I guess GPS, now people can’t read maps or a calculator. Now people have a hard time with arithmetic. That’s not, I guess, at the same level as this is, but some of the ideas are the same.
Andy:
The calculator is an interesting example because when I was in grade school, I was told, “Don’t rely on the calculator. You’re not going to always have a calculator when you grow up, learn your arithmetic.” And I think that was good advice to follow, but it was also wrong because we always have a calculator in our pocket, but I mean, maybe it was still worth learning the arithmetic anyway because that helps us and helps our cognition in other ways maybe, even if we’re not doing two plus two in our heads all the time.
Fern:
I think those things are good. I mean, likewise, I was a geophysical oceanographer and a geologist before that. I sort of needed to know how to read a map and have spatial awareness as opposed to using a GPS. So you could argue that it’s a worthwhile skill or not argue that it’s a worthwhile skill. I think there’s going to be a lot of debates about that sort of thing going forward, right? Is SQL, do I really need that skill? Does that raise to the calculator? You know what I mean?
Andy:
I guess there’s also the question of like your cognitive resources. I mean, by using the calculator, I may be saving mental energy for something else that I can do. So I don’t know, maybe that’s a factor too.
Fern:
Yeah. I mean, you could argue that for sure. I think there’s going to be a lot of debate and arguments about all of this. And I honestly don’t know, I’m still thinking of where I sit with this. I’m just throwing the questions out there at this point and thinking, sort of taking things to some extremes that I was talking to a guy the other night who was telling me, “Don’t think of it that way. Think of it as all of the new possibilities.” He was at his dentist’s office the other day and he had an x-ray and then the dentist’s x-ray software had AI in it and pointed out where the cavities were and which had to be filled now and which didn’t. So it was the partnership between the dentist basically and their software. It didn’t diminish the dentist, but it made it easier for them. And he had all sorts of other examples like that where you could get excited about what the future could look like.
Andy:
Now that’s where I hope that the dentist is not falling victim to automation bias. I’m assuming that they’re well trained and they have these, their organization has made sure that cognitive atrophy is not happening, that they’re still staying on top of their skills and yeah, that could be, maybe that’s a good model for kind of how that could work in enterprises too. But like I’d like to get into that a little bit more to like what companies, organizations should be doing or just people on a personal level to make sure that they’re staying sharp and kind of keeping the skills they need available to them. I mean, you talked about like this concept of AI-free days. Are people doing that?
Fern:
They’re talking about it. I haven’t seen … How do you enforce that? There’s so much shadow AI out there. But I think that organizations, they need to create cultures where questioning AI is expected. So that includes the fact that they need to have visibility into how decisions are made and mechanisms for auditing outputs and incentives for employees to challenge recommendations. Otherwise, human in the loop quickly becomes sort of human after the loop and the automation bias sort of takes over. I have heard of some companies asking their analysts to come up with hypotheses first before asking the AI to analyze the data. On the other hand, I’ve also heard of companies that say they’re going to be laying off all of their analysts because they’re going to have the business users use natural language interfaces to data and just ask all the questions themselves, right?
Real democratization of analytics. So there are different companies, different approaches I think. There are companies that put all sorts of data and AI literacy programs in place that are now extending this to think about ethics and the ethics of AI of which you could argue that this cognitive atrophy is an ethical question on some level, right?
So some of them are doing that. I guess it depends what kind of company you’re going to work for and what companies, which company type can be competitive in the future.
Andy:
Yeah. The ethical implications, I suppose. I mean, if we’re talking about is it ethical to … If we’re talking about, I guess societal implications, there’s a lot of ethics there, like how is this going to play out? So gosh, yeah, there’s that whole side of things too.
Fern:
And it’s one part of our economy, right? I mean, Microsoft did a study where they looked at what roles will still be needed and that was interesting. They just sort of went through content creators, X, plumbers, check. They had a methodology. It was pretty interesting of how they actually went through all of these roles. I mean, so certainly, unless a robot’s going to do physical therapy, which is I guess entirely possible, there’s physical therapists, there’s plumbers, there’s electricians, there’s physicians. I don’t think people are going to really … There’s lots and lots of roles that still exist. Go ahead. Sorry,
Andy:
Even plumbers, I mean, right now that’s the job seems like probably be pretty unaffected by AI, but I mean, you can take a picture of … I’ve done this, like take a picture of the piping under your sink and ask Claude a question, like, “How do I fix that? ” And it’ll give you an answer. That’s true. So I mean, maybe they’ll have to be thinking about the cognitive atrophy thing too and making sure they’re not following victim to automation bias. It’s just kind of crazy to think how far everything the tentacles can stretch.
Speaker 1:
Yes.
Andy:
So we’ve been through all these bubbles, the. Com bubble, the cloud big data. I mean, those weren’t really bubbles necessarily, but these revolutions, I should say, I’m not calling AI a bubble right now, but I mean, what do you think is actually going to happen here? How do you see this revolution playing out? I mean, is our future like Wally where we’re just kind of passive and letting AI take care of everything and we’re kind of in our recliners kicking back and maybe that’s nice sometimes, but maybe that kind of is not great at the same time or is it more Star Trek? I don’t know.
Fern:
Reality is always much more complicated than what we think will happen. And in the survey that I was talking about, what really struck me most, at least so far, was that respondents overwhelmingly believe that agents are going to perform more work while humans remain responsible for the outcomes. So they expect that agents are going to write code, generate reports, automate workflows, they’re going to support decisions. And at the same time, they expect that humans are going to provide judgment and accountability and governance and business context. I do think that the unresolved question is really how many humans, organizations are going to need to perform those functions. And interestingly, in the survey that I was talking about, a number of the respondents explicitly predicted fewer developers, flatter headcount growth and more work accomplished by fewer people. So that suggests workforce compression rather than it’s not like workforce replacement but compression.
So if we just talk about the future of knowledge work, that’s going to involve a shift from execution towards oversight governance and decision makers, and the organizations that succeed, I don’t think will be the ones that automate everything. I think they’re going to be the ones that successfully combine AI capabilities with human judgment, trust, expertise and accountability. So you’re going to have to ensure that humans remain capable of exercising the judgment that keeps the whole human in the loop thing functioning. And as I said, I think it’s going to be a question of thinking through what roles look like, the importance of expertise, how to develop expertise, how to bring people, younger people through this evolving AI landscape so that they do have expertise and even though that expertise may look different, but I think that those, at least maybe that’s the optimistic side of me, thinks that those are the companies that are going to succeed because they have human engagement, they’re using automation so they’re balancing it properly
Andy:
Balance. Balance is always key, isn’t it?
Fern:
Yes, indeed.
Andy:
Always the answer. You can always count on balance. Well, Dr. Fern Halper, thank you so much for joining me here today. It’s just fascinating to kind of explore this frontier and I think it’s the human cognition side of the whole AI discussion is something people should probably start thinking about. And of course, your book, Data Makes the World Go Around addresses a lot of the stuff that TechChannel covers as well. So I think our audience will have a solid interest in that. So I think you can find that on-
Fern:
Barnes and Noble, any of your favorite online retailers.
Andy:
Absolutely. All right. Well, Fern, thanks so much for joining me.
Fern:
Thank you Andy. It’s been a pleasure talking to you.
Andy:
You as well. Take care.