Why Your Enterprise Data Isn’t Good Enough for AI, With Craig Mullins
Craig Mullins, author of "The Cost of Good Enough Data: Why Modern Architectures Fail at Scale," goes on IT Social Hour to explain how AI is changing data management standards
Listen to the audio-only version
The following transcript has been edited for clarity:
Andy:
Hello, welcome to another episode of IT Social Hour. I’m Andy Wig, senior editor at TechChannel. And today we have on the program Craig Mullins. He’s a database expert who is a regular contributor at TechChannel. You’ve probably seen him if you’re a reader. And he is also the author of the new book, “The Cost of Good Enough Data: Why Modern Architectures Fail at Scale.” And there you go. There’s the book. And so we’re going to talk about that and this new challenge or old challenge really that organizations are facing or something they’re being forced to face right now, which is data debt that has accumulated. Now AI has kind of come along and now they have to reckon with that whole situation. So Craig, thanks for coming on the program and explaining what’s going on out there with data debt and AI.
Craig:
Thanks for having me. I look forward to having a great conversation.
Andy:
Yeah. And before we get into it, maybe we should just have you tell our audience a little bit about your background and what you do as a database expert.
Craig:
Sure. Happy to do that. Presently, I’m an independent consultant. I have my own company, Mullins Consulting Inc, where I focus on data related technologies. I do a lot of work on IBM Z mainframes, DB2 for ZOS, but really anything data related. I’ve been recognized by IBM as a Gold Consultant. And what that is is there’s a group of technical experts that IBM has recognized and we operate independently from IBM or any other vendor, but we’ve been kind of blessed as having deep expertise on IBM technology. So IBM will at times bring us in on projects, recommend us, and it’s been kind of a boon to my consultancy. A little bit more about my background. I started out as a programmer doing COBOL programming, moved to database administration for both IMS and DB2. Worked at a lot of software vendors, Platinum Technology, which has been acquired and long gone, but also BMC Software, Neon.
And for a period of time, I worked as a research director at Gartner covering database systems and DBA issues. So do a lot of writing, written 10 books now. And as you mentioned, do a lot of writing for tech channels.
Andy:
Okay.
Craig:
Keeping busy.
Andy:
I think we’ve firmly established your expertise here. I think people have a good reason to listen to you. So yeah, based on the title of your book, I’m assuming there was a time when maybe there was good enough data, quote unquote, good enough data was actually good enough. Now AI has come along and kind of changed that. So can you maybe set the table by explaining what AI has done to the standard of good enough data?
Craig:
Sure, sure. When you think about it, AI has kind of completely changed the conversation about data standards. For a long time, organizations kind of lived by an unwritten rule. If the data was good enough to run the application, if it produced the monthly reports, if we were able to still do business, it was good enough. And yeah, maybe there were some inconsistencies—duplicate records here, some missing values there, some questionable metadata, but people learned to work around that. It wasn’t ideal, but it wasn’t catastrophic either.
Now we bring AI into the mix and IBM coined this phrase that I really like. It’s called there’s no AI without IA, where IA is your information architecture. So the information, your data is what feeds AI. AI doesn’t execute business rules that developers have written. It learns from the data that it’s given that it ingests and it uses that data to make recommendations and predictions and provide answers, write programs. And if you think about it, if the data that it’s getting is flawed, then it’s going to make problematic decisions. AI isn’t going to fix your data. AI is going to use that data and make assumptions that that is accurate, what you wanted to feed to me as the AI guy, and I will build based on that. And that’s why good enough isn’t really good enough anymore. Think about it like, all right, maybe you’ve got a misspelled customer name in an operational system. And that might be annoying. And every time you send out a report, the customer gets M-U-L-L-E-N-S instead of M-U-L-L-I-N-S. And maybe there’s a product code that’s inconsistent and you cause a reporting problem. But if those inconsistencies get built into the AI, then it’s going to make all of its decisions based on inaccurate data. And that’s why AI really is raising the value of data standards.
And a standard, what do I mean by that? Well, it used to be something that a DBA or a data architect cared about. We build our standards for our organization, but that’s changed because it’s a business issue today. It’s those consistent definitions and the strong metadata and the clear lineage and the data governance that contribute to whether that AI application can really produce answers that people trust.
Andy:
And so it sounds like part of the problem is when there’s this inconsistency in the data like you mentioned, maybe in the past or in the present still, there’s a human that will recognize that inconsistency and deal with it while an AI will just try to make that inconsistency maybe make sense, basically take both data points as sources of truth and then try to fudge it so it makes sense even though it’s completely wrong, but then also it might sound kind of confident in the meantime. Am I on the right track there?
Craig:
Yeah, I think you’re kind of on the right track. I mean a couple nuances there. You’re talking about AI isn’t really going to think. It’s going to, based on its data, make some assumptions and move forward. So it’s not like it’s a human brain where you said a human person can sit down, see something’s wrong and make a change. Unfortunately, there hasn’t been enough of that or we wouldn’t be dealing with good enough data. The data would’ve been fixed. So humans a lot of times just let it slide. And we’re capable of operating in this gray area that AI doesn’t really have a comprehension of a gray area. It’s just going to take it, accept it, move forward. This is the truth. This is what I’m building all my assumptions of from here forward. So you wind up with something a little bit bad that gets magnified.
So in the book I talk about garbage in, garbage out, but it’s with AI, garbage in, amplified garbage out. So it gets amplified and moved out and creates things even worse as it migrates throughout your systems.
Andy:
What start says maybe a little problem, maybe affecting one customer becomes something that gets everywhere. So you mentioned letting it slide as you deal with data imperfections. So I guess that’s really getting to the debt thing with technical debt where there’s things you need to really update, but you find workarounds, you let it slide. And then eventually though, I guess you got to pay the piper.
Craig:
True. The concept of technical debt has been around for a while and a lot of that is as our systems get old and build up, eventually you’ve got to replace them and get something new. You add fresh stuff to it and then it’s more of a hardware type issue. But with data debt, it’s all these years of neglect. And in some cases it’s even worse than good enough. It’s just bad. One of the things that I often find is metadata, which is the data that describes your data. And there’s a lot more to it and I know that people out there who deal with metadata for their career aren’t going to like that definition, but it’s enough. So it’s the data that describes what the data is and it is what AI will hang onto in terms of describing things. So if you’re going to feed your database into your AI systems and your columns, for example, are C1, C2, C3, C4, instead of customer name, address, phone number, AI is going to be working in the dark. You’re going to have to give it more information than that.
Andy:
What’s C1?
Craig:
Yeah. And you’d be surprised at the number of databases I see out there that still are built like the early ’80s when we were worried about conserving system space, whereas now we can have extremely long names. Use those extremely long names. That may be difficult a little bit from programmers, but for AI it’s ideal. And even with programmers now we’re using IDEs. We’re not typing all these things in ourselves. So the more descriptive things are, the better it’s going to be removing that debt from our systems. Repositories, what’s the other word that I’m looking for? Well, data catalogs are another term that they’re using now, which describe all of the additional metadata. The stuff I was talking about there was the technical metadata that allows the database system to hang and create. But then there is the definition of what column 38 in this particular report means. A lot of organizations have neglected that.
The repositories that were being built in the ’90s, the data dictionaries that were being built in the ’80s have been neglected. The data catalog, every decade we got a different name for essentially the same thing. But the data catalog is the one today which is kind of generating things for you in some cases using AI, in some cases allowing us to do things non-invasively and create the things we need for data governance. So those type of things need to be better managed in order to move forward and remove this data debt. And it’s something that I think AI is beneficial enough that organizations are going to stop just paying lip service to it and actually do something about it.
Andy:
Okay. And that dovetails great into my next question, which was just why do organizations care so much about making AI work at scale in the first place? Or why should they? What results are they chasing?
Craig:
Well, I think there’s a lot of different answers to that question. What results are they chasing? Depends on the organization and how well they’ve really adopted their AI practice. So there’s this meme out there that someone screamed, “What do we want? ” And the executives are saying, “AI.” “When do we want it?” “Now.” “What do we want it to do?” “I don’t know.” And unfortunately that rings true in a lot of cases. It’s the latest hype. I want AI, I need it now. So the better response to that is not where can we use AI? It’s like, what business problems are we trying to solve? And that dovetails into the conversation about will AI replace people? And you see some of the news stories these days. I think Microsoft just laid off a bunch of people and kind of saying, and that’s related to AI and they’re thinking AI is going to replace people.
Oracle’s been doing a lot of that too, but it shouldn’t be that. AI is more about making people better at what they do. It should help your employees make better decisions, find information faster, identify patterns that they would normally miss and automate the routine work so they can focus on higher-value activities. And for large enterprises, that’s really where the ROI comes from. It’s automating your business processes in an intelligent manner that makes your employees more effective and more productive.
As a DBA guy, I go back a number of decades and it was all about automation. Automate your processes. And that was great. And a lot of organizations did that. This is more intelligently automating your processes, where the processes in some cases can make decisions for themselves. And that always gets into the, do I turn my system over to the automation? Do I turn my system over to the AI without a human buffer? No, never. You trust it to a certain extent, but verify it. There has to be a human in the loop. And the more effectively we can create that mechanism for the human to be involved in the loop, the better off the AI is going to be in terms of how it’s going to automate our systems and make things more practical. I think I just went down a rabbit hole there from the question you asked.
Andy:
No, that was great. Yeah. I think you established that there’s not a lot of argument that these are results that people want that is not hard to see the value of this automation and intelligent management and all that. So my question is, and people have been talking about AI not just for the last three, four years, for the last 20, 30 years in different waves and stuff. And so people have seen this AI age coming at some point, but yet we’re here and so many organizations, most maybe, aren’t prepared to leverage their data to take advantage of AI. So do you have any thoughts on why so many enterprises are in that situation? Was it just something that was really, was the technology so the development of the technology that unpredictable that people were kind of left flatfooted a little bit?
Craig:
Yeah, probably. But I’m going to be a little bit more jaded about my answer. It’s not just AI, it’s everything. Organizations really are, they’re concerned about conducting business and making their numbers for the next quarter. And if whatever you’re proposing to them doesn’t do that, the chances of it happening are slimmer and slimmer and slimmer. So I mean the next thing that’s going to happen is quantum. Quantum technology’s around the corner and people aren’t preparing for that. People don’t even understand that. And that’s fine. But to dovetail it back to AI, I think that you’re right, the technology wasn’t there. There was this aha moment and it’s OpenAI with ChatGPT is who I give credit for it. That’s when everyone and they said, “Hey, use this thing, it’s AI.” And you ask it a question and it’s like you look at the answer and you go, “Oh my God, was that really a computer that gave me that?”
And that is what caused this whole juggernaut to happen, the ability of these LLMs to produce really high-quality information. And yeah, there are hallucinations and yet there are issues. But if 20 years ago you’d gone to somebody and said, “Sit down here and write my business report based on this input from my manager,” and they spit out a template that’s 80% there, you’d have been, “My Lord, that’s wonderful.” And today it’s almost yawn, yawn, take it for granted.
The next step is as, and I don’t know whether it’s tomorrow, next year, or decades down the road where AI becomes, for lack of a better word, sentient and you can’t really tell the difference between it and a human being and the generalized AI moment. It’s coming. Will it be in my lifetime? I don’t know, but again, it’s going to be something where we’re not prepared when it gets here.
Andy:
Yeah. I mean, you mentioned we’ve gotten to the yawn part so fast. It does seem fast, does it? I mean, this was just mind-blowing technology three or four years ago. I mean, still is, I think. But I mean, I guess you get accustomed to these things so fast. And you mentioned when, if I guess you’re posing it as when it becomes sentient. Do you think people will adjust to that just as quickly or is that going to be a whole ‘nother… Are our minds going to be so blown that there’s no recovering from that one?
Craig:
Well, I guess a lot of it depends on how we have built safeguards. Are we going to let AI take over? And now we’re getting into science fiction, but it may not be science fiction, it may actually be science within our lifetime. And I don’t know that that’s necessarily going to be such a yawn moment because you’ve got all kinds of ethical questions that then enter into it. If it’s sentient, does it have the same rights that we do? And I don’t know where we go with that. That’s for the philosophers. Well, I hope it’s for the philosophers and not the politicians, but it’s probably going to be for the politicians and that’s scary on another level.
Andy:
Yeah. Well, hopefully those politicians are listening to the right people, I guess. That’s about the best we can hope for.
Craig:
Who are the right people is the question.
Andy:
Maybe you and me.
Craig:
Yeah, you and me. There we go.
Andy:
Maybe just you. Maybe not me. And so as you talk to customers, I’m curious, are they surprised to learn that their data isn’t good enough to scale AI the way they want to?
Craig:
Some are, some aren’t. And it all depends on who the person is. Generally speaking, the higher up in management you go, the more you’re insulated from the garbage. So if I’m talking to executives, I’m talking to middle level people all the way up to the CIO, those people are more surprised. The people who deal with data on a daily basis, whether it’s your end users, your data management people, they know they’ve been dealing with it forever. There’s been books, I’m looking over here at my shelf of books and I’ve got all these data strategy books and data quality books. And they go back 20, 30 years that have been talking about the problems with our databases that we don’t fix. And the one thing I give a lot of presentations and when I get to the point where I’m talking about data, I’ll say, all right, I challenge anyone in this room, go back to your organization, find a database, find a table where you’re pretty comfortable with understanding what that data is and bring it up and browse it.
I guarantee you within the first 100, 200 records, you’re going to find problems. And I don’t get pushback. Everyone’s sort of in the room going, “Yeah, you’re right. The data isn’t as clean as we want it to be or think it should be, but it is good enough to run our systems,” which gets back to the whole idea of good enough data. It all comes back to the data.
Andy:
Yeah. And so they’re admitting this, but at the same time before, I mean again, humans were able to deal with those imperfections and adjust because they’re agile and they do think while AI, it’s a different situation. I think I’m getting it. So as far as scaling AI successfully then, getting to the solutions portion of the program, I think you touched on some of it a little bit, but what are the main factors organizations should be addressing if they really want to get their hands around this thing?
Craig:
I think that I would say it is a management problem, a data management problem, and that needs to be addressed first. So what does your organization do in terms of data governance? What are the policies and procedures you have in place and the way that you’ve enacted the governance of your data? Who or what organization is in charge of the quality of that data? And what needs to be done to the current systems to ensure that, let’s say we make a change of the data to correct it, that it doesn’t just get populated wrongly again in the future? So it’s that information architecture that needs to be built behind these vast systems of data that we have. We have a lot of data, whether it is the systems of record that run our businesses or it’s the data lakes where we’ve accumulated lots of data from multiple sources over the years.
But what we do to make sure that these things are accurate, managed, governed and usable in such a way that we don’t have to worry about it not being good enough has been neglected in my opinion over the years. So it’s this, all right, heads down. I don’t want to blame any one type of technology, but move it to the cloud, move it, do it quick. DevOps, get it in, build it, run it quick. And DevOps is one of these things where I always say for too long, DevOps was Dev (yelling) Ops (whispering). It was let’s build it and put it out that quick. Whereas the operations are how do we make sure that the data that we’re using is accurate, that it’s controlled, managed, and governed and it moves through the process accurately and correctly? And that’s getting better, but it needs to get a lot more better in terms of the overall information architecture.
The other thing is we’re always enamored with the shiny new bauble. So I’ll make a statement that might cause some people to disagree, but a relational database is a relational database. Yeah, I have my favorites. I love DB2 for z/OS. It’s great. I’ve used it for years. But you could do the same thing with Oracle. You can do the same thing with Postgres. And to look into the way in which systems are developed at organizations, generally speaking, I see anywhere from three to 10 different database management systems, relational database management systems.
And why? All right, give me one on z/OS, give me one on Linux. That should be enough. And move forward from there. The other thing, and I’m not saying anything negative about the technology or the company, but something that really amazes me is Snowflake. Its market valuation is through the roof. It’s a relational database management system. Yeah, they’ve done some interesting things with it and scalability type things. But why is the market valuation of a company that has one relational database management system so far out of whack when you compare it with an IBM, which has not just a relational database management system, but all this other stuff? So stop having to have what’s this shiny new bauble that is the same almost as what I already have implemented and move forward. Use what you can have, move forward with what you have.
Maybe a little bit off track here is COBOL. The systems that are built with COBOL on mainframes have been running for decades, and they contain the business logic, the rules that run your organization if you’re a large enough enterprise. Now everyone’s talking about, well, we need to replace COBOL. Why? I mean seriously, why? It works. And they say, “Well, we don’t have enough COBOL programmers. Have you looked at how easy it is to code COBOL?” When I was in college, they didn’t teach COBOL and that was in the early ’80s. So I learned 22 languages and not COBOL. First job out of college, COBOL. Didn’t take me long at all. I was up and running with it. If you can code in Java, you can code in COBOL. If you can code in C, you can code in COBOL. So the shiny object syndrome needs to go away and all the money we’d spend on converting our COBOL systems. Instead, let’s build our data infrastructure appropriately so we can support AI and move from good enough data to great data.
Andy:
Okay. So instead of maybe worrying about translating all your COBOL and converting it, just learn COBOL. It might actually be faster it sounds like, or more effective in some cases or many cases. Interesting.
Craig:
If I were talking to a college graduate who was having a hard time getting a job, I’d say learn COBOL, get a COBOL book, take a COBOL class, and I bet you could get hired tomorrow.
Andy:
I’m also wondering as all these organizations struggle to scale up AI, it seems like they might take solace in the fact that it seems like everyone’s struggling to scale up AI. Does that seem right? Is it everyone that’s struggling or I mean, I guess you do hear success stories, but I’m wondering if can we take solace in the fact that we’re all kind of in the same boat? Or is there someone that’s actually way ahead and they’re going to scale up faster than anybody and take over whatever industry that is? I don’t know.
Craig:
I mean, sure. I mean, misery loves company, right? So I guess you can take solace in everybody having struggles and, I don’t know. It’s not like I’ve gone out and taken a look at who’s ahead and who isn’t and who might be ready to change the industries, but I guess it’s possible that there are organizations that are doing better than others. In fact, it’s probably probable. And instead of taking solace in the fact that a lot of people are struggling, if I were running a business, I would be doing everything I could in order to change that equation from taking solace to being that person who’s out in front. And how would I do that? I guess you can tell what I’m going to say. Address your data architecture. Make sure that you’ve got quality data. Your information architecture is what guides the quality of your AI.
So before you say, buy the latest LLM, generate a new RAG model, ingest all your data into it, clean it all up. Make sure that what’s feeding it is accurate, worthwhile and good so that when you actually do it, you aren’t one of these people who’s now looking for the misery loves company because we just ingested all kinds of bad data.
Andy:
So yeah, don’t treat it as a misery loves company situation. Treat it as an opportunity. The fact that while there are other people that are in the same boat, let’s not be in that boat, I guess.
Mullins: Right.
Andy: Yeah. So last question I wanted to ask you is just about, it’s a job question and we touched on it a little bit earlier, about is AI going to replace jobs and that kind of thing. And I want to drill down a little bit more on that and talk about specifically enterprise architects, DBAs, data architects. How should they be changing their mindsets or preparing given the fact that just the whole practice of data management has to change as you’ve outlined here? What should people who work with data every day in this way do to make sure their skills stay relevant?
Craig:
Yeah, that’s a great question. And I think number one is do not neglect the core fundamentals. By that I’m talking about things like data modeling, SQL proficiency, performance tuning, security. Understanding how those things work for the data in your organization are important, but they’re table stakes. They’re not really differentiators. You move up to say, all right, what are the more durable skills that I should be learning now? And it would be things like cloud architecture, in-depth security, data governance as I touched upon, automation, DevOps, and not just those sort of technical type issues, but the ability to communicate to executives and non-technical leaders to be able to explain the trade-offs between technology X and technology Y. What’s the difference between a cloud architecture and an on-premises? And what’s a hybrid architecture and why should I care about all three of these? What are the cost implications?
A lot of times people have said, we’re moving from on-premises to the cloud because it’s cheaper. And I’m standing in the corner laughing because they get the bill and it’s not cheaper. It’s all going to cost you and someone’s going to charge for it. So implement them based on where they make the most sense, understanding how they’re charged so that you’re not with a cloud, why do we move to the cloud? Well, we move to the cloud because it’s easier to scale and we can scale flexibly. And it’s elastic. So things have issues and we scale up, not caring about what that means to the bill. Believe me, if you scale up, that cloud service provider isn’t just going to incur that cost on their own, they’re going to bill you. So getting into “good enough,” I’ve been writing good enough SQL statements. Yeah, they’re not really that effective in terms, I mean that efficient, but they’re working.
Then we scale up and we scale up and we get more data and we get more data and that SQL statement that used to run sub seconds is taking seconds or even minutes. And now that bill goes up and up and up. So worry about things like that, understanding the difference between coding things to work and coding things efficiently and being able to transfer it into a business language that understands pricing. And you talk to a lot of DBAs and it’s like, yeah, I can tune SQL and I can make it more efficient. And I’ll say, all right, what does that mean in terms of the amount of money that was saved?
Some people will say, well, I saved this number of MIPs or this number of MSUs. So I have this number that’s magic, that’s the number of dollars per MSU, so I saved that. It’s like, no, you didn’t. You don’t understand how mainframes are priced. Is it a ruling for our average price? Is it a tailor-fit pricing price? And one’s going to have a different answer than the other. And most technicians have no idea what those pricing policies are and what the licensing is in their organization. Learn it, find it, ingest it and make it part of your vocabulary to make yourself more employable and more valuable to your employer. I guess I’m saying that you want to transfer from being a keeper of systems and be more of a strategic translator between business outcomes and platform choices and your AI era operations. And if you can do that, then you will be viewed as valuable.
So what you have to avoid though is don’t jettison the core fundamentals. They underlie everything. So you keep that and you add these things on. And of course learning about things like AI and machine learning and GenAI and where that makes sense to use. That’s kind of table stakes these days, but putting them all together, understanding it with pricing, understanding the impact to the business and what you’re charged—that’s a valuable employee.
Andy:
So it sounds like it’s kind of ironic where to be more AI relevant, you actually have to be more involved in the real world. I don’t know, almost have your foot in both worlds really and be that bridge between maybe the executives or the people managing the company and the AI and being like, I don’t know, kind of that knower who can maybe see into the systems a little bit, but they can also see into the real world and massage things.
Craig:
It’s a delicate balance and too often people want to stay in their comfort zone. And it’s like, I’m a DBA and I understand all the bits and bytes of Db2 and I can change ZPARMs and know exactly what it means, how it’s going to impact the system. Okay, but what does that mean in terms of the business outcome and the cost savings, if any? Don’t care. And then there are the people who are the business process people, and either they came out of a business school or if they came from IT, they moved away from the technology to the business and they’re clueless about the actual technology decisions behind the scenes. Someone who can communicate in both worlds is quite valuable.
Andy:
Yeah. Well, it sounds like there’s a lot of discussion about the dangers facing employees right now. What are their jobs going to look like? What is also kind of an opportunity, I guess, to position yourself to—
Craig:
Absolutely.
Andy:
To thrive in this brave new world we find ourselves in.
Craig:
Yeah. I mean one of the things that people always ask me too is, are DBAs obsolete? Are we getting rid of the job of the DBA? And I’ve got all kinds of things that I say about that, but the answer’s no, it’s not. You’ve got a database system, you need to administer it. Even if you go 20 years in the future. I’ll make a statement—I won’t be working in IT anymore. But 20 years in the future, if AI becomes sentient, it will be an AI DBA administering the system. DBAs have to exist because they administer the databases that store the data that our business relies on to conduct business.
Andy:
They’re not going anywhere.
Andy:
There’ll always be a DBA, whether it’s digital or human.
Craig:
Yeah, it’ll be a robot, right?
Andy:
Well, this has been great. I really hope people can take away a couple pointers from this that can either help them in their own careers or on an organizational level help them get up to speed with AI and keep up and make sure they’re also positioned to continue to thrive. So Craig, thank you so much for joining me here on IT Social Hour. This has been a lot of fun.
Craig:
Thanks for inviting me. I’ve definitely enjoyed the conversation. And as you can probably tell, I could talk about data for hours.
Andy:
And for our audience out there, I just need to let you know you can find more content like this at techchannel.com. Go to techchannel.com/subscribe to get on our newsletter list and make sure you don’t miss any of it. And it’s all free. So there you go. All right, catch you all later. Bye.