The Importance of Refactoring
Charlie Guarino: Hi everybody, this is Charlie Guarino. Welcome to another edition of TechTalk SMB. This month I am joined by what I'm going to say is a household legend in our industry: Mr. Ted Holt. He has been in our community for years and years. Ted has worked in the IT industry for over 40 years, primarily with IBM Midrange Systems, but you might know him as when he served as senior technical editor and writer for IT Jungle. In addition to that, Ted has taught at the university, community college, and vocational technical school levels as well. He is also the author and co-author of six books and has published hundreds and hundreds of articles and technical tips in various publications. Today he works as a senior software developer for Lane Home Furnishings, located in Mississippi. Ted, what a pleasure it is to be with you here today.
Ted Holt: It's my pleasure, Charlie. Thanks for having me.
Charlie: Thank you, Ted. Ted, you and I go back quite a ways and today I want to focus on a topic that I know is near and dear to your heart, and that is source code refactoring. But before we get into that, I know that—because we have this friendship—I know some interesting facts about you I think people might be interested in learning about. I remember once when you and I were on trip together and you were collecting postcards to mail out. I did ask you who the postcards were sent out to and you said well just some friends but I'm not quite sure you even knew you were sending them to—just some random people I suppose. Then you later told me that you belong to this organization or this website that facilitates sending postcards to people around the world. Tell me more about that.
Ted: All right well back in the 90s when I would go to COMMON or something like that, I would take my wife and some of the kids along. We would do a little sightseeing while we were there. My wife and I found out that postcards had better photographs than what we could make with a camera—of course that was the days before digital photography. So we bought a lot of postcards. I've been collecting postcards for 30 some years I guess, and a few years ago I found this website called Postcrossing.com. The way it work is you sign up and they give you some names and addresses and you send postcards to these people. When those people receive the postcards, they register them and then Postcrossing gives your name and address to some people, and then the next thing you know you're getting postcards in the mail. So when I go home one day I may have a postcard from Germany, I may have a postcard from Russia, I may have a postcard from Canada, may have one from the U.S.—and so I have gotten to know some of these people long distance. I've learned a lot about the world and been exposed to all sorts of things I never heard about otherwise. So it's a fun little hobby and it's not terribly expensive.
Charlie: How many postcards do you think you have in your collection these days?
Ted: I don't know, several thousand. I'm going to guess probably somewhere around 4,000 to 5,000 postcards.
Charlie: Wow. That's an amazing hobby. Do you get repeat postcards from some of the same people?
Ted: Yeah. There's a few of them; we sort of have become regulars. The town I live in, Tupelo, is Elvis Presley's birthplace. The little house where he was born is five miles from my house. So I mail Elvis postcards all over the world and there are four people now that I send an Elvis postcard to every month, and they send me postcards from where they live. So yeah, some of them have become regulars.
Charlie: You live in two different orbits, so to speak. You have obviously the IBM i community, but you have this whole other persona that we don't even—we don't know much about. Or maybe other people do, but I don't. I think it's fascinating.
Ted: Well the IBM i stuff is sort of a sideline. I've got a rich and full life and I could fill a dozen lifetimes with all the interests I have.
Charlie: Wow. I think that's—first of all I think that's wonderful and that's great life advice. Thank you. What a great quote. I actually might steal that one from you. It's such a repeatable, great quote. Thank you for sharing that quote. Let's get into refactoring though because this is why I wanted to really talk to you, and this to me is such a fascinating topic, especially today because we hear so many different companies struggling with their current code base and particularly old programs how they're—they're struggling to make these things modern with all the pressure they're getting from either internal resources or outside external, whatever the case is. And they can't necessarily deliver new functions fast enough because the code itself is restricting them from doing things. So let's get into this, Ted. The first thing I want to ask you, just from the top, is what is refactoring? Give me me some information about that.
Ted: Well refactoring is rewriting a piece of source code without changing the behavior. In other words you spend time redoing the source code and when you're finished, it doesn't do anything differently from what it did before. So the internals stay the same but the behaviors—I mean the internals change but the behavior stays the same. I kind of think of it like editing a magazine article. I've written a lot of magazine articles and my first draft is pretty rough. It has my ideas that I want, but then I go back and I polish it to make it more understandable, and we're doing the same thing with source code.
Charlie: What's the point of that? What's—why I am doing that? What's the point of that?
Ted: Well the problem with a lot of this old code is it's hard to work on. When I taught at university, we would tell our students that there's good programs and there's bad programs. They may both produce the right results, but a program can still be bad. The way you know a good program is it's easy to read, understand, modify, and debug. And so what we want to do is we want to take a program that's not living up to that standard, we want to bring it to that standard so that we will be able to enhance it or to change it according to the business needs.
Charlie: You know we got into this discussion of refactoring, Ted, and part of it maybe is how do we identify which program to refactor? But before we even go into that, I have a question for you. You know we use a lot of different terms in our industry and one that I think gets used interchangeably with refactoring is modernization, modernizing. Okay, is there a difference between those two terms? Should I be using one instead of another one at any given time?
Ted: Well refactoring and modernizing are not necessarily the same thing. Refactoring is rewriting source code in such a way that the internals change, but the external behavior doesn't. That could be a part of. It could be a tool that you use in the modernization process, but the two are not exactly related—or they're not synonymous, for sure.
Charlie: But if somebody does use the terms interchangeably, they wouldn't be entirely wrong because I know a lot of people just tend to focus on one of those two terms.
Ted: Well I don't know that mod—refactoring necessarily has to be part of a modernizing process. You might refactor code even though you're not modernizing.
Charlie: Sure.
Ted: Refactoring really has more to do with being able to work on your code. I mean I hate a program that's a monolithic program that has thousands and thousands of lines. It's like Whac-a-Mole down at Chuck E. Cheese. You fix one thing and something else breaks, so that's not good code. So you might refactor to keep those things from happening.
Charlie: That's true. So let's get a little bit deeper into this, Ted. So we—the other term as I mentioned earlier. I was going to start saying was this thing called code smell, and you hear this term—we have this conversation of refactoring and the term code smell will invariably will come up. What is code smell?
Ted: Okay. Well a code smell—I don't know. I don't use the word, the verb "suck," in an intransitive way. I use the word "stink" instead. I say that code stinks. Other people might say that code sucks, but anyway a code smell is just code that you look at and say, "that's terrible." It raises a lot of red flags. It's code that says "hey, there may be a deeper problem here." One guy who writes about refactoring is a fellow named Kent Beck, and he said he used his grandmother's child rearing philosophy: If it stinks, change it [laughs].
Charlie: Well that sounds like good advice right there.
Ted: But think about it. Let's say you look at a program and you know, you've never seen this program before. You open it up and start at the source and you see goto, goto, goto. Well you're obviously repulsed by that, right? Because we've known for 50 years that goto is bad; we're not supposed to use it. So anyway, goto is a code smell. So another code smell for me is duplicate code that could be duplicated within the same source member. It could be duplicated from one source member to another, but duplicate code is a source—is a code smell. It stinks. Deep nesting. I worked on a program years ago. I remember it had 12 levels of if-then-elses and do-whiles and all that stuff, nested down 12 levels. That's ridiculous. That's a code smell.
Charlie: And certainly very hard to debug and just to analyze and review.
Ted: Sure, of course. Long subroutines or saying if. An if that has six lines and then an end is no big deal. An if that has 600 lines before the end if? That's a code smell. That stinks. We need to refactor that. We need to rewrite that piece of code.
Charlie: Goto, I mean, that's the—with no pun intended—that is the go-to talking point when I hear about people refactoring. They always goto to the gotos. That's one thing that they look at all the time. But you mentioned duplicated code, duplicated sections of code, deep nesting, things like that. What other examples might there be in RPG—or maybe just in any kind of code, in any other language? Is there any typical things that would typically you would go after first? What is the low-hanging fruit here when you're trying to refactor a program other than what we've spoken about so far?
Ted: Okay well the first thing for me is to try to get the code divided off into sub procedures. My programming is really a lot of sub procedures tied together. They pass their parameters back and forth, and I try to have sub procedures within controlling logic. So for me the first thing to look for is a lack of a sub procedures, and that's the thing I concentrate on most. And what works hand in glove with that is trying to get rid of global variables. Goto gets a lot of bad press and deservedly so. I think global variables are just as evil. I hate global data at all, so I'm going to try to get this program built—refactor into sub procedures—and if there's a variable that's only used inside a sub procedure, I'm going to make it local to that. I'm going to try to get all of the global data I can.
Charlie: Again to make it more concise and easy to debug.
Ted: Right, well to keep—exactly. It's just that we want this code to be reliable. We want it to be rock solid and run and do what it's supposed to do without any undesirable side-effects, and global variables are notorious for causing side-effects.
Charlie: You know one of the problems, Ted, that we recognize with code in general is that it doesn't necessarily rust, right? It doesn't—so it's hard—I mean I suppose we can look at it and say it looks like it's old style, but from a C level—from a CEO level perspective, they may not recognize the true value of refactoring, because in their view, potentially—well, it works. It's producing results, so why should I make any effort at all to deliberately refactor my code?
Ted: Okay well I would say if there is no business case, then don't refactor it. You've got an old program and you don't need to enhance that program, it's working properly. So what if it is written in an old version of RPG or so what if it has gotos in it, whatever. As long as that program is working and doing its job and you don't to modify it, that's fine. But what if you do need to enhance it? What if you do need to take this thing and integrate it with, I don't know, web services or something, something that did not even exist when that program was written? Well now you're going—you know the question is can you work on the program or not? Can you enhance it? If you cannot enhance it, then you just don't have a choice. You've got to refactor.
Charlie: Hmm. You know one thing we didn't mention when it comes to refactoring is using embedded SQL. That's a big one. Is that considered refactoring if I'm going to—if I'm embedding SQL?
Ted: Well if you don't—let me put it this way: If you have, say, a loop in an RPG program and it does a read equal and such and it's updating a bunch of rows, records—well you could replace that with an SQL update that does it all in one fell swoop and no loop and all that. Is it refactoring? Well if the behavior doesn't change, then yes, it is refactoring, but let me add to that. I do not see native I/O as a code smell, so I would not say to refactor just because it's not SQL, just because it's native. Native I/O to me is still a perfectly valid way of doing I/O. But to answer your question, it is refactoring as long as the behavior doesn't change.
Charlie: You know one thing I hear is that—and you said it already in one way—is some programs are difficult to modify, and the more difficult it is to modify, the more expensive it will be because it will take longer to add new functions, for example, to a program. So is that a good definition of technical debt at that point?
Ted: Yeah, you're hitting right on technical debt. Refactoring is a way of paying off technical debt. So yeah, if the program can't be enhanced—well put it—I think of it's kind of like, let's say you come into work every day and they give you a shovel and say "get down in that hole and dig it deeper." That's just increasing your technical debt. Refactoring is a way of standing up on the side and shoveling dirt down into the hole and starting to bring it back to what it needs to be. Get rid of that hole.
Charlie: You know technical debt is an interesting topic to me because there are cases—I suppose when you recognize that you do need to do something because you have a gun to your head and you need to deliver something very, very quickly. You might need—I've heard people saying "well, I had to get it in. It's 4:00 on Friday. I need it by Friday at 5:00 so I just put a hard-coded value in today just to get this thing out into production." Is that a fair argument for deliberately adding technical debt? Do you even consider that technical debt adding a hard-coded value?
Ted: Oh certainly, that's technical debt. That's technical debt for sure. Hard coding is another one of those things that we should never do, but we all have to do it sometimes. I understand the business has got to stay in business. It's got to ship product or whatever it does, and sometimes you just have to hold your nose and do something the way it's not done, right? It seems like there's a lot of times we never get back to it to fix it the right way, but there's the old saying from the Bible that says you reap what you sow. And if we do these things, no matter what our intentions are, we're still going to have face up to it eventually. Technical debt has got to be repaid.
Charlie: Well that's the problem, Ted. I hear so many stories of "oh, we put that in temporarily" and then of course—then it just becomes a permanent part of the production code and we never go back because there's always an additional or always a new, urgent distraction, right, and we can't get back to undo what we have. Maybe that's where—this is how we wound up with what we have today in our code bases.
Ted: Well this is slightly changing the subject, but I think it addresses what you're talking about. What I try to do in my programming is I try to have good programming practices, okay? Years ago when I was on System/34 and I have O specs, or System/36 whatever, I learned not to use a blank after my O specs because it was going to cause me problems sooner or later, okay? I had all of these sort routines and such that I found the darn, you know, #G sort or whatever. I learned you clear out your work files beforehand, not after. So in other words I had—and this was 30 something years ago—but the point I want to make is all through my 40 years or so that I've been doing this, I have tried to develop the best practices. And if you can go into writing a new program with best practices, then the chances that you're going to generate technical debt are less and the chances that you're going to have to refactor are less.
Charlie: Hmm, so you've had some success with refactoring. I mean if you're brought into a shop for example, or you're in a shop that you know the code base is simply riddled with technical debt everywhere, how do you approach management to make a case for that? Or do you even approach management? Do you just start doing it on your own?
Ted: Well I refactor on my own as far as I can get away with it. Normally when you're working on a program you can take some time to refactor—and that's another point I'd like to make is these refactor gurus say don't try to enhance and refactor at the same time. Either refactor or enhance, but not both together. So I don't know. There's one shop I did some work for and I told them I'd love to show your programmers how to write COBOL without using gotos, and the IT director wouldn't go with it. Well one of the results of that is that I no longer do any work for that shop. Sometimes you just can't persuade people that they need to change the way that they're doing some things. So yeah, some of them are not going to be interested in refactoring, but it's interesting. That shop that I'm talking about right now has got a customized package that does what they want. They couldn't go to Oracle—they tried, didn't make it—and now the owners are wanting to throw out this iSeries because it is antiquated. The reason is that these people won't change what they're doing. So refactoring is a good way to modernize the systems and bring them up to date. It's a good tool in your toolbox, but I don't know. If people won't do it, you can't make them.
Charlie: So what is your response when you hear somebody say "well you know what? Even the worst, the most poorly written code on these new systems runs so fast that we don't necessarily need to worry about refactoring code, because the systems just crunch through these programs lightning fast anyway."
Ted: Well you might refactor a program for performance reasons. That might be a good reason to refactor, but refactoring is not really about performance. Refactoring is about maintainability. Let's just say that you've got an accounts payable system and you've got some place for an email address. Somebody comes to you and says we need to put in two, three or four email addresses. Can you do that overnight or is this going to be a six-month project? How you've written your code and how you designed it enters into all of that.
Charlie: So we talked about business cases for refactoring. Do you have any other examples about making a genuine business case for refactoring where you know management will get on board? I mean do you have to reduce this discussion or this argument just to pure dollars and cents? Is that a good way to approach to convince, you know, the most staunch hardliners that are out there to put budget towards this?
Ted: Well I've never thought about needing a whole lot of budget for refactoring anyway. To my way of thinking, a programmer ought to just do it as a matter of course. You know when somebody assigns me a project, well I just don't tell them I'm going to refactor on that thing for a day before I get it to the enhancement part. So I've never really had to sell people on it as such. I've just done it as part of my trying to be the best programmer that I can be.
Charlie: Right so you're not making a conscious move and saying "okay, we're going to take every program in this code base right now and just from A to Z start refactoring them," so to speak.
Ted: I've never done that.
Charlie: Yeah.
Ted: No. I just have refactored whenever I found a need to refactor and I haven't made a big deal out of it. But I think a lot of times what we do is we try to, what I call, stab at it. We just try to fix—patch the program and make it work. We really ought to just step back and try and say what is this thing we're trying to do and is there a better way to do that, but again the thing to keep in mind is you're not trying to change the behavior. You're trying to retain the existing behavior.
Charlie: You know one of the topics that comes up all the time in this discussion is diminishing returns, and what I mean by that Ted is at what point—because you will reach this point at some point—it's about balance. At what point are you obsessing on refactoring vs. making a real productive enhancement to a program? You know there has to be some balance when you say "okay, I'm done with refactoring. Now I can move on with the next program." How do you determine that? This is not an easy question to answer, but how do you address that answer?
Ted: Well I realize that different people can do things different ways and both ways are good. In other words everybody's code doesn't have to look like mine. It can look different and yet it can be just as good as mine and maybe even better, but I don't feel like that I have to rewrite other people's code to make it look like my code. So the first thing is get your ego out of the way. You know as long as the code is good code, it may not be the way you'd done it, but leave it alone. Resist the urge to refactor unnecessarily. Really you just want to get it to the point where you will be able to add the enhancement that you couldn't add otherwise, and once you reach that point, stop.
Charlie: If you were working for me for example, and I said to you "hey Ted, I need you to add a new function to this program." You open up the code and you say "holy cow this thing really is in dire need," but the enhancement you need to add is only in one section. Would you just refactor that one section or would you do top to bottom a whole rewrite of the program?
Ted: I would refactor as little as I had to. I would only refactor the one section. I just don't see—let me put it this way: If you're shipping widgets for a dollar apiece—well, you're still going to be shipping widgets for a dollar apiece even if you refactor unnecessarily. So just refactor the minimal amount that you need to. At least that's the way I do it.
Charlie: So that's just good housekeeping for the source code.
Ted: It is. I think it's just like anything else. Just think about your house. Things break, you know something needs repainting whatever around your house. Well you can let it all stack up and then eventually you've got this huge need to do a $100,000 remodel on your house or something, whereas if you had kept it all up as it went along, you wouldn't have such a big project.
Charlie: Technical debt.
Ted: That's right. You got it.
Charlie: That's exactly what it is.
Ted: So refactoring and technical debt really are very much related.
Charlie: Right. With refactoring being the solution to technical debt or at least—
Ted: Yeah, the way to repay technical debt.
Charlie: Right.
Ted: At least a way to repay it.
Charlie: A way, right.
Ted: Maybe not the only way, but it is a way.
Charlie: Makes sense to me. So Ted, we've been talking about refactoring for quite some time now and I think I'm going to start wrapping it up here. But I just want to tell you: I have to share one more hope of mine and that is that we haven't really been together in person—that is at a conference—in a couple of years, as most of us have not been, obviously. But one thing that I do miss and I surely hope that will happen again when we get together again Ted, I will be able to once again partake in one of your very famous moon pies, because I know that's one of your favorite things to give out. Tell me about these moon pies. First of all for those who may be listening and not familiar with a moon pie, what is a moon pie and tell me why you're giving them out, because that's such a great thing. It's almost become like your signature reward to certain people. So tell me something quickly about the moon pies.
Ted: Okay. For those who don't know what a moon pie is, a moon pie is a little snack. It's two wafers with marshmallow cream in between them and covered with some sort of flavoring, chocolate or banana or vanilla or whatever, and they come from Chattanooga. So it's a Southern thing kind of like me. I'm obviously a Southern thing as well but every place you go down here, you can find them for sale; every little store has got them. But anyway several years ago when I was traveling—I guess up in your neck of the woods or up in that area—I brought along some moon pies and as I asked questions and people would answer, people in the audience, I would give them a moon pie as kind of a little reward. Most people up around the New York area I don't think had ever even heard of them, but they're good. I don't eat them all the time. I don't eat them very often. They're not exactly the healthiest thing to eat; they're full of sugar. But every once in awhile I'll get an urge for a moon pie and I'll go buy one.
Charlie: Well I certainly hope that when you're back on the road you continue that tradition, because it's such a wonderful thing that has become your thing actually. So please, please keep it up. It's a wonderful little trademark that you've created here.
Ted: Well Charlie I'll keep that in mind if I ever get back on the lecture circuit or conference circuit. I'll be glad to do that.
Charlie: Perfect. All right well listen Ted, I want to thank you very much for your time today. I can tell you that you've certainly hit all the big points that I can think of for refactoring. It's such an important topic and people really need to address it. It's—a lot of the code that we're running on these systems is indeed in need of some refactoring; I mean of course, right?
Ted: Right.
Charlie: It would be hard for me to think of any one shop that couldn't use some of the tips that you've outlined here in our discussion.
Ted: Yeah, I think everybody probably needs to do some refactoring too, and as I said awhile ago, I think the best thing is not to say "we're going to have a refactoring project." Just make it a habit that when you work on a program, look and see if there's something that needs to be refactored. And the idea is when you leave that program, at least it should be a little better than it was before you started on it, and maybe over time you really bring it up to the standards it needs to be.
Charlie: That sounds like perfect coding advice. Thank you for that, Ted, and thank you—as I said, thank you for joining me altogether. It was really a pleasure. Such a good thing to be able to chat with you again, Ted. I really appreciate your time today, and thank you everybody for listening to our podcast. Please look at the TechChannel website. You'll find lots of other great information out there: podcasts, articles, things like that, so it's worth your while for sure. And Ted, I'm going to give you the final word. Thank you very much for coming and your final words of wisdom to those who might be listening.
Ted: Well my final word is that you don't have to understand refactoring the way I want to understand it. I think there must be more to know in spite of all the reading I've done on it, so if anyone has any good refactoring tips, go to ITJungle.com and contact me through the contacts page there. I'm always looking for more information. I think it's a very important topic and I think that we can all learn how to do it together.
Charlie: Perfect. Thank you, Ted. It was a real pleasure. We'll speak to you again.
Ted: Good enough Charlie, thanks.