New: Subscribe to TechChannel podcasts on Apple.Charlie Guarino: Hi everybody. Welcome to another edition of TechTalk. I'm very happy today to be joining Andrew Clark. Andrew is the DevOps manager at Arcad Software. Andrew has 30 years of development experience on IBM i and he’s an expert on everything Db2, i, and cross platform SQL. Andrew's current responsibilities include both coding and using DevOps on a daily basis. Andrew, thank you so much for joining me. It's so great to have you here today.
Andrew Clark: Thank you Charlie. What a great introduction. I appreciate it.
Charlie: Absolutely. Well, we're all about great introductions here, right?
Andrew: That's right.
Charlie: That's really important. Andrew, I was reading your bio here and what jumps right out at me is DevOps; that's a term that I've been hearing for many, many years, well, over the last several years it has come into the fold and I really think that to a lot of developers out there in the industry right now. It's still a bit of a foreign term
Andrew: Right.
Charlie: So maybe you can just give me a quick definition or a little more insight into what DevOps actually is.
Andrew: Right, right. So DevOps is short for development and operations, a kind of a mix or a hybrid of the two, so for people that are comfortable with the development side of things and people that are comfortable with the operation or the QA, the testing side of things or maybe the deployment side of things. It's really kind of a meshing of both of those ideas and really what the idea is to simplify the process of deploying complicated projects especially with as much security and testing and also verification as possible within the company so before the code is deployed to end users and especially before the code is deployed out into the real world right so if you have customers outside of your company especially the cost is astronomical when there are bugs or errors introduced in your code when it gets out to the real world. Even within your own company within certain departments, it can be a catastrophic, a black eye for a company to actually publish errors in code that has been changed. So, the whole idea is there's a big curve that you've seen. It's called a shift left effect where basically the idea is to get all of the testing and the regression testing, all of that done on the left side at the development and operations side before the end users ever even see the code. That really ensures the best possible code quality. It takes a little bit of work to set up but once it goes, it's a pretty magical process. We have it integrated in our system and it's just kind of magical to make a change and then just see everything kind of build by itself and get deployed. It's a pretty incredible thing once you've got everything working.
Charlie: So we've both been doing this over 30 years, being in the industry, that is, and when I first started the traditional approach was just taking a set of programs and possibly waiting a week or even longer, maybe even a month and then deploy an entire set of things at one time. That is so not what the paradigm is today as you have already discussed so let’s talk about that because that's really an old approach waiting until the end to deploy this huge set of changes, right?
Andrew: Right. So that's kind of the waterfall method. That's a release kind of cycle where we're talking about quarterly or yearly releases in some companies or like you said, weekly releases. The whole idea here in DevOps is every unit that's useable that's ready to be tested can be pushed and changed and ready to be integrated into the environment as soon as it is ready so instead of having releases where there are 100's or thousands of objects, you might have releases where there's just a half dozen objects, right? You may have a file change and all the programs that are related to it or you might have a program with a new parameter and all the programs that call it or use it. So, you know every time you've got a unit of change in your DevOps cycle, then you can push that and make it available. It really does speed up the entire process of getting the end result to the end users so customers with as much security as possible.
Charlie: Which obviously goes right into the actual CICD definition continuous integration, continuous deployment obviously so we're always in the process of rolling out incremental changes.
Andrew: Right, exactly, yes. That's the whole point of DevOps is to get to the incremental changes as quickly as possible to the users.
Charlie: And it's been shown over time that that process overall does improve the overall code quality because—
Andrew: Absolutely, yup, yup, yup.
Charlie: No question in your mind.
Andrew: Yeah, yeah so when Gartner does studies and there's a state of DevOps, document that's out there and it's proven. It's a cost-effective thing as well, right? It actually saves companies money over the long run, an incredible amount of money at times. We do an ROI calculator and a 50-user shop, a 50-developer shop can save a million dollars a year right implementing DevOps correctly just based upon how their existing test cases have been taken care of in the past and what DevOps is able to do so it's a really proven cost savings. I think DORA is the name of the name of the company that does that study, and you can go out there and they do it on a yearly basis, pretty staggering amount of money that can be saved.
Charlie: You know I think one of the challenges in the IBM i space is that many shops have—they're single developer shops or they have two developers in their shop—
Andrew: Right.
Charlie: And I guess the notion of having a full-blown system to do DevOps may be foreign to them because maybe that's for much bigger shops. Maybe we don't need that technology in our shops, but I disagree with that. I think even a shop of one. Why don't you give me some thoughts on that?
Andrew: Yeah, absolutely right? The whole idea about DevOps is that a lot of the testing can be automated within the development cycle so if you have the test cases set up, you can do that testing and integrate it very quickly. You don't have to worry about these huge deployments like the bigger shops do but you can take advantage of a lot the things in DevOps especially things like Git, right? That's not a requirement but that certainly makes your life a lot easier and the automated testing is important no matter what the company size is. Writing bugs is still writing bugs, no matter how big the company is and you want to make sure you take care of that before the end users see it and it makes you look better. I mean let's be honest right? You're a one or two developer shop and you're the target if something goes wrong so if you've got some DevOps in place and all of a sudden you know you don't have to worry about those things. That makes your life better.
Charlie: Well, you said one of the magic words that's clearly in the industry right now is Git. Anybody who's coming out of school today knows about Git because it's being used everywhere, but again that's—I got back to the IBM i shops. It's only starting now to make its way into our community in a bigger way. Let's talk about Git because I know what it can do but I'd like to hear your thoughts on what is Git first of all but more importantly why is it important to use as developers and how can we benefit from implementing a Git solution?
Andrew: So the big benefit of Git in the open-source world is it allows distributed version control so essentially each developer can have their own copy of their objects on their own local PC. In the IBM i world, it's different because you still need the IBM i to actually do the compile so that whole distributed model is is kind of a foreign concept or it's a different concept than most Git developers are used too but Git, it just very simply, I mean it's source control. You can do version control with Git and you can do deployment with Git but that's not what it is. It's just about managing your source and the greatest thing about Git, the lifesaver is basically dealing with concurrent development. One of the biggest problems anywhere is when people are making change and they stop on each other's toes. You don't know about it. Someone might make a change and accidently undo something that someone else is doing when we're talking about, just copying out single source numbers. In the traditional IBM i world, you'd do something like you'd copy a tool at the library and then when you're done, you'd copy your source back into production. Well, what happens if someone else makes a change while you're doing that, right? I mean that can be catastrophic sometimes, so the biggest power of Git is really its ability to deal with concurrent changes and it doesn't allow you to make concurrent changes. There's no way that you can accidently step on somebody else which is the huge power of it and also it can do merging automatically, so I use Git on a daily basis and 90-95% of the time the merge just happens automatically. It's really pretty wonderful, right? You see some changes. It gets merged in and then you're done so for the typical developer you know the whole edit/compile/debug cycle, it's exactly the same using Git but you have all this protection from concurrent development that you never had before, and it really is a wonderful tool no matter what development environment you're working in.
Charlie: And of course, you can't forget the part with the Git diff to compare changes.
Andrew: Yes, of course. Right, right. The whole compare and merge are really pretty magical right, you know light years beyond compare PFM for sure.
Charlie: Yeah. With all the i's and d's on the—
Andrew: Yeah, the i's and d’s yeah, I know. That's not great.
Charlie: Well but that's what many shops are still using and of course let's go one step further with Git. So we talked about version control but now we're in an environment where we made some changes to source members and now we're ready to promote them.
Andrew: Right.
Charlie: Now promote can take on several forms. It can go directly to production but certainly there might be a test phase and obviously—
Andrew: Right.
Charlie: There should be a test phase in between this. Let's talk about that and let's talk about the typical deployment cycle.
Andrew: Right so DevOps really assumes that if you're going to do a full DevOps cycle, what that really means is the full DevOps cycle includes the same edit/compile/debug step that the developer is always comfortable with. Then there's also the automated test side, so what will happen is there will be test cases that are created that can be run automatically and in batch. They can be run headless. They don't necessarily need to have a screen associated with them which is kind of foreign to a lot of people. Now there's tooling that allows you to do that but I mean the "best"—I'm doing the air quotes—the "best" way to do those things is to have your program set up as like source programs and modules and they can be called from test cases and they verify the results so the developer will run his own smoke test locally but then he might also run all of the test cases associated with the objects that he's changing. Now he's not going to do a full regression test. He's just going to do kind of a small subset of those things, you know a unit test on the code that he's working on. Then in the DevOps cycle, he pushes that change to some kind of an automated build system and then what that will do is it will run a full regression test; every time there's a change, it will run a full regression test on the objects that are changed before it actually does something like deploy to the next step whether that's QA or production or however it's configured. The other thing is we talk about not only regression tests but we also talk about code quality, things like SonarLint, so you might have rules in there that ensure that your code meets a certain kind of quality metric and we can talk about that for an entire hour or things like SonarQube that have bugs, that have security holes in your code possible security exposures or poor programming styles that could lead to bugs and so your code has to pass all those quality checks before it can go onto the next cycle right before it can go and be integrated so all of that happens automatically and if it fails any of those things, basically the code is returned to you with a failure and you have to make the changes before it can go into the next step.
Charlie: So that's very powerful, everything you've just described to me but you know what? Even with all of those testings and all of those safety checks in place, certainly code does get moved to production that will have errors so how does Git help us in those instances?
Andrew: Well at that point really the DevOps cycle, it tries to stop those errors but if an error goes out into the real world, the cycle is essentially the same so you can do something with Git with hot fixes that are nice. We can make branches. That's a way to make a quick change. Even though you're using Git, the methodology is very similar to what you might do today. You copy it to some hot fix kind of a location. You make the change and then you skip some of those steps in between and you do an emergency promotion thing so there are ways using Git that are very similar to something you might be doing today but really in that kind of a case; you're making what I would consider a hot fix. There's no wait for the change that you make, and you redeploy and you go through those steps but even with a hot fix, you're going to run that full suite of test cases again just to make sure that everything is kosher before that change gets pushed into production.
Charlie: You know when I first started looking at Git awhile back, the first thing that I was introduced to was not a graphical tool, but it was a command line using Git—
Andrew: Right.
Charlie: I think that's how it was first designed or created with a command line and you hear some developers who insist on using command—what's called a command line Git VS a graphical tool. Why would you usher people to a graphical tool VS a command line?
Andrew: Well, Git is command line driven. Git is a what was written by Linus Torvalds and it's a C program right so it is a command line so all the shells are essentially working with a Git command line so they're just shells on top of what the Git command line does but Git command line allows you to do everything and it is the most powerful way to work with Git just like on 5250 at the command line, you can do a lot of things at command line that you can't necessarily do through work spool file or you know any of the 5250 displays or work member PDM. There are amazing commands that you can do if you really know what you're doing on the command line that don't have any kind of an interface. There's no shell to them, no screens or menus or anything that allow you to do the same amount of power so the shells, things like EGit which is a plug in to Eclipse for the—for Git does make it much easier. It's a quicker learning curve to Git and honestly I use it everyday and 90% of the time, 95% of the time, actually probably 99% of the time I do everything within EGit but there are cases, there are things like some tags and things that are difficult to do in the EGit interface and if you really know what you're doing with a Git command line, then you can understand how the graphical interface is working and plus do some additional things that are actually more difficult to do inside of those shells.
Charlie: So, the way I learned Git was command line and I found them on my own journey that knowing some of the commands helped me more quickly adapt to the graphical tools—
Andrew: Of course.
Charlie: And as you said under the covers the commands are being run anyway.
Andrew: Right. The commands are being run anyway and it is important to understand what the concepts are. I would say that the Git command line can look intimidating but it's good to know the basics and if you want to use that, I just find that it's faster to use the graphical shell because I'm within Eclipse doing my development anyway but certainly switching out the command line, you need to be comfortable. There are cases where you need to use that and it's important just like anything to understand kind of what's happening behind the scenes to really get the full power of Git. There's no question about it.
Charlie: We started this conversation Andrew with the term DevOps and that's a term that people use everyday and that's a term that's in the vernacular of the community but the newer term I suppose is DevSecOps—
Andrew: Right.
Charlie: And where did that—you know what's the origin of that? Why did we add Sec or security into the mix over here?
Andrew: Yeah so I talked about that a little bit. That's the security component of the automated deployment so after the developer has pushed his change to the repository, then automated test cases get run, so that's all classically DevOps but then also there's automated security checking as well and those are things like SonarQube and Sonar—SonarLint and Sonatype so those verify that your code doesn't have any inherent like SonarQube is making sure that you're not doing anything in your code that's inherently dangerous, things that might create you know null pointer errors, those kinds of things right you know and then Sonatype actually verifies third party exposures right so the big thing in the open-source world is using all of these third party packages. There are dozens if not hundreds of them that are included with your development stack and the problem is if there's a security exposure in one of those third-party tools and you ship your product with those third party tools, you're still liable for the exposure; things like the whole Target data breech was because they used a third party tool that had an exposure in it and they didn't fix it. It was a known fixable thing and they continued to have production code that was using this exposure so those are the things that are caught automatically in DevSecOps. I kind of just include that in DevOps to group it together. Classically, it wasn't but it's just all part of the automated cycle the part that comes after deployment and before integration or right after the developer has pushed his change, the automated testing and security checks happen and then deployment right so it's the in-between step and that's important.
Charlie: You mentioned a lot of different tools here. You mentioned Sonatype and SonarQube but if I'm a smaller shop, do I need to implement that all these different things all at once or how do I begin my journey if I really want to go down this path as I think everybody should certainly but—
Andrew: Right.
Charlie: How do I get started? Do I need to get all these expert tools or what's your recommendation for somebody?
Andrew: No, no. What you can if you're a smaller shop the way to get your feet wet is just to start working with Git inside of whatever tooling that you're comfortable with and the easiest way in my mind is with RDI. Hopefully you're using RDI for your development at least for your editor for RPG especially and it's got some really easy integration with this perspective called iProjects that allows you to continue to work with your source string files right in your QSIS source files so you don't have to actually change anything about how your system is configured. That allows you to kind of get your feet wet, get going with Git and start the DevOps cycle and actually start doing the concurrent development that prepares and merges those things that Git allows you to do. Then you can start thinking about once that's integrated and you are using Git, then it's easy to hook up other tools like Jenkins which is kind of automated build system and deploy some other tools that these things like SonarQube that allow you to check your code. So in the IBM i world, honestly SonarQube and Sonatype don't have as much applicability as they do in the open-source world but there are products from companies like Arcad that do integrate directly with Jenkins that allow you to do those same things on IBM i for sure.
Charlie: Right so but there is some planning. You need to do some planning; you don't just jump into this. You need to have a plan ahead of time—
Andrew: Yup.
Charlie: Somebody who understands this process and can guide you through the implementation of this.
Andrew: Yup, yup, absolutely right and so whatever experience you have in-house is always the best experience to use so if you have people that are comfortable with tools like VSCode or if you have people that are comfortable with tools like GitLab instead of Git Hub or Bit Bucket so any of these Git based repositories, most of them are Cloud based but if you have exposure to that, then that's always the best place to start is someone that really knows this information but if you don't have any of that experience, like I said and like you said, the easiest way is just to get your feet wet at things like COMMON and use podcasts and just kind of get some exposure to the Git command line, starting RDI, see how the integration works there and that really starts your journey. That will really open your eyes I think to the possibilities of what's out there.
Charlie: And you know it's really so important in my view and I think you would agree; it's so important to properly manage source code. It really is a very strategic asset of any enterprise.
Andrew: It is.
Charlie: Millions or billions of dollars of real money go through your applications on a monthly, yearly basis whatever the case is so it's really very important to manage it as a real strategic asset of your company.
Andrew: Absolutely. It's invaluable; I mean, if you're a software company, it's your entire asset and if you're not a software company, it’s still a lot of time, it's the advantage that you have over other companies in your space is your source code and so to create source code with or to create objects with quality less than they should be is a disadvantage in your business. That costs you money and the whole driving force behind DevOps is return on investment. Like I said, there's calculators out there that show millions of dollars can be saved just by implementing the DevOps cycle correctly.
Charlie: And as you said earlier which I just want to reiterate because it really struck home at me is that obviously the earlier in the process that you identify it is such—
Andrew: Right.
Charlie: An issue, you know the better off everybody will be and—
Andrew: Right
Charlie: The less costly it will certainly be.
Andrew: Right. If you look at the cost, that curve right, they give you a little curve and basically if the developer can fix the bug, it's like a 1x. It's whatever the cost is x1 and then if it gets to the next cycle, if it gets to the QA cycle you know to the testing cycle and it has to be fixed, it's a 4x kind of cost. Then if it gets to the deployment step and there's a bug, it's a 10X kind of cost. Then if finally, if you get up to the real world, it's 640x cost so whatever that cost would have been to fix it in-house, 640x that is your cost not only in real world time and hours spent but also loss of reputation. When you ship things out that aren't correct, then that hurts your business because people don't trust you know the software that you're developing anymore so it's a massive cost. It has been studied, as I said.
Charlie: That's so hard to gain back everybody's confidence and trust—
Andrew: Yeah, absolutely.
Charlie: Absolutely.
Andrew: Absolutely.
Charlie: Wow. Andrew, this is a lot of food for thought for here and I really appreciate you coming today and chatting with me on this whole DevOps cycle and DevSecOps. It's a fascinating topic. It really is and it's really another important step in if you're running a shop, you need to look at this and this has to be an important component of the overall mix of things that you do in your application life right and in the SDLC I suppose, right?
Andrew: Yes, absolutely. Yeah, no. I appreciate it and this is a huge topic. We're very briefly covering some very complicated subjects here in a very short amount of time but yeah, it's something that you could go on for hours and hours about on each one of these, Git could be time and security could be time and automated deployment. All of those things are just a continuous learning process.
Charlie: Well I certainly appreciate your time. Obviously, you're well positioned. When your bio says DevOps Product Manager at Arcad, all of your experience clearly came through. It really did and I think a lot of people are going to really benefit from the experience that you've brought to this discussion and the examples that you've given us so thank you so much.
Andrew: Yeah, thank you. I hope so. I hope everyone you know get some benefit from it. I appreciate it.
Charlie: Great. So, thank you everybody for listening today. This is Charlie Guarino as I said. I will be back with another podcast next month. In the meantime, make sure to check out the other content on TechChannel and by the way, you can also subscribe to their weekly newsletters, webinars, ebooks, solutions directory, and a whole plethora of other things. It's really worth your time. Thanks very much everybody for joining me. Until next time, see you. Bye bye.