Application Testing: You Need It When You Need It
Polverini & Partners CTO Pascal Polverini on the critical role of testing in IBM i application development
This transcript is edited for clarity.
Charlie Guarino: Hi everybody. This is Charlie Guarino. Welcome to another edition of TechTalk SMB. Today I am very happy to be sitting with Pascal Polverini. Pascal is the founder and CTO of Polverini & Partners, an IBM i company based in Italy. Polverini & Partners focuses on IBM i modernization, digital transformation including AI integration, software testing, Code Coverage, anything that will help you improve your applications. Pascal, it is so great to see you and chat with you today. Thank you for joining me here today.
Pascal Polverini: For me too, Charlie. Great to see you and looking forward to meet you soon in Milan.
Charlie: Yes, we’ll be together in COMMON Europe. If anybody is interested in taking a visit there, it’s well worth your time to go. If you’re anywhere in the European area, I strongly suggest you consider visiting—it’s going to be a great conference. So thank you, I look forward to it as well. Pascal, the reason why we’re here today is to talk about a very important component of IBM i application development, and it’s testing. That may seem to a lot of people to be something that we all know about, but I still, in all my years and probably all my decades now of application development, I always feel that this does not get the attention—in some cases the attention that it needs or—well let’s just say the attention it deserves or the focus that it needs. There is so much to testing and so I want to talk about testing, application testing and maybe some facets of it. But before we get into just testing in general, what has been your experience? What have you seen in the industry, maybe from starting from 20 or even 30 years ago, a historical perspective in testing practices and how might they have changed through the years?
Pascal: Look, we all know that testing is important and we know why we always miss time to do proper testing, because deadline could be made for any project. And then of course you know that you develop something, then you test it and then you are ready to put your product in production, but somehow from mysterious cosmic reason the time to test always shrinks—always—and we always miss the time to do testing. Ideally, we would use tooling, automation, whatever can help reduce this time to do what we need in testing.
Charlie: You know it’s an interesting point you’re making, but speaking as myself or somebody who has developed code for many, many years, I recognize the fact also from a customer perspective if they have, as we say, the proverbial gun to your head, something has to get put in today at 5 o’clock or 3 o’clock, we have this sharp deadline, and this is where I find that testing often gets compromised—
Pascal: Yup.
Charlie: Because we just don’t have enough time and we put it in, as we say, on a wing and a prayer, we hope it goes live. But what are the dangers of that?
Pascal: The danger is huge. If you’ve got that bug, the later you find the bug, the more it will cost you. I don’t know if you’re aware, one example is Amazon. Amazon is huge, right? Surely they don’t miss capabilities, but it happened a few years ago in the UK. For two hours, the website of Amazon, every product was available for one cent. One cent. For two hours many, many people just order anything. This was a big issue. It cost a lot, many providers talk back to you know the customers to explain that this was a bug and then to unbug the order, but in some situations, this was just not possible. That was a big loss because of a bug that was found later and already in production. This is why if we could test everything beforehand, obviously everybody will sleep better, but this is actually a challenge.
Charlie: That’s an interesting point you make, but I’m going to turn the tables on you now for a moment, because the typical configuration—or maybe not typical but a common configuration—of developers in IBM i shops is they may just be a very small team. That’s not uncommon. Certainly, IBM i can run huge corporations and there are companies who have hundreds or maybe thousands perhaps of developers, but then the other side of that are companies who have only one developer, or just two or three for example. How do you explain the idea of software testing to them where the responsibility is just as big I think, but there’s only one or two people on this testing side? How do you make your point that testing is so important when there’s potentially a much more casual way of things going into production?
Pascal: They are different things here because the best would be that the person who is developing should not be the same person that is testing, because you may not see what is wrong within your code and if you don’t see it, you will not be able to test it properly. If you got another perspective, it will be more secure and you will cover more cases for sure, right? And also you will be impartial, which is another aspect. This is one point. Another point is really at the time it depends on what you need to test, and here we can talk a lot. Any application as an average you can have a UI, a database, and coding. You can have altogether, then this is more end to end testing or pieces. If we talk about pieces, you can split your test with unit test and we know this but this is not enough. What you really need before a release before going to production is to do end to end testing, and this could require a lot of effort—not just time, but also effort. Let’s take the example of where within your application you surely have tables that are used, and your test should also incorporate those tables. If you want to really do a proper testing, you will need first to isolate all of those tables you are about to use, to copy them before, to copy them after you run your test, and eventually to reinitialize all of them any time you replay your test. And this is not it—you will have to at the end to compare then with the initial copy. Then just this is discouraging the old test because this is a lot of work, and also within this work there are many, many occasions to do mistakes as well, right? Then it is not an easy task, but we all know that it’s like insurance. You don’t need it until you need it, right? This is the story with testing. Then the best would be to find some automation if you want to go for a proper and global testing, because any time you do some change in your code—if you could be able to rerun, let’s say 100 recorded test scenarios, you will have more security about oh, okay I can rerun these 100 scenarios and check if I don’t break anything with my last change, which is a point. Computing is about this. The goal of computing is always about automating things and testing is a perfect candidate that should be automated, but to automate it, of course if you have no means, no tool, you do it manually and you need a lot of resources, but this is not truly automated. What you will want is a robot or something similar, a tool according to what you want, but this is one of the purpose of computing is to automate things that can be automated.
Charlie: One of the things that comes mind, I believe, when people say have somebody test your code, is they’re testing an interactive program. So they’re sitting there on a keyboard and they’re testing input and output of things and how data is being manipulated in the database, but not all programs are interactive programs. Certainly not. I mean maybe they’re web-based, but a high percentage of those are also batch programs.
Pascal: Yup.
Charlie: So how does a user participate in the testing process of that?
Pascal: There is the user experience and the testing engine. Ideally what you will want is to have a testing engine that can work anywhere—batch, interactive, web environment—because you want to centralize what’s going on with the test and to have plugin or methodology when you’re got a green screen. The same when you’ve got a web application and to have some automation—again, that can simplify the human task. We know that for batch, it is pretty simple because you mainly have to encompass only the database. The command can be run pretty easily, but you just have to put your effort for the practice of testing within the database. For green screen, you need something else to automate also the sequence, the navigation through your screens, and there are different technology, different web label to do that, to automate this aspect. But the point is to centralize everything with the same global engine to encapsulate everything within your app—from web UI, from a green UI, or without UI with outside access with SQL for instance—and if you can have the same engine to encapsulate all of this for your testing, of course this will be best. Another point which is a bit harder for web testing is that you’ve got asynchronous jobs because what you want to test is of course your web page, your UI, but also the back-end code plus the database. And because this is asynchronous it is a little bit more difficult, but here too it takes some global technology that can encapsulate all of this to enable the test for you. And then the user experience: The more you can find a solution or a methodology that can split what can be done by an end user vs. a programmer, the better, because the end user knows better the application than the programmer. If you can find a way where okay, here is the end user. Just click here, here, and here and do what you do every day. We will record your testing automatically. This would be the best. Then this could generate script or whatever that can then be passed to the programmer that can deploy it to replay the test again and again. But at least the end user will be able to generate a lot of test scenarios and here you leverage on your team between end users and programmers.
Charlie: You mentioned automated testing a couple of times already in our little discussion here and you did mention the number was—I just wrote it down—you said about 100 different tests, or 100 different tries to test the results.
Pascal: Yes. Yup.
Charlie: So that’s an interesting question, and the question is is there any formula that I can apply globally that might reflect how many different scenarios I should run? For example, if I make just one line of code change, should I try 100 different variations? Because certainly one line that is in the wrong place in code can have very dramatic effects and very drastic effects.
Pascal: Yes, yes, yes.
Charlie: The question is is there any formula that I can apply—and maybe there isn’t—but is there any kind of formula that I can apply to say okay, well this change is very isolated, so we only need to have 50 attempts. Or maybe we need to have 1000 different scenarios, I don’t know.
Pascal: Well in this case or to understand how many scenarios you will need, you have no choice but to use the Code Coverage, the API available from IBM. The only downside is you need RDi [Rational Developer for IBM i], which the majority of the community uses, but you need an interactive action to see the result of the Code Coverage through RDi, okay? It’s pretty easy to launch the Code Coverage—it’s like debugging your program—but it is pretty hard to leverage the result because what you will want always in an automated fashion is to be able to use the Code Coverage also automatically without interaction, without human task within a global DevOps pipeline, within a global cycle. Our site works a lot with Code Coverage and there are different ways to rewrite interpreter about this IBM API to take advantage of the result in different formats. Ideally what you will want is the result of the Code Coverage in a simple database. Then you can leverage SQL to do the merging of Code Coverage or to incorporate the Code Coverage result at industrial level for your testing and then yes, we talk about real automation.
Charlie: Code Coverage is a whole other discussion. Certainly you’re referring to the CODECOV command that’s a native command in IBM i. It gives us the ability to track lines of code that have been executed, which is wonderful. And it’s probably—in my view, sadly—one of the least-used commands that developers tend to take advantage of. But you’re absolutely right, the value that it can provide is enormous. And that opens up an entire other set of questions, a whole other discussion about Code Coverage specifically, and one question that comes to my mind immediately—people who do use it ask me all the time—is it’s a tool, it’s a guide for us. But if we are implementing something into production and we are using Code Coverage, what is a good number? Because one of the real metrics that Code Coverage provides us is the percentage of lines of code that have been executed.
Pascal: Yes. Yes.
Charlie: Do I need to have 100% of all my code covered before I can promote it to production? That might be a little unrealistic, because a lot of those routines are forever handling and things like that. So how do you use this tool in a more valuable way to get the best benefit from it?
Pascal: What we do is of course we look at the first Code Coverage after a few scenarios using the same programs and modules, and if we eat only 30% or 40%, we definitely understand yup, we need more scenarios with different inputs to navigate in different area of the code, okay? When you start to reach 50-60% it can be good but maybe not, because then we have to see two things. One is maybe I’ve got some dead code, some routine procedures that are never called within my code. Then of course it will never be covered, right? Then you can identify that. We also a need internally to understand that automatically, because also sometimes you can also have in your code exceptional routines, like it could run but this IF may happen once every 10 years, who knows? You will know then you can identify some code that are likely not to be used, or very rarely. Therefore this will help you to understand if 80% is good enough or not. But globally, from our experience if you reach 80%-plus you are pretty good to go to promotion. You get a safer confidence about the quality of your testing, because then you will merge different scenarios and cover the majority of cases, right?
Charlie: That’s a fair point—and that goes back originally when we talked about why a programmer is probably not the best person to do testing, because they’re only going to use this program and test it with a known set of data that they have, and they’re not going to try different scenarios or things like that. Or they’re not going—if it’s an interactive program—go through other screens or things like that. That’s been my experience. Programmers only test with the section of code that they know. You know what I’m saying?
Pascal: Yes, yes and another thing Charlie, something we also use with our framework of things we put together. We have the Code Coverage, which we leverage a lot. Again, we rewrote the old interpreter, then we use it in a very easy way. Sometimes what we do is we leverage VS Code to see the result of the Code Voverage through a source code. We use the Code Coverage indicator that we generate on the IFS and with this we can use VS Code to compare several versions of program you run through your test scenarios and then usually it becomes very easy to understand what is missing or not to do the next test or to simply debug or fix it easier, right? We leverage different technologies with the Code Coverage, and this accelerates a lot the understanding the quality of the test or the debugging of an issue.
Charlie: Yeah, you mention that word debug, because that to me is once code is already in production—it’s not testing in the traditional sense. So Code Coverage can indeed be used as a debugging tool.
Pascal: Yes, yes, yes definitely, big time. Of course, yes, yes.
Charlie: Helping you identify different routines or even dead code, certainly.
Pascal: Yes, yes, yes. With this, one thing you’ve got that you understand a lot about legacy code usually, but also the impact of any change, because maybe what you change could change a flag that will enable to go in a different area of your code. And with Code Coverage, you will immediately see that.
Charlie: So how do you make the case if I’m in a small shop? Let’s go through the different scenarios. If I’m in a small shop, how do we make the case to educate management, for example, on the importance of this? Because you know the further you are away from IT necessarily, you may not understand or appreciate the true value of the amount of effort that should go into testing, I think.
Pascal: Yes.
Charlie: How do we as developers educate management to make them allow us to allocate more time to testing before we move things to production? How do we educate them? What’s a fair way to make that case?
Pascal: From an IT manager perspective, the question is simple. Anytime there is a new release—you know the date, the date could be shifted for a few days—but at some point there is a new release. But at that point the IT manager, his only concern is are we sure there is no bug in this new release? When this is going to production, nothing will be broken. This is an important issue and the more you test beforehand, the more you get confidence. This just has no price. If I can assure you every time you change something—because of course you have to change your software. This is a business requirement every month, every week, you have to add, change things—and the main issue is like if nothing else is broken with your code, and if I can give you the means to give you this confidence, this is very important. Everybody will sleep better but also your customer, right? This is the main aspect. Another aspect is for the programmer, for the techie side, just think about how much time you spent today in testing. Even if you do it manually, as an average people spend almost a quarter of their time—sometimes more, it depends what you are doing—but as an average of their time in programming, developing and maintaining your software. If you can have a tool that can do all of this automatically and you just monitor scenarios, you will save quality time to do development, not anymore testing, because this could be again automated. It is as well important. You just have to do some math and you may understand that okay, if my programmers spend two days per month, one week per month to do testing, maybe I should look at possibilities to automate all of this and this will save a lot of time and increase the quality, which is the point. I mean if you are using software, it’s because you want this software to produce something without bugs, right?
Charlie: Absolutely. Right, so the automated testing tools allow developers to make better use of their time, and quite frankly, we all know these are very complex systems. If we were living in a world where we were only doing little DOS programs—
Pascal: Yes, yes of course.
Charlie: But the world of IT is not like that anymore. We have so many different things affecting our applications. We have external, we have APIs. We have everything.
Pascal: Exactly—and internally, we work a lot about cross technology, which is the point. Because the more you go, the more you’ve got different technology, different access. This is why if you want to automate things, you have to have a global vision about okay, I’ve got a web app. I may have also some green-screen app. I’ve got an external SQL app. I need something—a global testing engine for everything to centralize the whole task.
Charlie: Let’s broaden this conversation a little bit. We focused on testing, but it goes beyond that. You know when I introduced you before, I said that we talked about IBM i modernization and digital transformation. One of the biggest things I talk about all the time is modularization.
Pascal: Yes.
Charlie: That’s such an important thing. And the more modules you have—or should I say the more smaller modules you have—each one is easier to test. And that also ultimately is going to improve the lifecycle and the extendibility and extensibility of your code and also reduce, naturally as a byproduct of that, the testing cycle.
Pascal: Yeah, the modularization. Here we can still talk about testing, we can also talk about unique testing, but there are different levels of testing but at the end to have our end-to-end testing, what you really need is a system where okay, I start to invest on testing today. I do something and tomorrow I’ve got something more, and I accumulate different recorded test scenarios. The more I go, the more this will be exponential and the more the time that we spend to do testing will be less and the quality will increase. The point, the purpose is to set up a system where you’re testing effort will decrease exponentially. To do that you need to automate everything: per module, per piece of application, per entire transaction within your application. You can cross different things but the more you create test scenarios, the more gain you will have at the end.
Charlie: I’m going to switch gears on you again, because technology always evolves. We know this of course, and the big thing on everybody’s mind today is AI—of course.
Pascal: Yes, yes.
Charlie: Artificial intelligence, and surely AI has had some influence in the realm of testing and Code Coverage and all those things—test driven development, things like that. What have you seen out there, some real-world examples of how AI is helping in this process?
Pascal: Where AI is helping—for instance to come back to the Code Coverage subject. Let’s go to a different scenario. You want to understand—I’ve got this program, I only cover 70%. I would like to cover 90% at least, because I know there isn’t a lot of dead code. Then it is all a matter of my input data, which should be different to enable the navigation through the code. That’s where AI could help, because also with what we do, we log everything and we’ve got a lot of statistics about what is going on through the testing. Then we can give to AI here all of my test about this program from different angles, but I only cover this 70%. I want to cover more. AI can help you to generate synthetic data that will be different to enable you to navigate in a different section of your code. We already play with this. It is not straight-forward, but AI could on some occasions help a lot to understand what could be different to enter the code or to do a different test scenario.
Charlie: You know that’s not a small point that you just made. To be able to create, as you said, synthetic data for different testing scenarios, that is a huge benefit in my view.
Pascal: Yes, yes, yes. It is. It is huge, but here I play with AI in many, many other situations, and the more you can help AI with automation, the better. Because again, AI has got this random area which enables AI to actually be AI, because it cannot be deterministic always, but this is why you need to think about a full automated cycle. Because you try once, then you can retry, you can retry many, many times and the more you can retry, the more you will leverage AI and vice versa, because AI could help with statistics as well as understanding when it is going well or bad. Then you have to think about global automation again, and testing is really about this as well. But we already have some nice experimentation with this and it could generate some synthetic data when we’ve got a good density of information.
Charlie: You know I’m very excited about AI. I think as wonderful and as powerful as it is even today, we are really just in the first few steps of this, I think. And in the last two years certainly it’s on everybody’s mind now, even though this is not a new topic. But now it’s become so democratic. Everybody has access to it now at a reasonable cost, which has not always been the case.
Pascal: Yes.
Charlie: So I think we are going to see some significant changes in application development in every facet, testing too as we’ve talked about. So I’m excited to see where this is going to go. It’s an exciting time I think to be in application development.
Pascal: Yes, yes indeed, it is true. But again to leverage AI, for me it’s a matter of having the right data, having a volume of data, but also the quality of data. Then to have some automation pre-made to enable AI to go better.
Charlie: That’s true. I don’t want AI to generate 5,000 rows of synthetic data with the same conditions every time. That will do nothing for me, obviously [laughs]. But that’s the benefit of AI. It can come up with different scenarios, more than I can possibly think of or want to think of.
Pascal: Of course, yes, yes, yes. But, and again if you think about it, if you can set up a system where you integrate AI but you’ve got a cycle, then eventually you will get what you want because what you don’t want is to manually interact with here is my code, here are my statistics, here is AI. No, you want to again automate all of this. The more you can do that, the more you have quality result.
Charlie: Well once again, AI is just another tool in our tool box or in our tool set.
Pascal: Yes, exactly.
Charlie: It’s another tool, an important tool but it’s a good tool, and that’s how we have to understand it and manage it, I think.
Pascal: Exactly, yes.
Charlie: So we’re approaching the end of our conversation already, but I just want to hear some final thoughts. What are your thoughts on somebody who is just starting their career in software development? What words of wisdom based on your experience can you share with us to emphasize the importance of testing as an important part of the software development lifecycle, SDLC? How can you impart some of your knowledge based on some of your war stories as we say, things that you’ve lived through to really stress the importance of this to somebody who is just new to the game?
Pascal: The main concept would be to—again if you think about it computing, it’s about automating a process. I will take a simple example. Remember the very first time you learn coding computing, and then you understand the loop, do why, or do until. And it may be easy or it may be hard, but if you don’t use it, maybe you’ve got an example with, I don’t know, a number from 1 to 5. Without the loop, you may write your code and repeat the same block five times. Here you will not leverage the capability of the computing, which is use a loop. It can be five times or 5000. This will always work. This is now automated. This thinking of automation, again, this is what is reinforced by the testing, and testing at the end for me is automating or doing computing over computing and whatever could be computed is the purpose of the whole subject. If you don’t think about it, you may spend a lot of time in ugly task, because no one likes to repeat the same task over and over. This is not interesting. What is interesting in computing and coding is to search for a solution because you have to solve a problem and to find the right design, the right algorithm to do that. What you want to avoid is to repeat the same tasks because it is not yet automated, not computed. Then testing is something that needs to be automated big time and the sooner you understand that the better. Because if you can make your code test-driven, then bingo. You do something, then you already have what is serving to test it, then you don’t spend anymore time on it and your life will be happier because what you will code will be far more interesting because all the boring part will be automated.
Charlie: Not to mention if there is indeed a bug in production and I have spent—I’m sure you have as well—many weekends trying to recover data and fix issues. So you talk about enjoying life, the less time I have to spend weekends and all through the night fixing—you know, repairing data—the better for everybody. And that’s a direct reflection of testing.
Pascal: Exactly, exactly. It will help everyone at every level, right?
Charlie: Right.
Pascal: And everyone will sleep better.
Charlie: Everybody will sleep better, exactly. That’s really good. I think that’s a great place to leave this. This has been a great conversation with you. I so enjoy chatting with you, Pascal. It’s always a delight to speak with you and see you, and I’m so happy to be seeing you again later this year. I want to thank you so much for joining me here today.
Pascal: Thanks to you, this was great. Thanks a lot.
Charlie: I really enjoyed this, and I think people who listen to this podcast are going to get great value listening to all the insights that you’ve shared with us today. So thank you for that, too.
Pascal: You’re very welcome. This was a real pleasure.
Charlie: Great, thank you. All right everybody, I’m going to wrap this up. Thank you again to Pascal Polverini, a real pleasure, and keep in mind that you should check out of course the TechChannel website. There are still other podcasts on there and lots of other interesting articles and the like for you to research. So until next time, we’ll see you then. Bye now, everybody.