The Big Secret About APIs
This transcript is edited for clarity.
Charlie Guarino: Hi everybody. This is Charlie Guarino. Welcome to another edition of TechTalk SMB. I’m so happy today to announce that I have as a guest one of the senior developers on our own team at Central Park Data Systems, Mr. Rainier Sajous. I have had the very good fortune to work with Rainier for many, many years. He really is a senior developer. He does some of our toughest assignments. I’m so glad you’re joining us here today. Rainier, thank you for joining us.
Rainier Sajous: Thank you, Charlie. It’s a great honor for me to be here, and let the good times roll.
Charlie: Let the good times roll, great. Okay so the reason why I did ask you to join me today Ranier is because I know we do a lot of work with our clients using APIs. We talk about modernization and the term API economy comes up all the time. We really are in an API economy in that it seems that this is what the world is using today to communicate with each other, with trading partners and the like. What’s been your experience with APIs? I mean this is part of your everyday job?
Rainier: Yes, it is part of my everyday job. We have a lot of clients that are doing transactions every day, weekends, 24/7 shops where they’re consuming and also hosting web services and APIs, microservices. It has many different names. I deal with them all the time. Yes, we’re always deploying new things, adding new features, making it always better, and APIs tend to be like the framework of how you communicate now. It’s platform agnostic so basically you put your API out there and everybody can connect to you or you can get to them.
Charlie: Rainier, I know that working with APIs on IBM i specifically, we’ve seen how the technology has changed through the years, and what I mean by that is the tooling that we’ve been given by IBM to work with APIs has only gotten better with every release and every technology refresh. I mean for example I know in the old days—whatever that even means—we were using the old HTP GitHub, but there’s been some vast improvement in there. You want to talk about some of that? First of all, let’s talk about the old techniques, and then how it has gotten better.
Rainier: It’s been a little while that had the HTP GitHub and Git Blobs came out. Those are SQL functions where IBM put the SQL command on top of Java to allow you to go out and consume other web services or APIs that are out there—microservices. Now they have the newer ones where it seems to be more cURL-based from my understanding. It’s a lot faster and it’s pretty much doing the same job, just a little different.
Charlie: You mention that it does work faster, it performs better, but what have you seen performance-wise in the amount of time it takes to consume one of these services using the old and the new?
Rainier: Well where the older web services were Java based—if anyone has ran Java on the i, they know that it’s a little consuming. It takes up some resources. You have to build a Java environment and there are some hoops you got to go through. Leaving Java and moving it to cURL, the calls are probably, I would say, almost 200% faster. It’s head over heels faster, the newer commands compared to the old ones.
Charlie: And just to be clear we’re talking about consuming web services, or APIs as you say, because obviously in this discussion, it goes both ways. We can both consume and serve, and we’ll talk about the different things, but right now with these new functions, we’re talking about consuming web services.
Rainier: Correct.
Charlie: Okay and the newer functions we’re talking about, the new SQL functions, how difficult if at all was it to make the transition to start using these new functions? Was it something that you just roll it right in? I know the actual syntax format of the newer commands look a little bit different—you know they have the underscores—but what was the transition like to use the newer functions?
Rainier: At a very high level, they do the same exact job. When you get down a little bit lower, you’ll notice certain things, like IBM kind of flipped the data portion and the header specs or the options. When you called one of their APIs before, it was the URL and then there was like the header. If the call requires data, then there would be the data parameter. Now they kind of flipped the data and the header or the options—as they call it, parameters—so if you’re going from the Java one, you have to know that basically it’s the same three variables, but they kind of flipped the order on the last two, which would be the data and the header.
Charlie: I know that we have one big service program that does all the consuming that we use, and we’ve included in the service program all of the function both the legacy and now the newer ones as well. What was the purpose of that? Why do you have to keep both of them in there and just one big service program that handles APIs?
Rainer: I’m glad you asked that question. Also, there are differences between the underpinning of the two. [With] the Java web services, the certificates are actually built into the Java environment. The certificates are using a typical IBM certificate store so and a lot of times when something wasn’t working and you might need a new certificate, or it worked with the old one but now it’s not working with the new one. Sometimes you have to export the certificate from the Java certificate store that’s built into Java and then import it into your new certificate store that IBM recommends you make for the new SQL HTP functions. Another caveat to keep in mind is that the old SQL functions were all XML-based, so your header and header spec, your option specs needed to be in XML. The newer one is JSON-based, so you have almost the same options in the header and everything, but now everything has to be JSON syntax.
Charlie: So if I isolate in my shop all of the APIs being consumed, the programming as we’ve done in a service program, we can immediately start using for all of the programs that need APIs being called, they can immediately start using the newer functions right away?
Rainier: Yes. That’s why we did it that way through the service programs, because in the service program we have an options parameter that comes in. That’s a series of flags how you want the service program to perform. One of the flags was to use the new or the old service calls, so basically from all the programs nothing really changed, but inside based on the parameter we could decide whether you were going to use the new one or the old one.
Charlie: Why would anybody ever want to still use the old one if the new one is so much better?
Rainier: Well like I said, it’s still kind of there for legacy, because sometimes we haven’t gotten to that application yet. It’s also there sometimes proactively, because when we have a new customer coming on and we can’t seem to get it working on the new one, we run the old one just to see if it does work. And if it does work, what’s the main difference? Why is it working on the old one and not the new one? To me it’s just another tool in the shop, the same thing as if we had to use Postman or SoapUI to figure out how you’re going to do your web services or how you’re going to consume them.
Charlie: Yeah, and those tools you mentioned, Postman and SoapUI, those are very good I guess to initially learn about the data—you know, the payload—and see what’s coming back, and that’s something I guess you would need to know to do programs with a particular API.
Rainier: I tend to use that as a first step because before I even go into the whole making the program. The API microservice, web service, however you call it, I want to make sure it works, so I want to eliminate the IBM i from the whole thing. I go to Postman or SoapUI and just make sure the credentials work—you know, the information that was provided to me—I want to make sure it works because there’s nothing worse than when you’re building all these connections, you’re building a program and you find out the token maybe is bad or the URL is wrong or something like that. You’re spending all this time knocking yourself on the head trying to figure out what’s going on—
Charlie: You heard it here first: So SoapUI and Postman really are essential tools to get into this world.
Rainier: It also gives you a good snapshot because you can look at the raw data that’s being transmitted to and from SoapUI or Postman. You can also see what’s being sent and what the raw data coming back is. In the same regards with the data coming back and getting as much data as you can, I would recommend—even the older functions I always use the verbose versions, because then you can actually get the header back, get the verbose header so you can actually see the HTP header that’s sent back. You can see the code or the error codes and you can see things of that nature for what’s going on.
Charlie: You know we get into the world of APIs and there’s always different discussions, different technologies. For example when we first started getting into this years and years ago, the big API, the big man on campus as it were, was Soap and I know we have a couple of customers still using Soap because that’s what their customers are asking for or whatever. But what are you seeing out there as far as REST vs. Soap. I mean clearly by now I have to imagine most customers are now using REST.
Rainier: Yes, Soap came out first. REST was a little late to the show, but REST is a little more flexible. It’s not a protocol like Soap is, so it tends to more flexible and more customizable. If you deal with different marketplaces and everything, everybody has their—it’s you know, using the same REST API, you are doing your gets and puts and everything, but everybody has their own little flavor to it. How they’re going to secure it, how they’re going to be doing the tokens, and how they want the data to be sent, whether it is encrypted and all that stuff. So yes, I do see a lot more REST out there than Soap. Another reason I guess would be because Soap is also XML-based and REST services can be XML, JSON, or whatever you want it to be because it’s more flexible.
Charlie: But you’re finding today also in that regard JSON is clearly the king today.
Rainier: Yes. I haven’t received any new Soap calls. Some of our clients that do have Soap web services, we’re still maintaining them, and in some regards we’re maintaining the Soap and also introducing a REST web service—you know, APIs for the customer to consume. Because a lot of times like I said, we’re running into more customers asking for JSON, not XML.
Charlie: You touched on this briefly, you talked about newer versions of services becoming available, things like that. And that brings the conversation of API versioning. First, can you explain to everybody what versioning is and how it works in the world of APIs?
Rainier: Versioning is basically when you make an enhancement or change to the API web service program, where maybe you’re offering more information or you’re changing the way the data is going to be sent back to provide more information or details to the consumer and vice versa. When you’re consuming it, the person you’re connecting to may offer a new version of the API. A lot of times they do that so they can basically gradually migrate everybody to the newer version and everybody can get used to it. So what they tend to do is they have one URL. Sometimes they embed the version in the URL, so you’ll see like is it V2 or V3. When they come out with a new one, it’ll be V4 or V5. That allows you to develop while you’re still in production, and once you’re satisfied you’re getting the right information, you’ve made the correct modifications to your program, you can migrate over. So that’s basically why it’s done that way. It allows allows everybody to migrate at their path, your own timing. They usually set deadlines though—they usually say like, in six months we’re going to decrement the V2. You have to go to V4 so they give you six months to get yourself on board. I tend to use the same practice when we’re going to make a change or something like that, we tend to make a new URL or a different branch—you know, extension to the URL—so the user can get the new experience. But we don’t take away the older function for other customers or consumers that are not ready for it.
Charlie: Right so, we’ll have two or maybe even more versions of the same—
Rainier: Even more.
Charlie: Or similar program active at one time.
Rainier: Yup. Like I said before, we do have some clients where we have the legacy Soap service that we’re hosting and then some of the clients ask for it to be in JSON and stuff, so we introduce the REST version of it as well, and it has a different URL. So you can go to the Soap one or you can go to the REST one.
Charlie: Right. When we get a payload back from an API, we have to do something with this payload. We have to parse it usually, and there are couple of ways to parse data on IBM i. I know you can do it right from within RPG. Obviously if it is XML, XMLTO or DATA-INTO, but those are not the only two ways and I know you have a more favorable way of doing it. I know you like using an SQL. How would you normally parse a payload using SQL?
Rainier: Well yeah, there are some built-in functions that IBM provides like JSON Table and XML Table. I like XML and JSON Table solely because I can specifically ask for the data I want and pull out just what I need. Some of the other parsing options you mentioned before, you have to pull all the data in. So it’s nice in SQL where if I’m pulling data and I’m just only looking for the updated price, I don’t need to get everything. I can just basically pull the API or whatever, which will bring down all the data, but when I actually pull the data out using SQL and I only want the price, I just pull the price out.
Charlie: Rainier, I know that one issue that we have to deal with—and I imagine a lot of people have to deal with are character sets, differences in character sets, CCSIDs or CSIDs, depending on who you talk to—but what challenges have you had with CCSID or CSIDs? Why might they become a real issue when you get involved with programming with APIs?
Rainier: The #1 reason why CSIDs ends up being an issue is because IBM runs EBCDIC and the rest of the world pretty much doesn’t. So pretty much any data you’re getting from the outside world, it’s not going to be in the same character set that you’re accustomed to on the IBM i because everything on the i is EBCDIC. So you’re typically dealing with UTF-8 but sometimes you will bump into ASCII as well, and you need to be cognizant about that because when you’re converting data back and doing things like hashing data and storing data or zipping it, you want to make sure the data is in the right format so when the data is set or compared or whatever, you’re comparing apples to apples. IBM makes that kind of easy because you can apply a CSID on the declaration of a variable. You can apply it and say hey, this character can be UTF-8, this one can be EBCDIC—or actually default is EBCDIC and you say oh, this one is ASCII and you can kind of convert by moving the data between the variables. IBM will be nice enough to convert it. One thing you need to be cognizant of though is, especially coming from UTF-8 to EBCDIC, UTF-8 is quite large. Not all the characters are represented in EBCDIC. And that’s actually another point that’s different between those SQL function calls where the Java one throws in its own special character if it doesn’t how to convert it and the newer ones does kind of a default IBM thing. So you have to be cognizant that if there is a UTF-8 or some data that’s coming across that can’t convert, you’ll see an X15 sitting in the middle of your data and you need to know that X15 means there was no conversion for that character. So you need to do something with it because a lot of times now if you try to bring that data into your EBCDIC field with an X15, IBM is going balk at you and say hey, that’s not character data. So a lot of times we have to strip it out or change the X15 to either a blank or change it to another character so the person down the chain will know later that there was some unconvertable data at this point.
Charlie: So, it’s a topic that can’t be ignored is what I’m getting from this conversation.
Rainier: No [laughs], it can’t be ignored, because like I said before, the nature of the business is being able to deal with different character sets, and different character sets could mean different languages as well. When you start getting into foreign languages like in Asia and the different cuneiforms and stuff like that have different CCSIDs as well, you’re going to be dealing with that stuff. CCSID plays a big role in all of that and you need to be able to know how to convert back and forth. You need to make sure that you’re zipping something or hashing something or converting something and using those commands—you know, base 64 encoding or if you’re doing URL encoding, you need to make sure that you’re encoding it using the right CCSID—you want to make sure that you’re encoding it in ASCII so when the guy does the hash check on his side and he does the ASCII, you’re getting the same values. I’ve seen a lot of people have issues where they hash it but they didn’t convert the data to ASCII, so they have to hash it and then they send the hash over and it doesn’t match because you have an EBCDIC hash mashed up with an ASCII hash and they’re not the same—you’ll run into issues.
Charlie: So it almost sounds to me as if we should have a separate podcast just talking about CCSIDs and how to really address them under different circumstances.
Rainier: Yes, if you want a really dry [laughs] dry conversation, because that’s what it is going to be.
Charlie: Well, it’s funny you say that because I know a lot of conferences may shy away from having a session on this very topic because maybe it is so dry as you say. But it’s almost like, how can you ignore it? It’s such an important point of this whole topic.
Rainier: Well, yeah. I see it as a big pitfall but usually once the user or the programmer or developer kind of realizes—once you get your foot kind of dipped into it—you got to know it and know how to start dealing with it. Because once you deal with one character set, dealing with another one or what—it’s all the same at that point.
Charlie: Right. It’s just getting into the practice of understanding what they actually are.
Rainier: Yes, getting into the practice of understanding it and recognizing what’s going on when you get an error because of it. Because as I said it was a difference between the Java-based SQL HTP functions where Java was converting it to a character that could proceed later on the in the chain, but if you’re consuming it using the newer ones and you try to throw it into an EBCDIC field, you’ll get a conversion error and it will say you get a SQL error telling you hey, you’re trying to put data in this field that I can’t put it in. You dig deep enough you’ll see some X15 or something like that in there and sure enough, it’s not going to run and you’ll have to do something about it.
Charlie: All right so let’s keep forward with the conversation then. If we have a customer who is starting to use APIs, what other tools do we have at our disposal to monitor the progress and to track errors for error handling, things like logging for example. What tools are available that we can use to implement and get these APIs up and running?
Rainier: When it comes to monitoring and things of that nature, if you’re using the SQL functions, like I said I tend to use the verbose versions of those so I can get as much information as I can, and we do log that stuff. We tend to log it more on the IFS than in a database file, even though in some cases we have stored it in a database file, but generally I’m storing it out in the IFS. Again like I said we’re getting data in UTF-8 and ASCII and it’s just easier to log that stuff because the IFS supports those CCSIDs a lot more natively. So it’s just taking the data and just dumping in there. It makes it a little easier. When it comes to hosting and that stuff, IWS does provide it. If you turn on the debugging and logging features you can get the remote IP address. You can get more connection information that we do log and put in the database. So you can see who’s connecting, how often they’re connecting. You know we mentioned before, I think, throttling—or this might be a new topic. Basically, you may want to slow down somebody if he is hitting your machine way too often.
Charlie: We’ve seen throttling—for example if we have a tight loop and we’re hitting Amazon a lot in a very small amount of time. They are famous for throttling.
Rainier: Yes.
Charlie: So what does that actually mean? What does that mean when a service is throttling?
Rainier: It kind of just means that you may be up to no good. You’re making too many requests within a given time period and basically, they want to slow you down. They just want to make it more manageable for them and they want to make sure you’re not hogging up all their resources on your requests and you’re giving everybody else a fair shake on getting their stuff through.
Charlie: Right, so you have to handle that in the code somehow. You have to be able to handle that first recognize you’re being throttled and then handle it somehow. How do we do that with our customers?
Rainier: Our customers? I’ve developed, you know, kind of a program that just keeps looping through the logs, keeping a look at the time stamps, seeing in an average minute or given time frame how many times the requests come in and what’s the turnaround time, and sometimes how long the requests are, depending if they’re making big bulky requests or if they’re just making little requests. We balance that out where you don’t want too many people making huge requests and you don’t want people making like little like pinpricks either.
Charlie: Since we’re hosting, we’re serving.
Rainier: Yup, serving it up.
Charlie: Right, because I guess we’re talking both sides of the equation here, consuming and serving. We’re going back and forth. So let’s focus back again if you can on the consuming side, because I know we’ve gotten throttled by some out there.
Rainier: Like you said, Amazon is famous for it. So basically when you send the message to Amazon, you’ll get a message back. Amazon is nice enough to tell you they’re throttling you. They’ll give you an HTP code; they’ll give you a description saying you’re being throttled. So basically when you see that, you bring the data in, you say oh, they’re throttling me. So typically what we tend to do is we put in the code and we kind of put a delay in the system and then really try to request again and keep trying until we’re successful. Or I think in certain cases because again, Amazon kind of tells you when you’re being throttled, so you can’t make another request for 5 minutes or 30 seconds or whatever it is. So you kind of wait that allotted time and try it again. In most cases I think if you’ve gone through the whole cycle of trying and you’re still getting throttled, we tend to send out an email or we put a message out on our messages just basically saying that hey, this job has been throttled and it has happened more than five times in a row. Maybe something is going on or we need to look at the process so it doesn’t just sit there and endlessly loop.
Charlie: So this entire discussion, Rainier, or the bulk of it anyway, has talked about us consuming services. But we both know and we’ve said already that obviously IBM can also serve web services or APIs quite well, and we use IWS—as we said, Integrated Web Services.
Rainier: Yeah.
Charlie: What are some of the examples that we’ve done in the field using IWS at our client sites? Who would typically consume our web services?
Rainier: From our standpoint, it’s mainly customer-based, where we’re providing our customers with information—like it might be real time inventory. We’ve done some stuff where the customer basically got tired of storing copies of the invoices every night to their web server, so we created a web service for them so if a customer goes to their web page and asks us for a reprint of an invoice, we actually will produce it, encapsulate it and send it to the web service to the website to present it to the customer in a PDF format. Also internally we’ve done some stuff where we have some warehouse automation which has been using some web services. You have pickers and stuff like that where they’re requesting an item has been presented that you want to get an order for and after that order they probably maybe want shipping information for labeling purposes and things of that nature, and we’ve provided that kind of stuff. We’ve done a lot of different things. As you know, we also have provided other information like order statuses and things of that nature through web services.
Charlie: And the good thing of course because it’s an API, it can be someone running a PHP or Node—mobile app for example. Who cares?
Rainier: Right, yeah. It doesn’t make a difference who is making the request as long as they can speak HTP for the REST service and they can handle the payload when it comes back. That’s pretty much it.
Charlie: The final thing I want to talk to you about before we wrap this up is security, because that’s always front of mind in most cases, or probably all cases it should be. But if we’re consuming web services, I know we bump into security issues with whoever is providing the services. What are you seeing out there from security and how do you handle some of the security issues that come up?
Rainier: I would say like most of the marketplaces and stuff like that, they’re using some version of ORATH where basically you have a token and you have maybe a private token and stuff like that, and you have to make a token request. Then once you get the—I guess the transaction token or the actionable token, basically the life of that token could be anywhere from 5 minutes to like half an hour. I haven’t really seen one longer than a half hour, but basically you would use that as you’re doing your transactions, things of that nature. I’ve also seen hashing where you need to hash your payloads and what’s being sent over and using that as base security, so basically if anybody tampers with the payload, that changes—anyway the hash would be different so they know that it wouldn’t accept the payload. And of course you have your SSL security. That’s through HTTP and that’s certificate-based and you need to be aware of that as well. Like I said, especially with the newer SQL functions, you’ve got to make sure those HTP SSL certificates are in a certificate store that you need to reference in the options when you’re using the newer SQL functions to interact with and consume a web service or API or microservice, whatever you’re calling it these days.
Charlie: Yeah, one last thing I want to point out is that many shops don’t have what we consider a modern application, or well written code or legacy code—older code I should say, older style code—you know, more monolithic. It’s not modularized, things like that, but yet we’re still able to pull it off and introduce APIs. It’s more difficult perhaps to put into a monolith than it is in a modalized application, but what advice might you have for somebody if they have a real requirement—one of your best customers for example is demanding today that they use APIs. How do you prep that customer or how do you prep that code base to start using APIs? Are there any last best practices that you can recommend?
Rainier: If you have a lot of monolithic programs out there, the #1 thing is you really need to start modularizing and breaking them down. Usually when you’re introduced to a new topic like web services or you need to start interacting with the global economy, as you put it before tha, that’s a perfect time to start modulization and start cutting your code up. Even if you have that monolithic program, you can create a web service program or something like that that handles just this new stuff that you’re dealing with. I tend to kind of handle it that way where the monolithic code is out there and usually it’s reading from a file or something like that, so I would have a call either before I call the monolithic program or around it or whatever where I do my stuff. You know we reach out to the customer and we get the data or the payload, whatever we’re getting from them, then I’ll kind of format the data in a way that the monolithic program can consume it and convert it—basically let the monolithic program run. So it’s just kind of friending it, in a manner of speaking. So that’s what I tend to do in these situations with monolithic programs. I kind of front-end it a little bit, so you’re still able to use the monolithic program—even though it probably should be broken down because those things are horrible to service or modify, or add features to for that matter.
Charlie: So cool. We really are lucky. We get to work with a lot of different customers and sets of requirements. I know it’s something we need to do, because the demands of IT today are APIs for sure. I mean that’s how we started the conversation with the API economy and it’s completely true.
Rainier: The best part about it is when you go into the client’s shop. They always ask you: we’re getting a request for this, I don’t think this can be done. And I say yeah, it can be done. We can do it. Then afterwards, after it’s done, they say oh, that didn’t seem that difficult, and it’s really not. It’s just basically getting to the right mind space and understanding how these things interact, and it’s just like calling another program, but the program is on your machine. That’s all it really boils down to. Everybody knows how to call a program. If you’re a programmer, that’s all do is call programs. You should know how to do this. So it’s just basically understanding the nature of calling the program on another server, another site, wherever it is, wherever it could be. You can call it; you can use it. That’s what makes these APIs, microservices—a lot of people have names for them, but they’re all kind of the same thing. It’s really nothing but remote procedure calls is what it boils down to.
Charlie: You know in the end as you go down this road more and more with APIs and get into that model of programming, what that also does for you is it also gets you on a great path towards real digital transformation. Because as you start getting involved with newer technologies—even like AI, which is of course the big thing today—AI, that’s going to rely on APIs.
Rainier: It’s all APIs, yup. Pretty much everything that’s out there now, that’s been coming out lately, it’s all API-driven. You could technically start doing it internally on the i if you really wanted to—I haven’t really seen but you know, that could be another way of developing new stuff as well. Just start developing everything as an API. Maybe you could even consume it yourself using the same SQL functions as IWS and other methods as well. They’re out there and like I said, because all they are is remote procedure calls—remote however you want to say it—it’s just calling another program and getting data back and using it.
Charlie: Awesome. Rainier, I can’t thank you enough. Thanks for spending some time with me today. This was a great conversation. This is such an interesting and fascinating technology to me. I mean, I love working with these. I love talking about them. They’re fun I think, and they really do extend what you can do on your machine. It just opens the entire world, and that’s really where you need to look at everybody, because that’s the world today. If you’re going to have any meaningful applications at all, I think APIs is going to be for sure a big part of your tool set.
Rainier: A big part of the tool set, correct. Being part of the global economy, you need to be able to access information wherever it is, and this is your key to being able to access that information whether it resides on the IBM i, whether it resides on your website or your web server, whether it resides in another database somewhere else. These are the keys that you need to be able to use to access that data and to leverage that data, to make everything better.
Charlie: Absolutely. Well Rainier, once again, thank you very, very much. It’s always great talking to you. I mean obviously we work together, but it’s always a pleasure to talk about this kind of stuff with you. Thank you.
Rainier: Yeah, always a pleasure.
Charlie: All right, everybody. This brings us to the end of our podcast. Thank you so much for listening, and I do encourage you once again to visit the TechChannel website. There’s lots of good information out there, lots of good podcasts and articles. It’s really worth your time. Until next time, bye now everybody.