Peter Fandel on DevOps and Open System Collaboration
Reg Harbeck: Hi, this is Reg Harbeck and today I’m here with Peter Fandel, who is the director of product management at Rocket Software. Peter, welcome. How did you end up in the world of mainframe?
Peter Fandel: Ah, well thank you for having me today. How did I end up in the world of mainframe? Well I started out in factory automation and process control, bringing real time expert systems to those industries—and UNIX was, I guess, the primary operating system that I used for many years in that domain. Rocket acquired the company where I was working—oh, about 21 years ago—and decided they wanted to apply the technology we had built to the mainframe and mainframe applications. And so I came at the mainframe from an outsider’s perspective, more comfortable always with UNIX rather than z/OS. So the first sort of engineering team that I was in charge of at Rocket was the team that was porting open-source languages and tools to z/OS, and so that’s always been the perspective that I came at it from.
Reg: Now you say, according to z/OS—you’re obviously talking UNIX System Services and not Linux. Porting stuff to USS is one of the most interesting things in the industry, because on the one hand USS was the first truly compliant UNIX, but on the other hand I think probably the safest word used to describe it by people who have actually tried to port the two is clunky. What was your experience like?
Peter: Well very much so, and C programmers have for decades made assumptions in their programming. One really sort of obscure assumption is that the letters in the alphabet are contiguous when you translate them to numbers, and so you know coders will sometimes say, so long as this character is between lowercase and uppercase Z, then it must be a letter in the alphabet. That doesn’t work in ASCII, and so as many of you know in this audience, the mainframe predates the invention of ASCII, and so the heart of porting to the mainframe, to z/OS, is dealing with the character code page conversion issue. So anytime that the program interacts with the file system or with pipes, you need to worry about that, and that’s a lot of what goes into porting on the mainframe.
Reg: Well I have to admit my brain is just full of images of all the contrast between EBCDIC and ASCII. I mean I could sit here and list them off and put everybody to sleep, which is not the goal here. But now that said, there’s also interesting things, like big Indian, small Indian numbers and stuff like that. What do you find is the most effective approach to as dynamically as possible enable the smooth conversion between the distributed UNIX way of doing things and the mainframe USS way?
Peter: Well I would say what has made it smooth is we’ve been doing this for about ten years now, and so the engineering team has put in place a lot of automation—automation that in some cases that actually automates the porting itself. For some of our ports, we actually can generate z/OS compatible binaries just from the compile of Linux, thanks to a custom GCC glib C port that we did that takes care of the code page conversion in the compiling and the linking to the libraries. That doesn’t work for all of our ports but for BASH, for example, it does work and it means that we can generate a new port of BASH directly from the source code, download it from the community without a single edit to any of the sources. That’s the exception but that’s an example of automation that we’ve built up over the last ten years. Also, automation in testing is crucial. What we do is we have an automated harness that runs the built in tests that come with any open-source project, plus a whole suit of custom z/OS specific tests that the team has written over the years. We execute those tests, the full test suit on UBUNTU, and then we also execute it on z/OS. Anytime that the results are different, that’s where we investigate and that saves a lot of time to have that fully automated. We essentially drink our own champagne in that we use our own ports in our own internal DevOps tool chain for building and testing and releasing these ports.
Reg: What I hear you saying is this experience, that this is actually part of what you offer your customers as well is all this experience in being able to smoothly port stuff. Is that right?
Peter: Yes, exactly. Because we have built up an open-source DevOps tool chain that we use internally, and that’s what a lot of our customers are moving towards. They don’t want to be maintaining a separate DevOps tool chain just for building their mainframe applications. It’s expensive, but more importantly they’re finding it is harder and harder to find anybody to staff those teams because the folks—the gray beards, if you will—that are familiar with traditional mainframe interfaces and tools are fast retiring. They’re exiting the job market for one reason or another, and it’s really hard to get a kid out of school—or you know, a millennial developer—to embrace 3270 green screen emulators as an interface.
Reg: Now one of the things I hear implicit in what you’re saying is the C programming language. I’m going to bring us around to talk about COBOL before this recording is over, but tell me a bit about the role of C in all of this.
Peter: Well most of the DevOps tools and most languages used—especially the DevOps tools and to some extent languages that are used today in application development—are in fact written in C.
Reg: Like Python?
Peter: Git is written in C, curl is written in C, BASH is written in C. All of these open-source tools that developers expect to be available wherever they do their work, those are in fact written in C. So 99% of the code that we port to z/OS is written in C.
Reg: Oh. Now of course Python also is written in C. Are you finding you’re doing a lot of porting of Python directly, or is mainly C?
Peter: Well if it’s written entirely in Python, no porting is necessary most of the time; there’s nothing to it. But there are a significant number of important Python packages that are in fact written in C for performance reasons, and those do need to be ported just like any other C program.
Reg: Oh. Now I understand that one aspect of all of this porting and this open systems approach you’re doing is an involvement with this thing called Zowe. Any thoughts about how that kind of fits in the present and the future?
Peter: Oh yeah well Zowe is hugely important. It’s a different kind of open-source in that the development of these ports vs. Zowe is quite different work. What I refer to as the ports are languages and tools that have been around for a long time. Some of them—like you know, the zIIP tools and BASH in particular—have been around for probably 20 years, and we’re just making them work on z/OS. Zowe on the other hand is Bespoke open-source. Zowe is open-source that didn’t exist before it was introduced in 2019, and it was designed from the ground up to be for z/OS and crucially important for modernization. Zowe is essentially an ecosystem of interfaces to z/OS, and when I use the word interface, I mean all definitions of that word: so, graphical interface, programmatic interface, APIs, also command line interface. We’re coming out with a Zowe chat component within the Zowe community—it’s not released yet, but maybe at the end of next year—so all these different ways of interacting with a machine. Zowe is making it a modern interaction using modern technologies and modern interfaces to that z/OS back end. It is a crucial part of mainframe modernization.
Reg: What I hear you implying is that there’s more than one kind of open-source that you’re dealing with. There’s—
Peter: Yes, well—
Reg: You’re referring to the non-mainframe, and go ahead.
Peter: Yeah, right. I mean Zowe—a lot of it is written in Node.js, some of it is written in C, some of it is written in Java. But like I said, it’s open-source that was written expressly for the purpose of interfacing with z/OS, so there’s no porting involved.
Reg: Hmm. Got it. Yeah, that’s one of the big challenges of the whole open-source movement of course is porting. You know the old saying about UNIX? You know that UNIX is supposed to write once and port everywhere, it’s actually write once and debug everywhere. The nice thing about Bespoke, I guess, is they can be open-source for this particular platform, but what are some of these challenges that you see of—you know, bespoke—as distinct from the challenges of porting all over the universe in generic open-source?
Peter: Well the challenge there is that you actually have to code the features [laughs].
Reg: Yeah, right.
Peter: Right? With the porting, for the most part the porting team is not writing any new features. They’re not implementing new features. To some extent of course, they are. The feature of converting between ASCII and EBCDIC for example, the porting work is mostly a little bit of coding, a lot of testing and a lot of running of infrastructure and tooling for delivering the code to the customer. But in the case of Zowe, you’re developing new features from scratch, so you need a larger number of engineers. I would guess that you’ve probably got upwards of 100 active developers working in the Zowe community from a variety of different companies, so it’s a much larger project.
Reg: Now I’m going to get to security net. Before I do though, I just want to talk a little bit about more languages, because two other languages that I think we need to surface—one of them is COBOL and the other is VS Code, and of course VS Code is not precisely a language, and the neat thing is that COBOL even can run on top of it. But maybe if you could just get a sense of how you perceive the role of both COBOL and VS Code in this whole open-source approach on the mainframe?
Peter: Sure. Well in the case of COBOL, it’s really our reason for being in the porting team is to make the tools that enable a multi-platform off host not on the z/OS DevOps pipeline to be able to build COBOL programs and test them. So COBOL is essentially the test subject that needs the code checked out, the compile, the link, the test suit, the deployment. It’s the test subject of the DevOps pipeline, and so it’s one of the three I would say primary languages that mainframe applications are written in, and then our ports are used to build. So COBOL, PL1 and C to some extent programs over on the MVS side that need to be built by our tools running in a combination of off host and in UNIX system services. VS Code—well it is the IDE of choice for the new generation of developers, it’s hugely popular—and one of the components of Zowe is an extension to VS Code. It’s called the Zowe Explorer and it enables a developer on VS Code to browse mainframe data sets and edit mainframe data sets, edit COBOL, and then save those edits and then you know the DevOps pipeline can kick off the build and test after that.
Reg: Now of course for all of that you’re dealing with open-source, and open-source has two major reputations. One of them is flexibility, because you’re building on stuff that other people did, but the other is a bit of a security exposure. As they like to say, there’s no one neck to choke when things go wrong. You know that it’s kind of nice to have somebody who can take the blame, and because it’s kind of got that tragedy of the commons kind of feeling sometimes, I think large sites are rather skittish about using anything that doesn’t have somebody’s whose reputation is there. So can we talk a little bit about how does Rocket help organizations feel secure using open-source on mainframe?
Peter: Sure, yeah—well we take it very seriously. Our customers obviously are very concerned, with good reason with some of the examples like the Log4J example of almost a year ago, and we attacked this problem both proactively and reactively. On the proactive front, we run a lot of source scans on every release before we let it out the door to scan for known vulnerabilities, to scan for provenance of the code as well.
Reg: Very important.
Peter: So there’s two kinds of security and legal angles. One is are there vulnerabilities in the code, and then the other one is is this really open-source, because I don’t want to bring code into my shop and find out after the fact that it was labeled as open-source but in fact it has some proprietary code in it. So both of those have to be addressed, and so we use source code scans for both of those purposes and no release is allowed out the door if it’s got high or critical security vulnerabilities. We have an SLA in place that is modeled after our primary business partner that we adhere to, which is IBM’s and we adhere to exactly the same SLA that they do with respect to that, and so that’s our proactive measures. Our reactive measure is that there is a national vulnerabilities database which is administered by the National Institutes of Standards in Technology out in Colorado, and we have a sniffer server on our network that is listening to—it’s subscribed to the national vulnerabilities database—and any time a new vulnerability is published and made known to the world that concerns any of the ports that we provide to our customers, our team is immediately notified and our team then immediately goes in, takes a look at that vulnerability, finds out does it apply to z/OS, because many vulnerabilities don’t. Many are operating system dependent, only apply to Windows, only apply to UNIX or whatever—
Reg: Right.
Peter: So we determine if we’re vulnerable and then if we are, the team immediately gets to action to apply that patch and put out a new release.
Reg: Okay. Now one of the things I started—
Peter: That’s the reactive half of our countermeasures.
Reg: Good, good. One of the things that is sort of echoing in my back of my mind as you talk about this is the imminent prevalence of containers under zed/OS, as I like to say. You know that on the one hand, Linux containers, but on the other hand there’s some pretty reliable rumors that we’re going to have more than just Linux running these containers and we’re going to have this really interesting recursive experience that in a lot of ways is a container behaving in an open manner, so that zed/OS becomes essentially an open operating system in every sense, not just the original open edition MVS that UNIX System Services was. So maybe, especially from a DevOps perspective just to load this question down even further, but from a DevOps perspective, how can you see this open-source approach and all the ways that it echoes through the structure of the mainframe environment really being part of how you approach the open-source opportunities?
Peter: Well if you’re talking about running open-source in a container, I think most of the time it’s either the zCX containers, which is a fairly new product, or I don’t know if you’re referring to—I mean you can think of Linux on z. It’s not a container, it’s a virtual machine, I guess you can think of it. But in those two cases, there’s some things you can do running open-source on those two platforms, Linux on z and in a zCX container. However it’s limited to some degree because it is a different machine. It’s a virtual machine in both cases, and so any interaction with mainframe data sets—first you have to transfer the entire file, the entire data set across TCP/IP protocols from z/OS to that virtual machine before you can work on it. So you don’t have direct access to the mainframe data from a virtual machine or from Linux on z, and so there’s a lot of things you just can’t do—
Reg: Right.
Peter: So basically you often can’t go the easy route. It is a lot easier to get an open-source program that preexists to port if you will onto Linux on z or onto zCX, but you can’t solve all problems with them. And so really our z/OS ports address those customer use cases where they need direct access to the mainframe data.
Reg: Well that makes sense. Now then I guess taking a step further into the whole DevOps area, how do you see DevOps and open-source sort of interplaying as things move forward?
Peter: Well you see them more and more being one in the same in that it’s just 90% of DevOps tools that are commonly used are open-source, and so if you want to do modern DevOps, you’re using open-source. It’s just a fact. I mean there’s some tools that are proprietary tools used in DevOps pipelines, but the heart of it is all open-source, so you have no choice.
Reg: And of course—I mean DevOps is a deliberate mashing together of two separate terms. Ops—operations, think of automated operations, and then development, which is always been one of the biggest things on the mainframe. And of course the ultimate open-source part of that is Eclipse. Maybe if you can just kind of relate that to us and help to understand the role that plays and what it says about it?
Peter: Well Eclipse is—I see it as a great platform, but it’s not today’s platform. It’s kind of yesterday’s platform.
Reg: Okay.
Peter: And in fact it doesn’t run on the mainframe at all and we’re not planning on doing any kind of a port of that, but certainly there’s a lot of software out there that’s Eclipsed-based. It was a very successful open-source project. Actually it’s kind of interesting to contrast it to the Zowe Application Framework, which is part of Zowe. So one of the four Zowe interfaces is the Zowe Application Framework, which is also known as the Zowe Virtual Desktop. And that is a browser-based GUI experience into mainframe, into z/OS. It gives you a desktop experience similar to Macintosh OSX desktop or a Windows desktop with icons on the virtual desk and toolbar and many windows from different applications, the difference being that the Zowe Virtual Desktop is running inside of a tab of your web browser. So it’s zero footprint, whereas Eclipse was a thick client that you had to install, that you had to maintain and Java based, and so it became a bit of a headache for companies because if they had you know, 300 operators using an Eclipse-based program, they would have to maintain the latest patches of whatever Eclipse program they were running on those 300 workstations, but those workstations already had web browsers that are maintained anyway. So it’s zero additional work to maintain the client software necessary if you have a browser-based interface. Also the Eclipse framework is constrained in terms of the layout of the GUI. You’ve always got the tree browser on the left and the center panel and the menus and you have to work within those constraints to build your application that is Eclipsed-based, whereas the Zowe Virtual Desktop infrastructure makes use of react and angular web tool kits and basically you can do anything you want. So you can design an entirely Bespoke user experience and GUI on top of this virtual desktop all running in a browser, so it’s much more flexible for designing user interfaces.
Reg: Well one other axis or dimension of open-source of course is how do you get support, and I know that even with the security question being well mitigated by what you’re doing, that it is still a question. I mean if I’m a big company and I’ve already got a license for Rocket Software, I’m already going to get my support. But what if I as a developer or person just on the journey of learning the mainframe wants to get some of those open source you’ve ported to the mainframe environment? Do I even have options of getting this open-source as an individual user?
Peter: Sure. Yeah, our policy is not unusual in the open-source world. What Rocket does is we obviously are spending money and making an investment, and we have people who collect salaries who are doing all this open-source work. In order to fund that, we sell support products that support both Zowe and support those ports, the open-source language and tool ports. And the way that we do it is we let anybody download these ports and install them and run them on their systems. However, if you want the very latest binaries, if you want to be up to date on your security patches, then you need to be on paid support. So there is a delay of a few months before a security patch that we make available to our paying customers immediately is then made available to the public download site. So if you want to play with these ports, experiment with them and test systems, you can do that free of charge, and we even provide a website that you can go to where you can ask questions, you can ask for help—
Reg: Oh neat.
Peter: You don’t need to be a paying customer to have access to that website. It’s called the Rocket Forum. The customers, you just help each other. So our development team does answer questions on the forum, not at the same priority as our paying customers, but we do help out our user base that way.
Reg: I think people are going to really enjoy that, because when I interviewed Enzo Damato—you know, new mainframer who’s got a mainframe in his garage, and asked him what advice would you say and he said, you’ve got to get people who are still in school having access to mainframes. I’m thinking okay, well how do we do that? Well this is one of those ways, you know by making open-source mainframe stuff available for personal downloads, and if it’s not being updated and supported at the same speed as the full price production download, it’s still being updated and supported and there’s a community around it. I think that really matters a lot.
Peter: Absolutely. Yeah—and I mean your comment about young graduates. We have interns that we hire every year at Rocket Software who are still in university who were not even aware that the mainframe still existed, and we put them immediately to work, working on mainframe applications because we’re able to give them an environment which is completely familiar to them. They’ve used PuTTY to login to a UNIX which happens to be running on z/OS. They can issue their Git commands from the command line and BASH that they’re familiar with. Then we can give them a VS Code to do their editing of their programs in an IDE that they’re familiar with. So they’re up and running right away, being productive on the mainframe, thanks to Zowe and these ports.
Reg: Now are there some other aspects of the open-source experience that you’ve been sort of right in the middle of providing that you’d like to sort of share with people to kind of keep them aware of, including maybe some things that you’re looking forward to?
Peter: Um I’m not really sure what you mean.
Reg: You know just as the person who is really on the spot for Rocket for thinking about the future of open-source, because it’s so interesting. I mean open-source almost can’t exist without being balanced by closed-source, you might almost say. You know that you have to have some proprietary stuff, but there’s sort of like you know as little as possible in some ways, but on the other hand you need support as trustworthy as possible, and so this is a really interesting combination. On the one hand you can go pure play Ubuntu and just have everything, but the hardware—basically open-source or you can go pure play, you know Apple and IBM traditional models where everything has to be controlled and tightly controlled. Somewhere in there is sort of the aspirational space where the mainframe is. It takes on its role as the computer of the future, has to really be innovating, not merely discovering but innovating ways to do open-source in this really carefully balanced way, and I’m going to guess you’ve got some interesting observation and thoughts about how that comes to be and what you can do to make it happen.
Peter: So I’ve never actually thought about that. I’m very much reactive in terms of I am asked by our customers if we really need this or that open source ported. And our team, the ones that I work on, really aren’t working on proprietary commercial code. We don’t have time to think about it. We’re got our backlog of open source that needs to be ported and new features added to Zowe. I don’t really think about the proprietary code space at all.
Reg: And that’s fair, you know. I think you point out something really important about the mainframe that I’ve always discovered is distinct about the platform. We don’t do arbitrary bells and whistles on the mainframe, we do business value. We don’t come up with a new feature just in case somebody will like it because we’re held down by legacy, you know that we are so respectful of the past that when we create something new, you’re creating more past to take care of. So I think the lesson I take from you in this answer is this is about business value. This isn’t about eye in the sky ideas of what if we all did this and then try it out and see if it works? We’re talking about something that already works. Customers say, but it will work better if I did this. Maybe if you have some thoughts about some examples of the kind of thing that a big software company might not think of introducing, but when a customer said we need to do it, it’s something that’s right, of course.
Peter: Well I mean certainly in the area of security. That’s something that the mainframe continues to lead on. Like quantum security, for example.
Reg: Hmm right.
Peter: I mean the next release of z/OS is already going to be supporting quantum safe encryption and those kinds of areas. They’re very much at the forefront of keeping current.
Reg: Cool. Now I’m trying to think if there’s anything else we could address just from the perspective of both Rocket and open-source and all these things here. I’m just wondering is there anything else you sort of wanted to make sure we didn’t forget to talk about while we’ve got you on this interview?
Peter: Well I guess the most important thing I think that our customers need to be aware of is to stay current with their software. If you’re putting this open source into production, you really need to put in place the infrastructure to keep it current, because that’s that’s what keeps you safe.
Reg: Okay, that is really important advice. You know that is so important. People think open source is a license to be lazy, and it’s precisely the opposite. Vigilance—
Peter: Oh no, not at all [laughs].
Reg: Yeah.
Peter: Yeah, no. It’s a huge money-saver in terms of ramping up new staff, and obviously you’re not paying anything for the license for the software, only for support. But I think the largest savings is in getting people up to speed, because they don’t have to learn new software, and if you hire somebody from the outside—because the whole industry is using the same tool stack when it comes to software development and DevOps of course. I mean it’s different than the area—for example, databases and the lot.
Reg: Right. Okay. Maybe if I could ask you about a call to action. You know if somebody who is involved the industry, maybe they’re early in their career and thinking of what to do in their career. Maybe they’re a director or an architect or something and having listened to this, you know they’re thinking okay, what can I do to be taking part in this in a way that helps my organization or helps my career? What advice can you give them? What action should people be taking based on having listened to this conversation?
Peter: Yeah and I’m glad you asked that, because we get a lot of customers coming to us saying, I know that I need to modernize my DevOps pipeline, but I don’t know where to start. Or my executive told me that I have to modernize my DevOps tool chain, or I have to eliminate the DevOps stack that I have on the mainframe and do all the builds of my mainframe apps with this other tool chain that I know nothing about. How do I start? What we found through many years of experience is what doesn’t work and what does work. And what doesn’t work is to give the problem to the z/OS application development team to solve on their own, and it doesn’t work to give it to the open systems DevOps team and have them do it, because neither one knows about the other’s world. You really have to have a team put together of individuals from both sides who have an open mind. So you have the UNIX person who is familiar with Git and Ansible and CURL and Jenkins on the open systems side, and put that person in the same room with a z/OS person familiar with 3270 emulators and z/OS and MVS, and make them sit in a room together for many sessions until they understand each other’s world enough, and then they can work together on the problem. It’s got to be a team with elements from both sides to be successful.
Reg: That is so incredibly important, and it gets into a little bit of a human resources kind of thing. How do you get people who are oil and water to each other to mix together and make something wonderful? Are there any approaches you’ve seen that are especially useful for throwing these people into the same room and have live people come out afterwards?
Peter: I don’t know. Myers-Briggs tests [laughs]? You’ve got to have people that are open-minded.
Reg: Hmm. Good point. All right. So, Peter, this has been excellent. I don’t want to miss any other opportunity. If there’s anything else you want to share just for us to be keeping in mind.
Peter: I think that pretty much covered my world. No, I think that’s about it.
Reg: Okay well thank you so much, Peter. I’ve really enjoyed this and gotten a lot out of it. So I’ll be back with another podcast next month, but in the meantime check out the other content on TechChannel. You can also subscribe to their weekly newsletters, webinars, e-books, Solutions Directory and more on the subscription page. I’m Reg Harbeck.