10 Hidden Gems in the RPM Pile

From JSON Query Utility, to LFTP, to Service Commander and more, Jesse Gorzinski highlights 10 hidden open-source gems in the RPM pile

As you probably know, IBM provides hundreds of open-source packages in the RPM Package Manager (RPM) format. This collection of software, which I affectionately call “the RPM pile,” has become a quintessential component of IBM i. The Node.js and Python programming languages, for instance, are now standard vernacular. Some open source, however, is not as well-known.

Today, I am counting down some hidden gems, nestled within the pile of goodness. As with most open source, these tools will work best when running bash in an SSH session. If you’re new to this, check out the IBM i open source documentation (I recommend you use bash and set up your environment properly too).

I hope you find one or two of these hidden gems useful. Note that I include the RPM package names so you can easily find them.

10. JSON Query Utility (Package Name: ‘jq’)

JavaScript Object Notation (JSON) has become a de facto standard for text-based data representation. It can be generated or processed from nearly any programming language on any platform. It’s used for storing data on disk, business-to-business (B2B) data transfer, and many web/REST APIs. In short, it’s everywhere!

Db2 for i comes with built-in JSON support, but sometimes it’s easier to work with JSON in a non-database context. As it turns out, the RPM pile has a command-line tool that can inspect, sort, filter, and otherwise transform JSON data. This tool is called “jq” (short for “JSON Query”).

In Figure 1, I call a public API with the “curl” command. This particular service returns lots of information about the commit history of a repository. But, let’s say I only care about the most recent commit message and author. Using the “jq” command, I can extract the first entry, filter out the unwanted data, and transform it into a desired format.

Here’s a trick I learned from Luciano Mammino on Twitter: using “jq” to sort JSON before comparing two JSON data sets (see Figure 2)! Pretty neat!

Interested? Feel free to check out the jq project site for more information.

9. Hex Dump Utility (Package Name: ‘util-linux’)

Have you ever asked yourself these questions about a file’s contents? For instance:

• Is the data EBCDIC or ASCII?
• Are those tabs or spaces?
• Did some whitespace or badly-converted Unicode character (like Microsoft Word’s “smart quotes”) get in there?
• Did my editor save this UTF8 file with a byte-order mark (BOM)?

I’m sure you have. Inevitably, you need to look at a file’s actual bytes at some point or another. Sure, DSPF has an option to display the bytes in hex format, but 5250 is deprecated (let me dream!). It’s a lot easier to just run the “hexdump” utility in your SSH session! The tool comes with a variety of options to customize behavior, but the most commonly-used is probably “-C” which shows the hex bytes on the left and the ASCII equivalent on the right (see Figure 3).

8. LogRotate (Package Name: ‘logrotate’)

Log files are everywhere, and that’s a good thing! After all, logs are often our first stop when we need to check an application’s activity or diagnose problems. However, they can get out of hand. Sometimes they can take up many megabytes (or gigabytes!) of disk space. How can you keep them in check?

The logrotate tool can be used to automatically rotate, compress and even remove your log files. The HTTP Server for i can do this with log cycling, but not all products have such capability. So, having a specialized tool can be useful.

It’s very configurable, allowing you to customize the name format of archived logs, how many logs to keep or whether the archived logs should be compressed. Note that this is not a service that runs continuously. Instead you must run it periodically (or via the job scheduler). It will then analyze all the log files which it is configured to rotate and rotate the files as needed. See the project website for more info. IBM Champion Andy Youens has also written a splendid article about this handy utility!

7. Line Ending Conversion Tools (Package Name: ‘dos2unix’)

Of all the silly things in technology, the one that irritates me the most is differences in line endings! Windows uses a two-character sequence (“carriage return and line feed” or “CRLF”) for indicating a new line in a file. Most other platforms, including PASE on IBM i, use a single character (“line feed” or “LF”). This can cause lots of problems! For instance, if I’m using a Windows-based text editor, the file might be saved in the Windows format if I’m not careful. That doesn’t always work well. For instance, shell scripts might error out with “bad interpreter” or “file not found” messages. A simple “dos2unix” can fix it right up (see Figure 4)! There’s also a “unix2dos” for the other direction.

6. LFTP (Package Name: ‘lftp’)

Have you ever needed a command-line FTP client for the PASE environment? I sure have! Many people are familiar with the popular ‘ftp’ command, but the RPM pile gives us something better: lftp!

The “lftp” command has far too many features to list here, but some of my favorites are:

• Tab completion for remote file names
• Bash-like ability to run transfers in the background
• Support for multiple protocols, including ftp, sftp(ssh), and HTTP
• Transfer rate throttling
• Ability to download create a full mirror of a remote site with the ‘mirror’ directive

Check out the lftp website for more details.

5. Bash Built-ins (Package Name: ‘bash’)

The venerable bash shell is a mainstay for most people working on open source. Bash and other shells are fundamentally different from the classic CL command line: they always run commands in a child process, rather than in the current job. For those familiar with the single-job paradigm, this can cause confusion. For instance, the following sequence of commands in Figure 5 may seem nonsensical.

The “addlible” command says it added the library to the library list, but “dsplibl” doesn’t reflect that. Why? Because bash created a child job where the “addlible” actually ran! So, the library list of the original job remains unchanged.

To help with this type of conundrum, IBM has created three bash built-ins:

• “cl” which runs a CL command in the current job
• “liblist” which modifies or displays the library list of the current job
• “getjobid” which displays the job ID of the current job

So, if we’re running in bash, we can modify the library list of the current job by leveraging the new builtin, as demonstrated in Figure 6.

Similarly, one could use the “cl” built-in to run CL commands (ADDLIBLE, CHGJOB, ec) in the current job. Check out Kevin Adler’s blog on the topic for more info.

4. Terminal MUltipleXer (Package Name: ‘tmux’)

The “tmux” tool provides a lot of power over your SSH sessions. For one, you can close your SSH connection without killing your session, so you can reconnect and continue work later. You can also split up your terminal into multiple windows (see Figure 7). It’s a very powerful tool if you spend a lot of time in SSH sessions!

The above screenshot was taken from Kevin’s Ramblings.

3. Telnet 5250 Client (Package Name: ‘tn5250’)

Thanks to the “tn5250” tool, you can launch 5250 from within your SSH session. This could be handy if you can’t, or don’t want to, launch a separate emulator program like Access Client Solutions. Why would you need 5250? I have no idea. But hey, if you do, we’ve got you covered (see Figure 8).

2. NCurses Disk Usage (Package Name: ‘ncdu’)

Have you ever wondered what’s taking up your disk space? The NCDU tool can provide an interactive and powerful interface for visualizing your disk usage. Just point it at any directory and see what’s taking up space! Like other tools, you can drill down into the individual directories as well (see Figure 9).

And yes, you can use it against /qsys.lib (see Figure 10).

1. Service Commander (Package Name: ‘service-commander’)

Many types of workloads run on IBM i, but there are different techniques that must be used to manage them. For some things, STRTCPSVR does the trick. For Node.js applications, go to a shell and run the “node” command. Plus, open-source workloads could be deployed anywhere in the filesystem. And how do you know what’s running? NETSTAT? WRKACTJOB? SQL? It can be overwhelming to manage all these jobs!

Wouldn’t it be great if there was a single tool that could manage all these different kinds of jobs? That’s why I created the Service Commander tool. With a single tool, you can:

• Start or stop any type of service
• Check which services are running
• Gather performance metrics on running services
• See which ports are currently open
• See what jobs a service is running in, or which job is listening on a certain port

See Figure 11 for a visual. Service Commander can be invoked from a shell, but it also has integration with STRTCPSVR. So, a single command can start, stop, or monitor jobs, whether you’re interested in IBM i host servers, Jenkins, TomCat, Python programs, Apache Camel or all the above.

More information, as well as a demo video and hands-on exercise can be found here.

What Else?

There you have it: ten “hidden gems” in open source. If you can make use of something in today’s article, great! If you know of something I missed, even better. Share your favorite hidden gems with TechChannel on Twitter or the social platforms listed on our IBM i open-source page. I look forward to hearing your favorite open-source gems!

---