10 Hidden Gems in the RPM Pile
From JSON Query Utility, to LFTP, to Service Commander and more, Jesse Gorzinski highlights 10 hidden open-source gems in the RPM pile
Today, I am counting down some hidden gems, nestled within the pile of goodness. As with most open source, these tools will work best when running bash in an SSH session. If you're new to this, check out the IBM i open source documentation (I recommend you use bash and set up your environment properly too).
I hope you find one or two of these hidden gems useful. Note that I include the RPM package names so you can easily find them.
Db2 for i comes with built-in JSON support, but sometimes it's easier to work with JSON in a non-database context. As it turns out, the RPM pile has a command-line tool that can inspect, sort, filter, and otherwise transform JSON data. This tool is called “jq” (short for "JSON Query").
In Figure 1, I call a public API with the “curl” command. This particular service returns lots of information about the commit history of a repository. But, let's say I only care about the most recent commit message and author. Using the “jq” command, I can extract the first entry, filter out the unwanted data, and transform it into a desired format.
Figure 1: Using “jq” to query, filter and format JSON
Here's a trick I learned from Luciano Mammino on Twitter: using “jq” to sort JSON before comparing two JSON data sets (see Figure 2)! Pretty neat!
Figure 2: A handy tip for comparing JSON
Interested? Feel free to check out the jq project site for more information.
9. Hex Dump Utility (Package Name: 'util-linux')Have you ever asked yourself these questions about a file's contents? For instance:
- Is the data EBCDIC or ASCII?
- Are those tabs or spaces?
- Did some whitespace or badly-converted Unicode character (like Microsoft Word's "smart quotes") get in there?
- Did my editor save this UTF8 file with a byte-order mark (BOM)?
Figure 3: The hexdump tool with a side-by-side display
8. LogRotate (Package Name: 'logrotate')
Log files are everywhere, and that's a good thing! After all, logs are often our first stop when we need to check an application's activity or diagnose problems. However, they can get out of hand. Sometimes they can take up many megabytes (or gigabytes!) of disk space. How can you keep them in check?The logrotate tool can be used to automatically rotate, compress and even remove your log files. The HTTP Server for i can do this with log cycling, but not all products have such capability. So, having a specialized tool can be useful.
It’s very configurable, allowing you to customize the name format of archived logs, how many logs to keep or whether the archived logs should be compressed. Note that this is not a service that runs continuously. Instead you must run it periodically (or via the job scheduler). It will then analyze all the log files which it is configured to rotate and rotate the files as needed. See the project website for more info. IBM Champion Andy Youens has also written a splendid article about this handy utility!
7. Line Ending Conversion Tools (Package Name: 'dos2unix')
Of all the silly things in technology, the one that irritates me the most is differences in line endings! Windows uses a two-character sequence ("carriage return and line feed" or "CRLF") for indicating a new line in a file. Most other platforms, including PASE on IBM i, use a single character ("line feed" or "LF"). This can cause lots of problems! For instance, if I'm using a Windows-based text editor, the file might be saved in the Windows format if I'm not careful. That doesn't always work well. For instance, shell scripts might error out with "bad interpreter" or "file not found" messages. A simple “dos2unix” can fix it right up (see Figure 4)! There's also a “unix2dos” for the other direction.Figure 4: Converting line endings to UNIX format
6. LFTP (Package Name: 'lftp')
Have you ever needed a command-line FTP client for the PASE environment? I sure have! Many people are familiar with the popular 'ftp' command, but the RPM pile gives us something better: lftp!The “lftp” command has far too many features to list here, but some of my favorites are:
- Tab completion for remote file names
- Bash-like ability to run transfers in the background
- Support for multiple protocols, including ftp, sftp(ssh), and HTTP
- Transfer rate throttling
- Ability to download create a full mirror of a remote site with the 'mirror' directive
5. Bash Built-ins (Package Name: 'bash')The venerable bash shell is a mainstay for most people working on open source. Bash and other shells are fundamentally different from the classic CL command line: they always run commands in a child process, rather than in the current job. For those familiar with the single-job paradigm, this can cause confusion. For instance, the following sequence of commands in Figure 5 may seem nonsensical.
Figure 5: Trying to modify the library list with the PASE “system” utility
The "addlible" command says it added the library to the library list, but "dsplibl" doesn't reflect that. Why? Because bash created a child job where the "addlible" actually ran! So, the library list of the original job remains unchanged.
To help with this type of conundrum, IBM has created three bash built-ins:
- “cl” which runs a CL command in the current job
- “liblist” which modifies or displays the library list of the current job
- “getjobid” which displays the job ID of the current job
Figure 6: Modifying the library list with the “liblist” built-in
Similarly, one could use the “cl” built-in to run CL commands (ADDLIBLE, CHGJOB, ec) in the current job. Check out Kevin Adler's blog on the topic for more info.
4. Terminal MUltipleXer (Package Name: 'tmux')The “tmux” tool provides a lot of power over your SSH sessions. For one, you can close your SSH connection without killing your session, so you can reconnect and continue work later. You can also split up your terminal into multiple windows (see Figure 7). It's a very powerful tool if you spend a lot of time in SSH sessions!
Figure 7. Terminal MUltipleXer with multiple windows
The above screenshot was taken from Kevin's Ramblings.
3. Telnet 5250 Client (Package Name: 'tn5250')Thanks to the “tn5250” tool, you can launch 5250 from within your SSH session. This could be handy if you can't, or don't want to, launch a separate emulator program like Access Client Solutions. Why would you need 5250? I have no idea. But hey, if you do, we've got you covered (see Figure 8).
Figure 8. 5250 emulation in an SSH terminal
2. NCurses Disk Usage (Package Name: 'ncdu')Have you ever wondered what's taking up your disk space? The NCDU tool can provide an interactive and powerful interface for visualizing your disk usage. Just point it at any directory and see what's taking up space! Like other tools, you can drill down into the individual directories as well (see Figure 9).
Figure 9. Analyzing IFS disk usage
And yes, you can use it against /qsys.lib (see Figure 10).
Figure 10. Analyzing disk usage within QSYS.LIB
1. Service Commander (Package Name: 'service-commander')Many types of workloads run on IBM i, but there are different techniques that must be used to manage them. For some things, STRTCPSVR does the trick. For Node.js applications, go to a shell and run the “node” command. Plus, open-source workloads could be deployed anywhere in the filesystem. And how do you know what's running? NETSTAT? WRKACTJOB? SQL? It can be overwhelming to manage all these jobs!
Wouldn't it be great if there was a single tool that could manage all these different kinds of jobs? That's why I created the Service Commander tool. With a single tool, you can:
- Start or stop any type of service
- Check which services are running
- Gather performance metrics on running services
- See which ports are currently open
- See what jobs a service is running in, or which job is listening on a certain port
More information, as well as a demo video and hands-on exercise can be found here.
Figure 11. Checking the status of several services
What Else?There you have it: ten "hidden gems" in open source. If you can make use of something in today's article, great! If you know of something I missed, even better. Share your favorite hidden gems with TechChannel on Twitter or the social platforms listed on our IBM i open-source page. I look forward to hearing your favorite open-source gems!
Jesse Gorzinski is the business architect of open source technologies.
See more by Jesse Gorzinski