Manage IBM i Jobs With Service Commander

IBM i can run more technology than ever before. As such, the types of jobs that can be run are quite diverse—ILE programs, Db2 tasks and PASE programs can happily coexist. This synergy is indeed powerful, but it does come with a cost. All these jobs can be difficult to manage!

Consider the simple notion of starting a job, for instance. A variety of different practices are used. For instance:

• Node.js jobs are started with the ‘node’ command in PASE.
• Java jobs are launched with the ‘java’ command in PASE, or the ‘java’ command in QSH, or with the RUNJVA CL command.
• Apache Tomcat servers have a ‘startup.sh’ shell script, and WildFly servers have a ‘standalone.sh’
• IBM i host servers are started with the STRHOSTSVR CL command.
• IBM i TCP servers are started with the STRTCPSVR CL command.
• IBM HTTP Server jobs are started with the STRHTTPSVR CL command.

… and the list goes on, especially when you factor in the many programming languages and turnkey solutions that are available. But the complexity doesn’t stop there. How do you know which ones are running? How do you leverage subsystems, capping groups and memory pools?

Back in 2020, I experienced this complexity firsthand. At the time, I was managing an IBM i demo system and had installed a bountiful set of open-source packages, including Apache Kafka (and a web-based visualizer), Apache Zookeeper, ActiveMQ, GitBucket, MariaDB, as well as several Python and Node.js programs. I found it tedious to manage all these jobs. I kept my own notes on where each thing was installed, how to start it and how to check on it. Eventually, though, I realized that my text file was unwieldy and inefficient, so I created Service Commander for IBM i!

Service Commander’s Features

The primary value of Service Commander is to provide a unified interface for managing all your IBM i services. The project documentation lists the following features:

• Start or stop services in a unified fashion
• Check the “liveliness” of your service by either port status or job name
• Customize the runtime environment variables of your job
• Submit jobs to batch easily, even with custom batch settings (use your own job description or submit as another user, for instance)
• Specify dependencies (for instance, if one application or service depends on another); Service Commander will start any dependencies as needed
• Define custom groups for your services and perform operations on those groups (by default, a group of “all” is defined)
• Query basic performance attributes of the services
• Provide some (limited) assistance in providing/managing log files

Service Commander is delivered as a command line tool. Preferably, it is run from within an SSH terminal, but QSH can work in a pinch.

Configuring Services

Each “service” that Service Commander is defined in a YAML-formatted file. That service’s short name (used with the ‘sc’ commands) is simply derived from the name of the YAML file. There are only two required fields: a start command and a “check alive” criteria. The “check alive” is either a job name or a port—and is what Service Commander uses to report status of the service.

The following basic example is a Node.js service that runs on port 8444. Here, I opted to give it a starting directory and a descriptive name as well.

Figure 1: A simple service definition for a Node.js app

Several other configuration options are available, too. In the following example, a Node.js application is submitted to batch under the job name “IBMIDASH” and is part of the “web_apps” group (since Service Commander can start/stop groups of services).

Figure 2: A Node.js application submitted to batch

Another common need is to run a job as a specific user. An example configuration for the PostgreSQL server shows how easy this can be. In fact, you can specify any options for the SBMJOB command that you’d like, allowing you to easily exploit subsystems, memory pools and capping groups!

Figure 3: Submitting a job as another user

However, most of the time, you don’t need to create or edit YAML files. Service Commander ships with an ‘scedit’ tool. This utility will ask you a few basic questions and create the configuration for you.

Figure 4: Using ‘scinit’ to create a service definition

Managing Services

Once services are configured, basic functions are available from the ‘sc’ command line. To check which ones are running, I can use ‘sc check’:

Figure 5: Checking which services are running

In the previous screenshot, you’ll see that one of them has a short name of ‘mqtt_db2.’ I can start it by running ‘sc start mqtt_db2’:

Figure 6: Starting a service

This particular demo program uses ActiveMQ server, and ‘sc’ knows to start the requisite service first. Running ‘sc check’ again shows both are now running:

Figure 7: Checking which services are running (again)

Also worth noting: Service Commander comes with knowledge of numerous IBM-provided services “out of the box” and they can be managed by adding the ‘-a’ flag:

Figure 8: Checking on IBM-provided services

Ad Hoc Services

Even without configuring specific services, Service Commander’s operations can be used on an “ad hoc service.” This allows for easy interrogation and management of jobs based on port or job name. The service name for an ad hoc service looks like one of the following:

• port:nnnn
• job:jobname
• job:subsystem/jobname

So, for instance, if you want to see which job is running on port 110, you could simply run ‘sc jobinfo port:110’

Figure 9: Getting job information for port 110

Want to see that job’s performance information? Just run ‘sc perfinfo port:110’

Figure 10: Getting performance information for the job running on port 110

Want to kill it? ‘sc stop port:110’ does the trick!

Figure 11: Killing the job using port 110

Even further, the tool will implicitly look for services matching the provided port number or job. Imagine a scenario where an application is failing to connect to port 9092. Someone needs to start that service. Sometimes, though, a system administrator might not even know what’s supposed to be running on port 9092! Plus, how is that service even started? Good news: If Service Commander knows about this service, the job is easy! Just run ‘sc start port:9092’!

Figure 12: Starting a service knowing only the port number

Checking Open Ports

System administrators or security specialists are often interested in what ports are “open” on the server. Thankfully, Service Commander comes with a hidden gem: the ‘scopenports’ utility. One quick command and you can see all the open ports and what’s running:

Figure 13: Listing all open ports

Sometimes, you might only care about your own jobs, so the tool has a ‘–mine’ option as well:
Figure 14: Listing open ports for the current user only

As you may have noticed, Service Commander is only able to tell you the details of services that have been configured. In the above screenshot, for instance, it knows what’s running on port 9233 because I have configured a service named “grafana” on that port. However, it doesn’t know what’s running on port 12082. I could use the ‘port:12082’ ad hoc definition to investigate further.

What’s Next?

If today’s post has convinced you that Service Commander could be useful to you, please stay tuned for my next post. There, I will discuss more advanced Service Commander topics, such as:

• Leveraging Service Commander to automatically start jobs at IPL
• Using Service Commander’s “cluster mode” to add resiliency and scalability to your application
• How Service Commander can help with application performance analysis
• Comparing Service Commander to the popular PM2 tool

In the meantime, please wander over to the project documentation for more information. And of course, this tool is open source, so feel free to send issues, comments or contributions!

---