Integrate IBM i With Anything Using Apache Camel
Jesse Gorzinski on one of his favorite integration tools, Apache Camel (also known as the "Swiss knife of integration")
As I mentioned in my previous blog post, integration of IBM i with open-source software is key for digital transformation. Today, I’d like to discuss one of my favorite integration tools: Apache Camel. Camel calls itself the “Swiss knife of integration.”
As we all know, a homogenous technology landscape does not exist. Solutions often consist of various components, each with their own APIs, databases and methodologies. Perhaps a solution runs on multiple OSes, or maybe it needs to interact with an external database or social media platform. Need to send email? Create SMS messages? Interact with Amazon S3 buckets? Send a file to a remote server? Blockchain? Naturally, all of these things require different protocols.
Thankfully, there are open-source packages that can understand all of these different protocols. So, it might make sense to use a high-level language to tie these technologies together.
Camel, though, can simplify these activities even further. So, let’s learn about it!
What Is Camel?
Camel is an Enterprise Integration framework, based on Enterprise Integration Patterns (EIPs) published by the Apache Software Foundation. These patterns represent common tasks needed when wiring various technologies together. An example EIP is the content-based router, illustrated in Figure 1 (image credit):
With a content-based router, the contents or attribute of a message determine where a message is sent or how it is processed. Camel makes it easy to apply these EIPs. For simple routes, though, EIPs may not be needed—just some “plumbing” is enough.
Before we get much further, I should clarify that Camel is a Java-based framework. There are many ways that Camel can be deployed. It can be launched as a Java web app using Spring (a web application framework). It can run natively on Kubernetes, and it’s available as a Quarkus extension.
In my opinion, though, the simplest way to deploy Camel is as a standalone Java application. This allows you to define your route with simple Java code and doesn’t require knowledge of Spring, Quarkus or Kubernetes. Plus, a standalone Java application can run virtually anywhere (including on IBM i, of course).
How Does It Work?
Camel is a form of Message-oriented Middleware (MoM). That is, it abstracts units of work into basic messages. These messages have a body and zero or more message headers. To integrate different technologies, one simply needs to define how the messages are passed between the necessary components. This is called a Camel route.
Elements of a route can be any of Camel’s many components. See the Camel components documentation for an exhaustive list of what components there are—warning, there are a lot! There’s even a component for IBM i, but a lot of platform-agnostic components (JDBC, file access, commands, etc) also work on IBM i.
A sample fictional route could perhaps be visually represented in Figure 2:
In this example, we have four endpoints
- Consume data: This could be the result of a Db2 query, data read from a data queue or message queue, the output of a command, etc.
- Process data: This could be an AI activity (yes, Camel can talk to IBM Watson!), a simple transformation, a REST API call, etc.
- Log activity: Maybe you need to write to a log file or see visual feedback in your terminal
- Send data somewhere: Twitter? Another database? Slack? Email? An FTP/sftp site? A Blockchain node?
Routes don’t have to be linear, but there’s a lot you can do with even a simple two-endpoint route.
Take these two routes in Figure 3, for instance:
As you can see, routes are typically defined via a Uniform Resource Identifier (URI), a string that uniquely identifies the resource. The URI contains the protocol (which identifies the component), a path and zero or more query parameters. And don’t worry, Camel has techniques (like Jasypt) for encrypting passwords. If you’re running Camel on IBM i, you can use “*CURRENT” to avoid needing to manage credentials altogether.
In the above example, each of these two routes consumes data from an IoT device and sends the reading to an IBM i message queue (see Figure 4). Now, we have light and temperature readings that I can process from any language on IBM i.
Of course, the data could have been routed into a database (using the JDBC component), stream file or data queue by just specifying a different target endpoint.
Want to control an IoT device from IBM i? Simply reverse the order of the route! Then, you can route messages from an IBM i message queue to the device. For example, the following SNDMSG command in Figure 5 can turn on a light bulb:
As I mentioned, Camel routes don’t have to be simple linear routes, since you can utilize EIPs within your route. You can perform asynchronous tasks with Wire Tap, where Camel will automatically manage a thread pool. Or, you can make a decision with a Content Based Router as mentioned earlier.
How Do I Get Started?
The best way to get started might be to look at some of the examples published in the IBM i OSS Examples repositoryon GitHub, more specifically the Camel directory. Examples currently include:
- A data queue to Kafka bridge (with information about using this as a Db2->Kafka bridge)
- A message queue to email bridge
- Disk monitor example that sends you an email when your disk usage exceeds a threshold
- Disk monitor example that sends messages to *SYSOPR message queue
The disk monitor examples demonstrate how to use several EIPs, including:
As a heads up, most standalone Java projects make use of Maven to download and manage dependencies. A project’s dependencies (various Camel components, for instance) are declared in a central file (pom.xml) for Maven to process. As you work through the examples, you will learn the basic maven commands needed to compile and run a Maven application.
There’s also a newly-created prototyping tool, Kameleon, found at https://kameleon.dev. With this tool, simply choose the Camel version, Java version and some simple project attributes. Select which Camel components you will be using, and Kameleon will generate a simple boilerplate project. All you need to do is fill in your route definitions with custom logic (see Figure 6)
Running Standalone Java Routes As a Service
There are several ways to launch Java routes as a persistently running background job on IBM i. You can use SBMJOB and QSH to submit to batch (a common technique). Or, from an SSH session, you can install coreutils-gnu and use the “nohup” command (for instance, “nohup mvn exec:java”). Perhaps a better way is to use the Service Commander utility, which has integration with STRTCPSVR/ENDTCPSVR.
How Can I Get Help?
With open source, you’re never alone on projects. There are several channels available where you can find help. For one, you can join the community forums on Ryver (join here first). Apache Camel also has a quite active community, and support channels are documented on the Camel website here. I recommend Camel on Zulip Chat, and commercial support is also available, including IBM i open-source support.
In any event, I hope you find Camel to be as helpful a tool as I do. At a minimum, it’s a great addition to your skillset, enabling you to accomplish integration activities swiftly. As an example, I was chatting with Charlie Guarino about Twitter a couple months ago. 20 minutes of live-coding on a Zoom call, and I was sending tweets from IBM i!
In a nutshell, Camel is a robust way to connect various pieces of your enterprise solution. What might you use it for? Share your thoughts in our community chat!