Enabling Applications to Use Large Memory Pages

In the second part of this series on large memory pages, AIX expert Mark Ray walks readers through how to enable applications to use them

This is part 2 of a series on large memory pages. Part 1 is available here. This section will deal with enabling applications to utilize large memory pages. But before we dive into that topic, we need to be aware of a very big caveat. Enabling large pages on your application is nothing like doing it for databases from one key aspect: support. As we’ve already seen, most database systems support large pages out of the box and have for many years. 

Applications are a different story. There are probably as many different types of applications out there as there are words in this article. It’s impossible to make a blanket statement and say yes, this app will benefit from large pages or no, that app will not. 

When it comes to applications, it all depends on the code. A few years ago, I wrote an article about the Feedback Directed Program Restructuring (FDPR) tool. The FDPR is a utility that lets you optimize already-compiled programming code. It’s a truly nifty tool and can breathe new life into tired, old applications. But the FDPR can also wreak havoc with code that can’t handle what the utility does. I’m bringing this up because the warning I issued about the FDPR applies here: Just because you can enable large pages for an application doesn’t mean you shouldrun right out and do so. 

You need to check with your application vendor to be absolutely certain the application in question will benefit from large pages. There are many instances where an app’s performance can degrade to the point of un-usability if forced to use large pages. So take this caveat to heart, and make very sure your application vendor has blessed the use of large pages. Doing this could save you headaches and embarrassment. 

Large Pages for Middleware, Utilities and Homegrown Applications

OK, so what about applications? How about middleware or utilities? What about programs that have been home grown yet might benefit from the use of large pages? In these cases, you have a couple of choices. 
The first is an environmental variable that cuts a broad swath to enable process’ use of large pages. The second method lets you target specific executables for large page use. Each method has its benefits and drawbacks, so the main takeaway from this part of our discussion will be that you need to know in advance which enablement method will work best with your code, and indeed, if it will work with large pages at all. Let’s talk firstly about a thing called “Loader Control.”

Loader Control

Loader Control is an environmental variable that when set properly instructs any applications that are started with it to use large memory pages for their data and heap segments. 

The loader control variable is declared as “LDR_CNTRL.”  Being a variable, it can be declared either in the current shell or in startup scripts. Think of it like any other variable in terms of implementation using the “export” function. Say you want to make sure you use the vt102 terminal definition for all of your command line work in AIX. You can use the export function to enable vt102 in your current shell, place it in a user’s .profile, or in the system profile. The same goes for loader control. 

To tell any application to use large pages for its data and heap segments, you invoke the loader control variable like this:

exportLDR_CNTRL=LARGE_PAGE_DATA=Y

I tend to shy away from telling folks to use loader control from within /etc/profile. Doing so will tell every piece of code to use large pages on system startup. You can easily make your system nonbootable by exhausting memory during the startup process. So after you’ve configured your large page pool, think carefully about which processes you want to use large pages and configure their use selectively.

You can check your implementation easily by using the “env”command; typing it at a command prompt will yield output that delineates all of the environmental settings in your system. Included in this output will be a line that looks exactly like your loader control declaration, minus the export.

In most cases, undoing your loader control setting is easy: just use the “unset” function, after stopping the process(s) that use it. For example, if you’ve been using loader control in a shell environment and want to reverse it, you can make this declaration:

unsetLDR_CNTRL

Now, your memory page scheme for your current shell will revert to the default 4K/64K scheme. However, be aware that this may not apply to all processes. While unset will work 90% of the time, there may be a case where you’ll need to bounce a process to undo a large page allocation.

Note that loader control can be used to instruct a program to use medium as well as large memory pages. And you can get pretty granular in this deployment. For example, you can instruct loader control to use medium pages for an executable’s data,text, stack and shared memory segments, or any combination of those four. The invocation for this purpose might look like this:

export LDR_CNTRL=DATAPSIZE=64K@TEXTPSIZE=64K@SHMPSIZE=64K <executable>

We could also omit the <executable> and just have loader control tell every process that’s invoked from the current shell to use medium pages—or put the declaration in startup scripts. These days, AIX does a pretty good job of determining when to use the medium or small memory pages for any given process, so the use of loader control for this purpose isn’t as widespread as it was many years ago. But especially in the case of home-grown code, give it a try – on a test system of course—and see how your code responds.

ldedit      

Now let’s move on to our next choice for large page deployment, outside of a database system. We’re going to consider a utility called “ldedit.” Whereas the loader control environment variable is good for operating on many executables at the same time, ldedit lets you mark individual executables as having the ability to use large pages. Consider loader control as a blunt instrument versus the surgical precision of ldedit. “Marking” an executable to use large pages with ldedit is simple. You would use it like this:

ldedit -blpdata <filename>

Where “-b” is the option flag for ldedit and lpdata simply stands for “large page data,” and <filename> is the program you want to be able to use large pages. And as with loader control, you can undo your change easily like this:

ldedit -bnolpdata <filename>

There are many ways to use ldedit; this is only one. And like loader control, ldedit can be used to mark individual executables to use 64K pages exclusively, 4K
pages exclusively or some combination of the two. For complete information, read more about ldedit in the IBM Knowledge Center. 
 

Supreme Pages      

We close our discussion of memory and tuning same in AIX with supreme pages. Supreme pages are 16GB in size and are used only is hyper-specialized, high performance computing environments. In my own 20-year career as a performance specialist, I’ve only seen them deployed twice, but no article about memory would be complete without them.

Supreme memory pages in AIX used to be called huge pages. A few years ago, some Linux distributions (notably Red Hat) developed “huge” memory pages of their own. These were memory blocks of either 2MB or 1GB in size; contrast this with AIX’s “huge” page definition of 16GB. You can see that in a mixed OS environment where you might have huge pages deployed in both Linux and AIX, monitoring their function could quickly become problematic in both oral and written reports. So what probably happened was the AIX developers needed a way to distinguish their huge pages from Linux huge pages. So they changed the name to “supreme.”

Whatever the name, the biggest memory page you can configure in AIX is 16GB in size. Supreme pages can only be configured from an HMC. Briefly, you bring up an LPAR’s partition profile and click on the “memory” tab. You then assign however many supreme pages you wish in that tab. Initially, you configure supreme pages while a new system is powered off. For existing systems, you follow the change procedure of adjusting the supreme page value then shutting down and restarting the partition. Monitoring supreme page usage in your system is done with SVMON and vmstat, with invocations similar to that using large pages; just substitute “16GB” in your page size declarations.

A second reason I’m only giving you a bare bones tour of supreme pages in this article is there are a number of considerations you must take into account to deploy them successfully, which are well outside of this article’s scope. You must calculate factors like the total number of I/O connections your server will have, the base memory configuration of your partitions, the fact that supreme pages require contiguous memory and each page must start on a 16GB boundary.

And, as if that weren’t enough, supreme pages cannot span processing units. Supreme pages work in systems that require massive parallelism; think of an insurance company that processes millions of queries per minute. All in all, I very much doubt any of you reading this will be working at a site that has the amount of memory necessary to make supreme pages a wise choice to begin with or has the parallelism requirements to make them worthwhile. That said, 99% of you will find the configuration and deployment of large memory pages quite adequate for your needs.

So there you have it: the Cook’s tour of memory in AIX and how it and databases, applications and any other piece of code can be tuned to use it effectively. I hope you’ve found this discussion useful, and as always, let me hear about your own performance projects via the comments section.

---