Skip to main content

Accessing the Data in Core Dumps

If you're like most UNIX administrators, you probably have a crontab or some other housekeeping program that regularly searches your systems for core dumps, backs them up (maybe), and then deletes them.

This leaves you to try to puzzle out their cause from information contained in the AIX error log. Of course, about half the time, core dumps aren't flagged, so other methods must be used to discern their cause, such as examining them with a utility like strings or some other reader.

Chances are, that's as far as you'll get. You'll dutifully inform your application developers or DBAs about the location of the core dump and when it occurred, and then move on to other duties. Meanwhile, the core dump will sit in its archive, forgotten about. And that's unfortunate.

Core dumps contain a snapshot of the memory address space for an application — or a database, utility or middleware — at the time of the event that caused the core dump to happen. That's a wealth of data that, if diagnosed, could help you avoid future core dumps. So how do you access this data? It can actually be done quickly and fairly easily.

Seriously, this information can be gathered in about 10 minutes. By entering a few commands, you'll not only know which program caused the core dump, but also the file or function that lies at fault and the working thread that was interrupted.

First, you need the proper tool to examine your core dump. The dbx utility is a standard debugger that ships with AIX; it's included on the operating system distribution DVDs. Part of the bos.adt.debug fileset, dbx is usually installed by default. It has many uses, from providing a controlled environment for running programs to diagnosing misbehaving processes live and giving you the ability to step through a program one line at a time. And yes, dbx can be used to examine core dumps. Here's how: To start your examination, change your working directory to its location and call up the debugger on the core file:

 dbx -C ./core

Your output will be similar to this:

[lpar:/home/raym] dbx -C ./core
Type 'help' for help.
warning: Object file is not specified, and the object file "perl5.10.1" mentioned in
 the core file doesn't exist in the current directory or doesn't match with
 the core file (ignored). Some info may not be available.
[using memory image in ./core]
reading symbolic information ...warning: cannot open perl5.10.1
warning: no source compiled with -g
Segmentation fault in extend_brk at 0xd011119c ($t1)
0xd011119c (extend_brk+0x2dc) 90040004            stw   r0,0x4(r4)

In this example, I'm using a core dump that was generated on a system I was working on that was using a 32-bit version of Perl and had recently been updated to AIX 7.1. This initial screen above is basically a summary of the conditions in memory at the time of the core dump. Note the object file name in line 3; it will likely point to the executable that caused the core dump. If you're only looking to identify the program that caused the dump, most of the time, this screen will give you your answer. However, this information can be a bit nebulous, so it's often worth digging deeper.

The last line of the output (dbx) is the dbx utility's command prompt. This is where you'll enter all of your commands. To determine which program caused the dump, enter the corefile command at the (dbx) prompt:

(dbx) corefile
 Process Name:  perl5.10.1
 Version:       430
 Signal:        SEGV
 Process Mode:  32 bit

At this point you'll be presented with several lines of information, including the name of the process that core dumped, the signal that caused the core dump — in this case a segmentation violation (SEGV pr); most core dumps are generated by this signal — and whether the process was executing in 32- or 64-bit mode. The identified process will, of course, point back to the executable having problems. But where in the programming code of that executable did the fault happen? To determine this, issue the dump command:

(dbx) dump
extend_brk(internal error: assertion failed at line 3915 in file frame.c
??, internal error: assertion failed at line 3915 in file frame.c
??, internal error: assertion failed at line 3915 in file frame.c
??) at 0xd011119c

In this case, we have a failed assert called from the frame.c file. An assert is simply a reaction to a Boolean expression that should always be true; a "false" reaction will generate a fault, causing the core dump and the executable to fail. Sometimes dbx points to a location in a specific file; sometimes it points to an offending function.

So with just two commands, you've pinpointed the program that faulted as well as the area of programming code that caused the fault. Now you're ready to fix the code. (In this case, as noted, we were running a 32-bit version of Perl on a system that had recently been upgraded to AIX v7.1. When we upgraded to a 64-bit version of Perl, our problems went away.)

Most of the time, running corefile and dump is sufficient for starting remediation of the problem that caused the core dump. However, if you work with application developers who require more information to write a patch or an upgrade, you should be aware of several other dbx commands that tell you about the status of the working thread that encountered the error:

(dbx) th -
 thread kernel                   user                         state  flag  held
        tid      pri sched state tid      pri nice sched state
        mode   scope    cancellation    join- boosted cursig wchan      function
                        pending state   able

>$t1    0x2ea02af 112 other run   0x000001 1   58   other running   0x0002 0x000 no
        kernel system   no      ed      yes   0       0

Entering th – at a dbx prompt tells you a lot about the thread's state at the time of the fault, and proc gives you a full dump of the process's memory usage. Enter coremap and you'll see all of the libraries the process had loaded as well as whether they were text or data segment. Likewise, the word map by itself gives you verbose information on all of the object files in use by your process.

Analyzing core dumps with the dbx utility provides a comprehensive picture of their causes as well as clear options for dealing with them. Details about individual core dumps can be packaged in an easily understood form and presented to your vendor, the appropriate support personal and even to non-technical personnel in your management chain.

Perhaps best of all, by using the dbx utility, in most instances you'll be able to diagnose and remediate the conditions that create core dumps with any outside help. This should lead to more reliable and stable AIX systems.