Basic AIX Security Strategies and Best Practices

Security is a very hot topic for a good reason. So much of our personally identifying data is now being stored that the security break-ins that have been happening have most likely affected everyone reading this article. Additionally, the penalties now for breaches of the various standards (HIPAA, PCI, etc) are significant. Good security requires a multi-layered approach that starts with people, then physical security, and then the various layers. It’s critical to look at the whole environment and see how security can be applied at each level.

In this article we will cover some of the basics of locking down your LPARs. This isn’t done by default, but it’s fairly easy to do. It ranges from default permissions and umasks, good usernames and passwords, logging, patching, and removing insecure daemons to integration with LDAP or active directory (AD).

Usernames and Passwords

This is one of the most basic protections. In order to have longer usernames and passwords, you need to make a system change. Changing the username length is almost always required if you want to integrate with LDAP or AD (active directory) and it requires a reboot.

Below is the command to increase the maximum username length to 36:

chdev -l sys0 -a max_logname=36
Then you check it as follows:
# lsattr -El sys0 | grep max_log
max_logname     36           Maximum login name length at boot time              True

The above change requires a reboot of the LPAR.

In order to have longer passwords, you need to use the chsec command. The version below causes the system to use ssha256 (up to 255 characters) for passwords. The next time local users change their password they will get a much longer, more secure password.

chsec -f /etc/security/login.cfg -s usw "pwd_algorithm=ssha256"

Then you can use getconf to check the setting

 getconf PASS_MAX
255 

Finally, I normally set the system up to automatically create home directories—this is important in an LDAP or AD environment.

chsec -f /etc/security/login.cfg -s usw -a mkhomeatlogin=true
 
tail /etc/security/login.cfg
        pwd_algorithm = ssha256
        mkhomeatlogin = true

Logging

Logging is a critical part of any system-protection strategy. Without logs, it’s impossible to know what’s been happening on the system. The syslog daemon (syslogd) starts by default on AIX, but the log configuration file isn’t set up to actually log everything. The first step is to correctly set up /etc/syslog.conf. It’s best to set up a separate file system for logs (e.g., /usr/local/logs) rather than use the default of /var/spool. If /var fills up, the system will crash; if your separate file system fills up, it will just stop logging. Although file systems should be monitored, it’s still wise to store logs in their own file system to protect against
large logs bringing down the system. Logs can be written to a file, sent to the console, logged to a central host across the network (be wary of this as the traffic
can be substantial), e-mailed to an administrator or sent to all logged-in users or any combination thereof. The most commonly used method is writing to a file in a file system. Once the file system is set up, code a /etc/syslog.conf file. Here’s an example file that writes to a local filesystem. It keeps the logs to no more than 2MB, then rotates and compresses them, keeping the last 10 logs. I do this on all LPARs and VIO servers.

#
mail.debug      /usr/local/logs/mailog rotate size 2m files 10 compress
*.emerg         /usr/local/logs/syslog rotate size 2m files 10 compress
*.alert         /usr/local/logs/syslog rotate size 2m files 10 compress
*.crit          /usr/local/logs/syslog rotate size 2m files 10 compress
*.err           /usr/local/logs/syslog rotate size 2m files 10 compress
auth.notice     /usr/local/logs/infolog rotate size 2m files 10 compress
*.info          /usr/local/logs/messages rotate size 2m files 10 compress

Go into /usr/local/logs and create each of the files above using touch.

Now you can stop (stopsrc -s syslogd) and start (startsrc -s syslogd) the logging daemon.

Insecure Daemons

The /etc/inetd.conf is over 120 lines long—some items are commented out but most are not. Many of the protocols listed in there have known security holes. One of the first things I do after setting up an lpar is to secure those protocols. I take a copy of /etc/inetd.conf to /etc/inetd.conf-orig and then delete everything in /etc/inetd.conf except the items I want to keep. I then do a “refresh -s inetd.” I do this on all my LPARs and my VIO servers. Typically, my inetd.conf looks like this:

#ftp      stream tcp6    nowait  root    /usr/sbin/ftpd          ftpd
#telnet stream  tcp6    nowait  root    /usr/sbin/telnetd         telnetd -a
#xmquery         dgram  udp6    wait     root     /usr/bin/xmtopas
  #dtspcd stream  tcp           nowait  root   /usr/dt/bin/dtspcd /usr/dt/bin/dtspcd

You will notice it’s only four lines and everything is commented out. On a NIM server you will see tftp and bootp uncommented. Occasionally when you do maintenance it uncomments or adds back in services. When the file is only 4 lines you can see immediately that it did that. I don’t use ftp and telnet because they are insecure and I have ssh and sftp instead. If you have to use telnet or ftp then you can uncomment them, but remember they send passwords, etc in clear text.

I would also recommend looking at /etc/rc.tcpip to see if snmp, sendmail and other daemons are starting. If you need snmp or sendmail to run then they should be properly configured to keep hackers from taking advantage of default exploits.

Time Synchronization

In a previous life I used to do a lot of security investigations. When issues involve multiple systems it’s a nightmare trying to follow the trails when the timestamps do not match. A simple fix for this is to implement NTP (network time protocol).  Most companies have an NTP server set up on their AD servers, but you can always point to the atomic clocks if necessary. You should set up ntp on your LPARs and your VIO servers.

Patching

At a minimum make sure you are running a fully supported version of the OS (VIO, AIX, IBM i, Linux). You can check this using the FLRT (fix level recommendation tool). It’s important to keep your patching up to date to proactively solve problems. In the AIX/VIO world there are two different kinds of patching. The first is fix packs (technology levels and service packs) and the second is efixes or ifixes (emergency or interim fixes). Fix packs are installed using install and efixes/ifixes are installed using emgr.

Technology levels and service packs are found at Fix Central. You should check here regularly for updates to your LPARs, VIO servers, server and I/O firmware and HMCs. Additionally, there are products installed, even at the latest service pack, that need updating—typically this includes Java, openssh and openssl.  Java patches are downloaded at Fix Central. openssh and openssl are downloaded at the Web Applications site.

I try to get a full patching window every six months unless it’s an emergency. You can use the FLRT and FLRTVC tools to determine what patching needs to occur.  Typically, I update the HMC first, then the server firmware, then the I/O firmware and VIO servers and finally the LPARs. However, you should look at the readme/description files for every update to make sure IBM does not have prerequisites that must be followed. This is particularly important with the HMC and server firmware interaction. There are also some requirements with POWER9 and adapter firmware because of the new trusted boot settings.

FLRTVC (efixes and ifixes)

Fix Level Recommendation Tool Vulnerability Checker (FLRTVC) comes in two flavors—the first is a script that you can run on the system that uses data from a file (apar.csv) to compare installed filesets and interim fixes against known security problems. The second option is to use the web based online tool (FLRTVC Online). This option allows you to upload the output from two commands to the web page and it produces the output that you need to identify security holes that need to be closed. I normally run it directly on the LPAR or VIO server.

To run flrtvc you first need to download the .zip file and then unzip it. You may also need to download the apar.csv file. If your LPAR/VIO does not have access to IBM then you will need to get the file from IBM and upload it to the LPAR yourself. You then edit the script and change SKIPDOWNLOAD from 0 to 1. It will now look for the apar.csv file in the same directory the script is in. Once that is done you can run it in compact mode and produce an output file as follows:

cd  /directory where flrtvc is
ksh93  ./flrtvc.ksh  >systemname-flrtvc-output.csv 

Then sftp or scp as asci the systemname-flrtvc-output.csv file to your computer and open it with Excel as a .csv file—the delimiter is |. There are a number of flags that you can use but for the most part I use none of them as I want to get everything.

I tend to have the output go to an NSF mounted filesystem so that all of my security reports are in one place. That way you can concatenate them together or at least just download them all from one place. You can also write scripts that grep on certain things in the output and email those to yourself.

The compact output from the FLRTVC script is best viewed in a spreadsheet and is broken down into the following columns:

• Fileset: shows the name of the fileset that is of issue i.e. bos.net.tcp.client
• Current Version: shows the currently installed level i.e. 7.1.3.45
• Type: will be either sec (for security) or hiper.
• EFix: you only see a value here if the actual efix for the problem is installed.
• Abstract: a description of the problem i.e. Vulnerability in BIND
• Unsafe Versions: a list of the fileset levels that are impacted  i.e. 7.1.3.0-7.1.3.45
• APARs: provide that actual APAR number i.e. CVE-2015-5477 or IV75031
• Bulletin URL: provides the URL where you can go to read about the vulnerability to get more information
• Download URL: provides the links to get the fix.

You can run flrtvc ahead of time and then download and prestage the updates. Flrtvc typically identifies efixes and ifixes that need to go on as well as Java, openssh, openssl and other updates that need to go on to the system.

Server Firmware and I/O Firmware

A large number of security and other fixes are done in firmware of some kind.  Server firmware and I/O firmware updates can resolve many issues and should be done at least once a year, preferably every six months. Those updates can be downloaded ahead of time and prestaged so that you are not dependent on the IBM sites during the maintenance window.

Timing of updates—typically I will wait until firmware, technology levels or service packs have been out for at least one month (preferably two) before I update to them. At that time, I will update my NIM server and then start to migrate the updates through test, dev, QA and finally, production. Having a good update strategy will save you a lot of downtime and will help with securing your systems.

Active Directory (AD) /LDAP Integration

When users are created on AIX systems (or any UNIX*) they’re assigned a default user (UID) and group (GID) number. The system permissions for files use those numbers. For that reason, you want to be sure that a user has the same UID and GID on every system. Doing this and setting permissions correctly should ensure they can’t access files they are not entitled to. You can either do this manually (think huge spreadsheets) or you can integrate your normal users into an AD or LDAP environment.

Implementing AD or LDAP integration on your AIX systems is much easier than it used to be. You will need to work closely with your AD/LDAP admin as well as the security team. A word to the wise, do not put root or other system accounts under the control of AD or LDAP—those have to be local. Instead, you should restrict them to console access only.

Enhanced Access

If you have admins or others who need enhanced access, then this should be provided using sudo or some other tool. If multiple users are logging in as root then there is no accountability. Using sudo causes everything to be logged. We used to have to install it using rpm and figure out all the prerequisites, etc.  Today, we go to the IBM Linux toolbox and download yum.sh and run it. This installs rpm and yum but it does require ftp access to IBM. Once yum is installed you can use yum to install sudo or other tools. You then use visudo to put together the rules. You can allow a user to become root with or without a password and you can also restrict them to only being able to use certain commands as root. This is very useful for level 1 support and DBAs who need privileges to perform certain tasks.

Backups

Everyone thinks about taking backups for data, but data is of no use if you have no OS. It’s critical to take regular mksysb (OS bootable) backups. I normally take them to my NIM server as that is where I would restore the system from. When discussing backups, you need to make sure these bare metal mksysb backups are part of any backup and disaster recovery plan. A mksysb should be taken at least monthly, and before and after any system maintenance. Additionally, I always have two disks (even on the SAN) on the system reserved for rootvg. One is active and the other is one I use to take an alt_disk_copy backup of rootvg before I make changes. You can never have enough backups!

A Multi-Silo Approach to Security

Security needs to be a team, multi-silo approach where everyone works together to provide the multiple layers that ensure the systems are as secure as possible. The security team should be able to provide you with the policies regarding usernames, etc. If you’re integrating with AD or LDAP then most of those policies are implemented there. While it takes a little time to implement the basic security measures above, it’s worth it to make your systems harder to break into. I build security in by default whenever I set up an LPAR or VIO server. It makes life much simpler to do it at that point. However, it’s still relatively easy to implement later although it will most likely require a reboot.

References

• FLRT Home Page
• FLRTVC Home Page
• Latest FLRTVC zip file (0.8.1)
• Apar.csv file
• FLRTVC Online Tool
• Fix Central (patches and updates)
• FLRT LITE (Check firmware and software supported levels)
• Web Applications (openssh, ldap, openssl, Kerberos)
• AIX Linux Toolbox

---