Chiral Software, Inc. 310-356-7869

Installing JBoss Application Server on Ubuntu Linux

October 31st, 2008, updated January 2011

Introduction - hosting applications with JBoss Application Server

Please install a SVG plugin for your browser, or use Firefox to view this image.

In a previous article we showed how to run non-root processes on port 80. Java applications have inherent security advantages because they are running in a virtual machine. We can do even more by not running the server as root, ever. In contrast, a server like Apache HTTPD typically does run at least partly as root, and indeed, there have been exploits that have targeted Apache HTTPD in the time before it drops root. We can avoid this entirely using the iptables port redirection.

In this article, we'll show how to install JBoss as a daemon which starts on system startup and can be managed like the other daemons. We'll also configure logging properly, and secure the JMX console. These are all the tasks necessary to deploy JBoss Application Server as a normal server process, integrated into the Ubuntu Linux environment.

We're on Ubuntu Server 8.04 LTS. Updated: We are now running on Ubuntu 10.10 Maverick Meerkat.

Update, January 2011: These instructions have been updated for Ubuntu Server 10.10 and 10.04, and JBoss AS 6.0. Many things have changed since the original version of this article was posted over two years ago. We will continue to revise this to focus only on JBoss AS 6.0, removing the older information.

The lengthy security discussion in this article shows just how difficult it is to properly and securely deploy JBoss. There are numerous web applications running "out of the box", and many of these have no password security, and do have wide access to system resources. An administrator who is not very careful can easily create major security problems by installing JBoss 6 without understanding all the apps that run by default. In our view, JBoss should be secure-by-default when it is bound to external interfaces.

Preparation

Before you begin, make sure that a JDK package is installed. Either sun-java6-jdk or the OpenJDK package will work.

Create a user which the JBoss AS will run as. We create a separate user with a home directory, but no ability to log in and no password.

useradd -d /usr/local/jboss -s /bin/sh jboss

-d /usr/local/jboss specifies /usr/local/jboss as the home directory. -s /bin/sh specifies a standard shell, which is necessary for executing commands. Not specifying a password also disables login or password verification. You can see in /etc/shadow that the password value for jboss is a single !, meaning no password is valid.

Download JBoss AS. We're using version 6.0 Final, released December 2010.

Pick a place to install the server. Unfortunately the directory structure of JBoss AS is not consistent with the structures for the other Linux services. A proper Linux server application would have the server software itself in one location, probably under /usr. The server's configuration data would be under /etc, and the server's applications would be under /srv, and logs would be placed under /var/log. That type of breakdown is not feasible with JBoss AS unfortunately.

See: good overview of the standard Linux filesystem hierarchy for a good overview of the standard layout for Linux. I hope that future releases of JBoss AS will have better ways of integrating with the Linux directory structure. Even nicer would be the ability to install it properly with apt-get.

For now, we'll put it under /usr/local, as the jboss_init_redhat.sh script suggests. Unzip the zip file there, set the ownership to jboss / jboss, and also create a symbolic link so it can be accessed as /usr/local/jboss.

After unpacking, make sure user and group ownership are set to jboss.

Installing the init script

Copy the RedHat init file to the init directory:

cp /usr/local/jboss/bin/jboss_init_redhat.sh /etc/init.d/jboss

A few modifications are needed. It expects the JBoss server to be installed in /usr/local/jboss so that does not need to be changed, unless you have installed it somewhere else. It runs the server as user jboss which we have done here. These are the changes:

  1. Edit the JAVAPTH line to use /usr/bin as the path. I hope that someday packages will assume that the Java executable is /usr/bin and won't use path specifications. Until then, we need to tell it that java is in /usr/bin, the same place it has been for five years now. The Linux Standards Base requires that a working Java 6 binary be in /usr/bin, so why apps don't assume it's there at this point makes no sense.
  2. The JBOSS_HOST line must be changed: JBOSS_BIND_ADDR=${JBOSS_HOST:-"-b 0.0.0.0"}. This binds it on all IP addresses. To bind to a specific address, replace the 0.0.0.0.
  3. This script relies on a program called pidof. On RedHat, this is in /sbin but in Ubuntu, it's in /bin, so fix that. Also, RedHat's pidof is slightly different from Ubuntu 8.04's. Ubuntu's requires an -x argument to detect shell script PIDs. Ubuntu's also does not want the same string format. So the line should be for procid in `pidof -x $JBOSS_HOME/bin/run.sh`; do. With that line, it is able to stop the JBoss process correctly. Update, 2011: pidof is no longer used in the init script.

You can download a copy of the JBoss init script for Ubuntu with all these changes already made.

Add JBoss to the init system correctly

# update-rc.d jboss defaults
 Adding system startup for /etc/init.d/jboss ...
   /etc/rc0.d/K20jboss -> ../init.d/jboss
   /etc/rc1.d/K20jboss -> ../init.d/jboss
   /etc/rc6.d/K20jboss -> ../init.d/jboss
   /etc/rc2.d/S20jboss -> ../init.d/jboss
   /etc/rc3.d/S20jboss -> ../init.d/jboss
   /etc/rc4.d/S20jboss -> ../init.d/jboss
   /etc/rc5.d/S20jboss -> ../init.d/jboss

Test it out by running it

Run JBoss:

/etc/init.d/jboss start

Now the default JBoss console should be available on the host's external interface, on port 80.

Stop JBoss:

/etc/init.d/jboss stop

Before we continue we need to secure the JMX console and all the other web applications JBoss AS starts with.

Putting the applications in /srv/jboss

Web servers often place their applications within either /srv or perhaps /var/www. It's easy to configure JBoss to look in some other directory for its applications.

Note: This setting is no longer valid in JBoss AS 6. We have not been able to find out the equivalent setting in JBoss AS 6 yet. Edit server/default/conf/jboss-service.xml. Find the org.jboss.deployment.scanner.URLDeploymentScanner mbean. Find the URLs attribute of this. The value of this attribute is a coma-separated list of directories. We use this value:

<attribute name="URLs">
    deploy/,
    /srv/jboss/
</attribute>

Please install a SVG plugin for your browser, or use Firefox to view this image.
WARNING!

Deploying an EAR in some directory other than the deploy directory is not compatible with Hibernate full-text search. For some reason, the scanner that scans the EAR's lib directory does not detect the lucene-core.jar as it is creating the the application's EntityManager. This means that the EntityManager that is created is not a FullTextEntityManager and so all the full text searches fail with ClassCastExceptions:

Exception during request processing:
Caused by javax.el.ELException with message: 
"java.lang.ClassCastException: org.jboss.seam.persistence.FullTextHibernateSessionProxy 
      cannot be cast to org.hibernate.impl.SessionImpl"

If your app uses the lucene-core.jar and a FullTextEntityManager, deploy it in server/default/deploy, not in some external directory. We are investigating this to see if there is some other solution.

Please install a SVG plugin for your browser, or use Firefox to view this image.
The WARs of JBoss

JBoss AS 6 ships with an excellent Admin Console application. It can be accessed at http://example.com/admin-console. The default username and password are both admin.

We need to secure JBoss before we deploy it. This means the following:

JBoss opens quite a number of TCP and UDP ports in its default configuration. It is necessary to have a firewall on the machine to protect these ports. We hope in future there will be a better way to manage which ports JBoss is opening.

As you can see, getting JBoss AS 6 into a secure operation mode is non-trivial, and it is far from secure-by-default which people expect today. Securing a JBoss installation is much like securing a Unix installation 20 years ago, involving a search for openings. It would be better if JBoss would include some kind of firewall feature that would let the administrator define, in one location, which applications could provide external network services. On a computer, you can scan for open ports, but within JBoss, rather than ports, there are web contexts, and there is no way to scan those.

Using the Admin Console, we click on the Applications node, and get a list of all the applications running:

JBoss AS 6 Admin Console
JBoss AS 6 Admin Console

JBoss deploys the following types of applications:

We need to secure WARs, both embedded and non-embedded. If any of the RARs are listening on ports we will find them during the NMAP scan.

Looking at this listing, we see the following web applications:

We'll go through each of these WARs and secure them.

The JMX console, admin console, and JBossWS console

None of these are password-protected by default. The JMX console can be used to issue various commands, shut down the server, and many other things which should not be open. The server administration console can be used to deploy or undeploy applications or other important actions. Neither should be visible to web visitors.

The consoles are ordinary web applications, so they can be secure in the same way any other Tomcat deployed web application is secured.

Default JBoss 6 web applications
Application File path URL context
Admin console common/deploy/admin-console.war /admin-console
JMX console common/deploy/jmx-console.war /jmx-console
JBossWS console common/deploy/jbossws-console.war /jbossws

Edit the relevant WEB-INF/web.xml files. The JMX Console and the Admin Console both use the same role in their security constraint, namely JBossAdmin. JBossWS console does have a security-constraint defined by default. For simplicity, it could be set to use the JBossAdmin role. Both the JMX Console and the Admin Console use users and roles defined in server/default/conf/props/jmx-console-*.properties. The JBossWS console uses the jbossws-.properties files.

Questions remaining: What about securing HornetQ? It uses netty as a network interface, and it has a user properties file, but I can't find where it binds to the network or listens for requests, or even which protocol it uses. It listens on port 5445, which should be blocked by firewall until it is disabled.

The ROOT.war application

This application is disabled if some other application is deployed which binds to the root context. Therefore the ROOT.war application can be left as-is. Alternatively, the entire WAR directory can be deleted, or a security constraint could be added.

JBossMQ

Update, JBoss 6: JBossMQ has been removed from JBoss 6, so no need to perform this step.

Unless you have an application running that requires JBossMQ, this application can be simply deleted. JBossMQ is a legacy service, and will be replaced with JBoss Messaging in the JBoss 5.0 release. As of the writing of this article (early November 2008), the JBoss 5.0 release is imminent.

Delete the entire directory:

rm -r  server/default/deploy/jms

You can also delete the JAR file:

rm server/default/lib/jbossmq.jar

JBossWS

This provides access to JBoss web services. Some applications do provide web services with no security constraints on them, such as a web service that provides public information. If you have such a web service in one of your applications, leave JBossWS as it is, because it must be open to the public. Otherwise, web applications that provide web services can provide their own security contraints. If there are no web services needed in the JBoss installation, the entire SAR can be removed. The directory to remove is deploy/jbossws.sar.

invoker.war - JNDI over HTTP

This is the legacy style "detached invoker" for doing RMI. EJB3 does not use this. It should be removed.

rm -r server/default/deploy/http-invoker.sar

Testing them all

Here is a table of the WARs and their URLs:

WARURLJBoss 6?
jmx-console.war /jmx-console Yes
management/console-mgr.sar/web-console.war /web-console
jboss-web.deployer/ROOT.war root context
jms/jbossmq-httpil.sar/jbossmq-httpil.war /jbossmq-httpil/HTTPServerILServlet/* No
jbossws.sar/jbossws-context.war /jbossws
http-invoker.sar/invoker.war invoker - see WEB-INF/web.xml, test /invoker/JNDIFactory/

After you have set up access restrctions on all the WARs, test them.

Configure logging

This step is optional. If you don't change anything logs will go in server/default/log. This isn't bad but it makes more sense for logs to go under /var/log, the same as the other system logs. Create a directory for the JBoss logs:

mkdir /var/log/jboss

We can direct all of the server's log output into this directory using the jboss.server.log.dir parameter. The best place to put that setting is either within /etc/init.d/jboss, as an argument to run.sh, or to edit bin/run.conf. In both cases, the way to set the parameter is to add the following option to the JVM invokation:

-Djboss.server.log.dir=/var/log/jboss

Edit deploy/jboss-web.deployer/server.xml. This is, of course, an ordinary Tomcat server configuration file. Switch from the outdated common log format to the more-useful combined log format. This is also where JBoss virtual hosts are configured, and each can be given its own log prefix. Note that this file refers to the jboss.server.log.dir parameter, so if that parameter is set, the logs will be deposited there.

Finally, JBoss may be logging too much in its default configuration. Edit server/default/conf/jboss-log4j.xml. We use EJB3 in our applications, and EJB3 uses Hibernate as its provider, and we find our logs full of Hibernate query logging. One parameter to check is the hibernate.show_sql parameter in your applications persistence.xml. Setting the parameter to true is helpful for debugging, but generates too much log traffic in real use. You can also add a configuration in the jboss-log4j.xml file to adjust the logging setting of the org.hibernate category:

<category name="org.hibernate">
      <priority value="TRACE"/>
      <appender-ref ref="HIBERNATE"/>
            </category>

This setting sends Hibernate logs to a separate appender, which could be to a different directory. This also turns on TRACE level logging. For a production system you want less logging than that.

For Hibernate to use log4j, the commons-logging.jar must be in the application's WEB-INF/lib.

You can also turn off console logging if you want to by removing the console appender reference from the root block at the end of the file.

What we haven't covered: SSL certificates

Most real-world applications will also need SSL certificates for at least some parts of the site. We'll cover that later in a separate article, where we talk about SSL certificates and also Seam Security.

Conclusion

We have installed JBoss AS in a way which is reasonably well integrated with Linux, using the /etc/init.d mechanisms to start JBoss on server boot. We've also secured some services which need to be secured. We've configured logging in a reasonable way. And don't forget, configure PermGen space correctly, or the server will have a lot of problems if it hosts more than a couple of applications.

It may seem like a lot of effort to install a simple application server, but it's actually quite fast once you have a check-list of the necessary steps.

I do hope that, in future, a few things will be different about the way JBoss AS is distributed:

All in all, Ubuntu Server is a great environment for running JBoss AS. The two together make it possible to build fantastic applications with the greatest security, stability and flexibility in any software platform we have seen.

Update, November 12th 2008:

Ubuntu Server 8.10 "Intrepid Ibex" ships with OpenJDK, so you get a /usr/bin/java in the default installation which provides Java 6. This is more convenient than installing Sun's JDK, which are somewhat cumbersome packages.

And, as of this month, the Linux Standards Base (LSB), which is the standard which allows software developers to create software packages which will install on any LSB-based distribution, includes support for Java 6. At this point, the support is called "trial use", meaning, it is not part of the LSB conformance tests. However, we can expect the major distros to support it, and we can expect it to become mandatory in future versions of LSB. This is good news for anyone shipping a Java application. You can rely on LSB-compliant distros (including Ubuntu and RedHat) to have a Java 6 binary at /usr/bin/java. Finally. Now I hope that all the Linux startup scripts for JBoss, Tomcat, etc, will start relying on LSB and a predictable Java 6 in a predictable location.

Copyright 2003-2013, Chiral Software, Inc. colophon