Analysis of problems encountered while launching a site with JBoss Seam

May 31st, 2008

This past week, we attended the Google I/O 2008 conference in San Francisco. In preparation for the conference, we switched our site from the "old" JSP / Servlet based site to a new site built entirely on JBoss Seam. This new site included our Android seminar signup form, an important feature, given that marketing of the seminar was beginning.

The site worked fine in testing. We pushed the site onto the server on the 27th, and it appeared to work. But after a couple of hours, the site would go down, returning 500 Internal Server Error responses. It came back up without problems when it was reset.

A look in the logs showed one problem:

66.249.70.171 - - [31/May/2008:12:32:41 -0700] "GET /index.seam?cid=440 HTTP/1.1" 200 21948 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.171 - - [31/May/2008:12:33:34 -0700] "GET /index.seam?cid=16 HTTP/1.1" 200 21948 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.171 - - [31/May/2008:12:34:50 -0700] "GET /index.seam?cid=35 HTTP/1.1" 200 21949 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.171 - - [31/May/2008:12:34:55 -0700] "GET /index.seam?cid=437 HTTP/1.1" 200 21949 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.171 - - [31/May/2008:12:35:32 -0700] "GET /index.seam?cid=460 HTTP/1.1" 200 21949 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.171 - - [31/May/2008:12:37:19 -0700] "GET /index.seam?cid=511 HTTP/1.1" 200 21949 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.171 - - [31/May/2008:12:37:30 -0700] "GET /index.seam?cid=52 HTTP/1.1" 200 21949 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

GoogleBot was requesting the same page, with a different conversation ID (CID) parameter. Each time it made such a request, a new Session would start:

12:40:47,811 INFO  [Contexts] starting up: org.jboss.seam.security.identity
12:40:47,812 INFO  [Contexts] starting up: org.jboss.seam.web.session
12:40:49,375 INFO  [Contexts] starting up: org.jboss.seam.security.identity
12:40:49,376 INFO  [Contexts] starting up: org.jboss.seam.web.session
12:40:50,887 INFO  [Contexts] starting up: org.jboss.seam.security.identity
12:40:50,888 INFO  [Contexts] starting up: org.jboss.seam.web.session
12:40:52,423 INFO  [Contexts] starting up: org.jboss.seam.security.identity
12:40:52,424 INFO  [Contexts] starting up: org.jboss.seam.web.session
12:40:54,046 INFO  [Contexts] starting up: org.jboss.seam.security.identity
12:40:54,047 INFO  [Contexts] starting up: org.jboss.seam.web.session

This isn't enough information to know conclusively what the problem is, but it is obviously bad behavior for a web server, and for GoogleBot. GoogleBot should ignore the cid request parameter. Seam should not create a conversation, or a session, for GoogleBot. And the website should not attach &cid=99 to links that GoogleBot is crawling.

Immediate actions taken:

1. We were using Seam 2.02GA. On the 28th, JBoss announced a Service Pack for Seam to correct a performance problem. We immediately installed it.
2. Add propagation=none to all the s:link tags except in cases where the conversation is actually needed.
3. Excluded login.seam and similar pages by adding them to our robots.txt file.
4. Log in to Google's webmaster tools and set the crawl speed to "slow":
   

These were immediate, obvious fixes to make before starting to analyze the problem itself. These changes helped some, but GoogleBot was still making hundreds of requests to pages distinguished only by cid and the server was still failing in a serious way, after a couple of hours:

12:18:56,619 INFO  [Contexts] starting up: org.jboss.seam.security.identity
12:18:56,621 INFO  [Contexts] starting up: org.jboss.seam.web.session
12:20:38,177 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
12:20:39,372 ERROR [CoyoteAdapter] An exception or error occurred in the container during the request processing
12:20:41,753 ERROR [Http11Processor] Error processing request
12:20:45,610 ERROR [Http11Protocol] Error reading request, ignored
12:20:48,143 ERROR [STDERR] Exception in thread "http-205.134.230.203-80-3"
12:20:48,144 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space
12:20:48,146 INFO  [Contexts] starting up: org.jboss.seam.security.identity
12:20:49,351 ERROR [[/]] Session event listener threw exception
java.lang.OutOfMemoryError: PermGen space
12:21:08,418 ERROR [[Faces Servlet]] Servlet.service() for servlet Faces Servlet threw exception
java.lang.OutOfMemoryError: PermGen space
12:21:10,802 ERROR [CoyoteAdapter] An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space
12:21:43,152 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
12:21:45,533 ERROR [CoyoteAdapter] An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space
12:21:46,879 INFO  [Contexts] starting up: org.jboss.seam.security.identity
12:21:48,075 ERROR [[/]] Session event listener threw exception
java.lang.OutOfMemoryError: PermGen space
12:21:58,817 ERROR [[default]] Servlet.service() for servlet default threw exception
        java.lang.OutOfMemoryError: PermGen space

That is a serious error. We made one more quick change, hoping to restore stability:

5. Edit the JBoss bin/run.conf file to increase the amount of memory allocated to the JVM:

   JAVA_OPTS="-Xms256m -Xmx1024m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
               

It's reasonable to allocate more memory to JBoss, because this server has plenty of RAM. However, this change did not help.

How GoogleBot crawls

Even after propagation="none" was added to all the visible s:links on the site, GoogleBot was still hitting URLs with conversation IDs. Why was it doing this?

Google's fetching process involves three separate steps:

1. Fetch a page
2. Parse the page to extract the links
3. Add the links to a queue of URLs to be fetched in the next round of crawling the site.

What was happening was a long list of links to fetch, all with conversation IDs, were in Google's crawling queue. It might take days for those to get out of the queue. We couldn't wait days for stability to return to the site; we had just talked with hundreds of people who wanted to sign up for our Android seminar. The site needed to work.

The solution was to write a Filter to give 404 Not Found HTTP responses to Google's requests for links with conversation ID parameter. Seam itself has a convenient @Filter annotation, allowing a programmer to include a Filter without needing to put it in web.xml. However, Filters defined using @Filter are called from within Seam, so conversation processing may have already taken place. We need a filter that happens before Seam happens.

It's easy enough to define such a filter using the standard web.xml method. However, we already have one other filter that happens before the Seam Filter happens:

<filter>
  <filter-name>UrlRewriteFilter</filter-name>
  <filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>
</filter>

<filter-mapping>
  <filter-name>UrlRewriteFilter</filter-name>
  <url-pattern>/*</url-pattern>
</filter-mapping>

Using URLRewrite to send a 404 for Googlebot's user-agent

URLRewrite is most often used to turn complicated URLs with GET parameters into search engine optimized (SEO) URLs. But it can do much more than that, we're already using it, and it's optimized for speed.

We create this rule:

<rule>

    <!-- a Perl-style regular expression which matches -->
    <!-- any URI which ends with a CID parameter -->
    <from>.*</from>

    <!-- First a condition: the user-agent is Googlebot -->
    <condition next="and" name="user-agent">Googlebot</condition>
    <!-- Note the use of the next="and" to chain this rule -->
    <!-- with the next rule -->

    <!-- and there is a conversation ID parameter -->
    <condition type="parameter" name="cid">.+</condition>
    
    <!-- NOTE: using a regexp like -->
    <!-- <from>cid=[0-9]+$</from> -->
    <!-- seems like it would work, BUT IT DOES NOT. -->
    <!-- The reason is that the request parameters -->
    <!-- are not visible in the from matching string.  -->
    <!-- Only the request URI is visible.  If you want -->
    <!-- to match on parameters, you must use the -->
    <!-- condition statement, with a type="parameter" -->

    <!-- Send a "404 Not Found" response code -->
    <set type="status">404</set>

    <!-- Setting this to null ends the filter chain; -->
    <!-- chain.doFilter() is not called -->
    <to>null</to>
 </rule>

My first effort was to use a from line like this:

<from>cid=[0-9]+$</from>

This doesn't work. The from pattern only gets the URI, without any GET parameters. To access the GET parameters, use the

<condition type="parameter" name="param_name">regexp</condition>

construction as shown. The advantage here is that it will work no matter which order the parameters come in. It uses the HttpServletRequest.getParameter() method to find the value.

Testing

curl is the tool to turn to for testing at this level. We have four variations to test:

Test	Expected result	Test	Expected result
Googlebot with cid	404	Googlebot without cid	200 (normal response)
Browser with cid	200 (normal response)	Browser without cid	200 (normal response)

Googlebot with cid (all these commands are all on the same line):

% curl --silent -D- 
   -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' 
   'http://chiralsoftware.com/colophon.seam?cid=474' | head
    HTTP/1.1 404 Not Found

Googlebot without cid:

% curl --silent -D- 
   -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' 
   'http://chiralsoftware.com/colophon.seam' | head
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-Powered-By: Servlet 2.4; JBoss-4.2.2.GA (build: SVNTag=JBoss_4_2_2_GA date=200710221139)/Tomcat-5.5
X-Powered-By: JSF/1.2
Set-Cookie: JSESSIONID=5B30A3C55921995CA80A2E5B6E351C23; Path=/
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Date: Sat, 31 May 2008 22:10:09 GMT

<html xmlns="http://www.w3.org/1999/xhtml">

Normal browser with CID:

% curl --silent -D- 
    -A 'Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14' 
    'http://chiralsoftware.com/colophon.seam?cid=999' | head
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-Powered-By: Servlet 2.4; JBoss-4.2.2.GA (build: SVNTag=JBoss_4_2_2_GA date=200710221139)/Tomcat-5.5
X-Powered-By: JSF/1.2
Set-Cookie: JSESSIONID=9D698C6E3F6A6967D6B2220D9F4E8C83; Path=/
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Date: Sat, 31 May 2008 22:13:28 GMT

<html xmlns="http://www.w3.org/1999/xhtml">

Normal browser without CID:

% curl --silent -D- 
    -A 'Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14' 
    'http://chiralsoftware.com/colophon.seam' | head
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-Powered-By: Servlet 2.4; JBoss-4.2.2.GA (build: SVNTag=JBoss_4_2_2_GA date=200710221139)/Tomcat-5.5
X-Powered-By: JSF/1.2
Set-Cookie: JSESSIONID=9D698C6E3F6A6967D6B2220D9F4E8C83; Path=/
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Date: Sat, 31 May 2008 22:13:28 GMT

<html xmlns="http://www.w3.org/1999/xhtml">

Success?

All tests pass. Googlebot is still crawling, but is no longer initiating sessions or using server resources as it hits URLs with CIDs:

66.249.70.171 - - [31/May/2008:15:05:31 -0700] "GET /blogs-where.seam?cid=135 HTTP/1.1" 404 946 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.171 - - [31/May/2008:15:05:33 -0700] "GET /blogs-where.seam?cid=650 HTTP/1.1" 404 946 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.171 - - [31/May/2008:15:05:35 -0700] "GET /blogs-where.seam?cid=634 HTTP/1.1" 404 946 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.171 - - [31/May/2008:15:05:37 -0700] "GET /blogs-where.seam?cid=496 HTTP/1.1" 404 946 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.171 - - [31/May/2008:15:05:39 -0700] "GET /blogs-where.seam?cid=432 HTTP/1.1" 404 946 "null" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Also, the server log does not show a new conversation starting with every Googlebot hit. This is good. It is important to not feed thousands of duplicate content URLs to Google. Doing that can penalize the placement of the site or result in less than optimal display in search results.

But the server is still not stable.

Permgen space, classloaders, and debug="true"

In Seam's components.xml, there is a setting to control whether Seam runs in debug mode:

<core:init debug="true"
    jndi-pattern="chiralv2//local"/>

Debug mode is handy. Errors result in an informative debug page, where it is possible to view the components which are active in any of the scopes (conversation, session, etc). Debug mode also comes with a special class loader which allows Seam to dynamically reload the application's classes, other than session beans and entity beans.

This same classloader must have been interacting with some classes in the Chiral Software website application in some way which was rapidly exhausting permgen space. For those who are familiar with developing for Tomcat, Tomcat also allows hot deployment and re-deployment of web applications. In practice, after half a dozen hot redeployments, the Tomcat server would often crash with an "out of permgen space" message. A similar situation is happening here.

Debug should be turned off for applications deployed on live servers. The hot redployment classloader is not designed for use in such situations, and in fact it has a bad interaction with our code.

Too many classes? The PermGen

Turning off debug mode helped some, but the server was still not stable. These errors are "out of PermGen space". It's important to understand how the Java Virtual Machine divides up memory, and what the PermGen is.

First, PermGen space is not memory used by an application. Applications store their data on the heap. Here's an application that runs out of memory:

import java.io.PrintStream;
import java.util.List;
import java.util.LinkedList;

public final class NotEnoughMemory {
    
    private static final PrintStream out = System.out;

    public static void main(String[] args) {
        out.println("I'm going to run out of memory");
        
        // Use a list so that we'll keep a reference
        // to the arrays we create, so they
        // can't be garbage collected.
        // This is an artificial memory leak
        final List l = new LinkedList();
        int increment = 1024 * 1024; // we'll add 1mb at a time
        while(true) {
            out.println("There are: " + (l.size() * increment / 1024) + " " +
                        "kilobytes taken up in the byte arrays.");
            l.add(new byte[increment]);
        }
    }

}

And this is the OutOfMemory error it gives us:

There are: 518144 kilobytes taken up in the byte arrays.
There are: 519168 kilobytes taken up in the byte arrays.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
   at NotEnoughMemory.main(NotEnoughMemory.java:21)

It has run out of Java heap space after allocating 519mb of byte arrays. Let's do a test by changing some of the JVM parameters to allow it more memory:

% java -Xmx1024m NotEnoughMemory
There are: 1046528 kilobytes taken up in the byte arrays.
There are: 1047552 kilobytes taken up in the byte arrays.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
   at NotEnoughMemory.main(NotEnoughMemory.java:21)

We have doubled the heap size from the default, and it runs right up to the 1gb we have allocated for it.

But the error that is stopping JBoss is a PermGen space error, not a heap error. For historical reasons, the Sun JVM stores class data in a separate memory space from class instance (object) data. Classes are stored in the "permanent generation" while instances (objects) are stored within the heap. For a great explanation, see Jon Masamitsu's article.

The default maximum permgen (class storage) space in the JVM is 64mb. That's enough for many uses of the JVM, but it's not enough on JBoss. Every application within JBoss uses its own classloader hierarchy, so if you have three Seam applications on one server, all the Seam classes are loaded into memory three times. And if your three apps uses other classes, like iText for PDF generation, those classes are also loaded three times into the PermGen space. Suddenly 64mb doesn't seem like very much memory.

How much PermGen space does a JBoss Seam application use?

Seam ships with a convenient command line utility called SeamGen. SeamGen uses an Ant build.xml file to quickly create a buildable and deployable Seam application. Let's use it.

I unzipped the JBoss Application Server into a temporary directory. I then run seam setup create-project repeatedly to create multiple Seam projects. The first time seam setup runs, I set values such as the location of the AS. In subsequent runs of seam setup, I need only to change the name of the project: myproject1, myproject2, myproject3, etc.

Within each project, I run ant explode to deploy it. Then I go into the exploded directory in the AS, and change debug to false in components.xml. I don't want to use the debug classloader, which may have a very different PermGen utilization than the standard classloader.

I started with five applications. The procedure is to start the server and then access some pages in each app to make sure the classes are loaded. In particular, I simply accessed the home.seam page, and then I clicked on the login link.

As I clicked on myproject5, I already got an exception:

21:41:51,868 ERROR [STDERR] Exception in thread "http-127.0.0.1-8080-1"
21:41:52,828 ERROR [STDERR] Exception in thread "ScannerThread"

Unfortunately it doesn't tell me any more about it, and the server is crashed and non-responsive. It needs to be terminated with kill -9.

This type of hard crash is a typical outcome of a Java memory error. Java can't operate normally after an OutOfMemory exception, and sometimes it's unable to construct new objects necessary for throwing exceptions or generating log entries. But sometimes it can. What happens depends on minute details of garbage collection and memory, which are influenced by tiny differences in timing. OutOfMemoryErrors are not perfectly reproducible for this reason. So I run it again, hoping to see the exception this time, and sure enough, while attempting to load myproject5:

21:47:10,309 ERROR [ExceptionFilter] handling uncaught exception
21:47:12,433 ERROR [[Faces Servlet]] Servlet.service() for servlet Faces Servlet threw exception
21:47:14,663 ERROR [CoyoteAdapter] An exception or error occurred in the container during the request processing
21:47:16,816 ERROR [Http11Processor] Error processing request
java.lang.OutOfMemoryError: PermGen space
        at java.lang.Throwable.getStackTraceElement(Native Method)
        at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
        at java.lang.Throwable.printStackTrace(Throwable.java:510)
        at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:59)
        

We also observed this while trying to stop the server occasionally:

[1]    Terminated                    ./bin/run.sh
laptop:/tmp/jboss-4.2.2.GA> Java HotSpot(TM) Server VM warning: 
  Exception java.lang.OutOfMemoryError occurred dispatching signal 
    SIGINT to handler- the VM may need to be forcibly terminated

Five do-nothing Seam projects, on a server with no load, are enough to run the server out of PermGen space with the default run.conf settings.

This is obviously why the server was not stable. There were four apps on it, and they use many more classes than a "do nothing" project.

Let's test how many more apps we can run by expanding the PermGen space. First, it won't be linear. The default PermGen space is 64mb. The JBoss AS must be taking up a significant amount of that space with its own internal classes, plus classes loaded by the JVM. The standard 64mb was able to run 4 simple "do-nothing" Seam apps. If we double to 128mb, we should expect to be able to run more than 8 Seam apps. The additional apps we get with 64mb more will give us a rough indicator of what we want to know: the PermGen space occupied by a bare-bones Seam app. I've set up 16 bare-bones projects.

In testing, when I had 16 projects, it would fail when I tried to access a page on the 15th. If I remove the 16th project, less PermGen space is needed, so the 15th project works successfully. We can say that 128mb of PermGen space, set using -XX:MaxPermSize=128m, gives us 14 "bare bones" apps. 64mb gave us four apps, so adding 64mb gave us roughly 10 additional apps. We can say that one bare-bones Seam app requires roughly 6.4mb of PermGen allocation. This lets us estimate that JBoss AS, without any apps loaded, is using roughly 32mb of PermGen space.

This gives us a simple formula for capacity planning:

PermGen space required, in MB = # of Seam apps * 6.4 + 32

Note that this forumla is approximate, and it's for bare-bones applications. Any real application is going to load more classes. If the application uses any other libraries, such as iText, it's going to load a lot more classes.

It's necessary to know this number for capacity planning, although in practice, RAM is cheap and server downtime is expensive, so I would at least double that number. 1gb of RAM is in the range of $100, so it's much cheaper to err on the side of some extra PermGen allocation.

For our capacity planning, we will use this formula and double the result, to be on the safe side.

Discussion

If we had had propagation="none" in all of our s:link tags from the very beginning, these URLs with conversation IDs would not have ended up in Google's crawl queue, and this problem would never have come up. However, this type of situation is something that would have been impossible to observe during pre-launch testing, because we don't have a Google crawler we can use in a test environment. In this case, we'll leave our new URLRewrite rule in place, just in case we ever have a link somewhere that is propagating conversation IDs.

We originally used s:link with propagation for a good reason. We want conversations to follow users around in the site. For example, if you type something in the search box, it should stay there as you click around the site. And if you have the site open in two windows or tabs at the same time, the search box on each should maintain its state independent of the other. That's exactly what Seam conversations are for, but that type of use has a very bad interaction with Googlebot.

How do we preserve our use goals (conversations for real users) while keeping the site up? One obvious thing to do would be to write a custom JSF tag which is a sub-class of Seam's s:link tag. That would be easy to do and it would work.

But since we're using URLRewrite, with an option for detecting user agents, why not continue? We can use an outbound rule to strip out cid=999 when the user-agent is Googlebot. This is easy to do. Remember that, unlike inbound rules, the regexp of an outbound rule has access to the full URL, including parameters. How to write this is left as an exercise for the reader.

chiralsoftware.com has a strong page rank and is aggressively crawled by Googlebot. On a site with this level of search engine visibility, it's important to plan for interactions between the site and Googlebot. During this problem with CIDs, we have had thousands of visits per day from Googlebot. Once Google has a site in this crawl category, crawler traffic can go up very quickly.

But Googlebot was not what was bringing the site down. The problem is that the default PermGen space allocation is unreasonably low for a server deployment which runs more than 5 applications. We did an analysis to determine the minimum PermGen requirements for a minimal Seam app, and came up with a helpful formula which can be used as a minimum. In real planning, the result of the formula should probably be doubled.

Conclusion

In this paper, we observed a failure of a website caused by an interaction between JBoss Seam conversations and Googlebot. We took some immediate corrective measures which were prudent, but did not solve the problem. We then explored the problem in more detail. We came up with one possible solution: write a Filter. However, we decided to use the URLRewrite filter that we already had, and add a rule to handle Googlebot in a special way. We tested and found that this solution works. Finally, we discussed ways to preserve our initial goals (have users in conversations) while interacting correctly with Googlebot. The goal could be achieved with a custom JSF tag, but it would be more elegant and simpler to use an outbound URLRewrite rule.

We solved the problem with Google's crawler, and that problem did need to be solved, but that wasn't what was causing the instability. Switching debug off fixed the problem. Seam's hot redployment debug classloader shouldn't be used on live sites.

But the real problem was lack of PermGen allocation. We now have a formula to go by for configuring that.

June 4th, 2008 - Follow-up

We've continued to monitor the site, looking for both stability and the impact of our URLRewrite rules. Here's what we see, after a few days:

• The site is rock-solid, as we expect with JBoss Application Server
• Seam is doing everything we want it to. Our Seam-based CRM system was much easier in Seam than it would have been with any of the other options. It would have taken at least ten times longer to code in PHP, and been less stable, for example.
• Googlebot is now crawling properly. All the cid links are gone from its crawl queue. It is now indexing real pages (only) at a reasonable rate.
• Traffic is increasing on the site. We think that the new design has improved the user experience, and articles such as this one are also drawing traffic.

For JBoss Seam consulting or training, call us at 310 356 7869 to discuss your needs.