I'm working late, so I need to vent. Please indulge me.

Working in a virtual machine is a wonderful thing. It is a clean, happy world where engineers have designed hypothetical, ideal execution environment for your code. Everything is peachy keen in Java world, because you can focus on solving a problem, because the VM handles nasty housekeeping issues like memory management.

Of course, this is crap.

My nice, neat virtual machine has to run on an actual, physical piece of hardware, with honest to God memory and processor limitations. It is nice that my Java code isn't terribly concerned with actual allocation and deallocation of physical memory, at some point is has to be concerned with the fact that allocation may fail.

Of course, this is crap, too.

There is more than one way to allocate memory, and there is more than one pool for which memory must be reserved. In the old, old days, when all we were doing was loading, linking, and executing bytecode, the concept of a single heap was close enough to reality. But then we added JIT compilers. And souped up real-time bytecode compilers with radically complicated memory models, garbage collectors, and an adaptive optimizer.

A simple heap of memory won't do for the most modern VMs. But didn't I say something about venting? Ah, yes.

I'm working on a large-ish Java system running in Weblogic's application server. That means I have loaded into my VM's memory space: all of the project classes, probably about 700, plus a lot of generated code based on the project classes; a good portion of the standard Java class library; a good portion of an in-house class library; lots of java extensions in the javax packages to support database access, EJBs, messaging, etc...;

Some time ago, we started seeing odd behavior from Weblogic. The symptoms started with the server's hot deploy feature failing. It would work once, then the server would enter into a loop that required a restart. We wrote this off to flaky software, as we could live without the hot deploy feature. Then, after a new chunk of code was integrated into the system, it started hanging. The server would stop processing EJB-based requests, then it would hang entirely.

I have rarely seen Weblogic hang, so this was really odd. However, the new code that we had integrated was complicated, using JMS, EJBs, and sockets to communicate with another system. The assumption was that something in this code was causing the server to freeze.

Then, the management console stopped working. Firing up the Weblogic console would reproducibly hang the server. This is getting worse.

Then we actually started getting out of memory exceptions. [Ah--finally he closes the loop] But looking at the application heap, there was plenty of free memory, both real and virtual.

So time to dig deeper. What VM are we using? 1.3 with hotspot. We're not running out of memory entirely, and when the server hangs, not all functions hang. Hmm. The hotspot compiler is really different than previous VMs, and it allocates a large buffer to hold its compiled code. We started wondering if the VM was loading so many classes that this buffer was full.

So we tried the shot in the dark fix of increasing the size of that buffer in the VM. And it worked. No hanging, the console is working, and we can hot deploy code again. In case you were wondering, the VM argument to change the size of the buffer is:

-XX:MaxPermSize=<size>

The -XX arguments are double-secret, non-documented VM arguments. The size argument, like other memory argument, is in bytes, but adding an "M" suffix specifies the size in megabytes. Check your platform for the initial size of the buffer; setting ours to 128 meg solved all these problems. I really dislike magic bullet, "one environment setting" kind of fixes, but this really seems to have been the problem.

I'm spelling this out in excruciating detail, so that Google's linky magic will catch these references. Maybe the next person who has this problem will find a fix more quickly than we did.

Oh, and not managing memory in Java is great fun, no?