Java OutOfMemoryError strange behaviour

To keep things in perspective, consider running this code with -Xmx64m:

static long sum;
public static void main(String[] args) {
  System.out.println("Warming up...");
  for (int i = 0; i < 100_000; i++) test(1);
  System.out.println("Main call");
  test(5_500_000);
  System.out.println("Sum: " + sum);
}

static void test(int size) {
//  for (int i = 0; i < 1; i++)
  {
    long[] a2 = new long[size];
    sum += a2.length;
  }
  long[] a1 = new long[size];
  sum += a1.length;
}

Depending on whether you do the warmup or skip it, it will blow or not blow. This is because the JITted code properly nulls out the var, whereas the interpreted code doesn’t. Both behaviors are acceptable under the Java Language Specification, which means that you are at the mercy of the JVM with this.

^{Tested with Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode) on OS X.}

Bytecode analysis

Look at the bytecode with the for loop (simple code, without the sum variable):

static void test(int);
  Code:
   0: iconst_0
   1: istore_1
   2: goto  12
   5: iload_0
   6: newarray long
   8: astore_2
   9: iinc  1, 1
   12:  iload_1
   13:  iconst_1
   14:  if_icmplt 5
   17:  iload_0
   18:  newarray long
   20:  astore_1
   21:  return

and without:

static void test(int);
  Code:
   0: iload_0
   1: newarray long
   3: astore_1
   4: iload_0
   5: newarray long
   7: astore_1
   8: return

No explicit nulling out in either case, but note that in the no-for example the same memory location is actually reused, in contrast with the for example. This would, if anything, lead to the expectation opposite to the observed behavior.

A twist…

Based on what we learned from the bytecode, try running this:

public static void main(String[] args) {
  {
    long[] a1 = new long[5_000_000];
  }
  long[] a2 = new long[0];
  long[] a3 = new long[5_000_000];
}

No OOME thrown. Comment out the declaration of a2, and it is back. We allocate more, but occupy less? Look at the bytecode:

public static void main(java.lang.String[]);
  Code:
     0: ldc           #16                 // int 5000000
     2: istore_1      
     3: ldc           #16                 // int 5000000
     5: newarray       long
     7: astore_2      
     8: iconst_0      
     9: newarray       long
    11: astore_2      
    12: ldc           #16                 // int 5000000
    14: newarray       long
    16: astore_3      
    17: return        

The location 2, used for a1, is reused for a2. The same is true for OP’s code, but now we overwrite the location with a reference to an innocuous zero-length array, and use another location to store the reference to our huge array.

To sum it up…

Java Language Specification does not specify that any garbage object must be collected and the JVM spec only says that the “frame” with local variables is destroyed as a whole upon method completion. Therefore all behaviors that we have witnessed are by the book. The invisible state of an object (mentioned in the document linked to by keppil) is just a way to describe what happens to go on in some implementations and under some circumstances, but is in no way any kind of canonical behavior.