Sandbox against malicious code in a Java application

1. Run the untrusted code in its own thread. This for example prevents problems with infinite loops and such, and makes the future steps easier. Have the main thread wait for the thread to finish, and if takes too long, kill it with Thread.stop. Thread.stop is deprecated, but since the untrusted code shouldn’t have access to any resources, it would be safe to kill it.

2. Set a SecurityManager on that Thread. Create a subclass of SecurityManager which overrides checkPermission(Permission perm) to simply throw a SecurityException for all permissions except a select few. There’s a list of methods and the permissions they require here: Permissions in the Java^TM 6 SDK.

3. Use a custom ClassLoader to load the untrusted code. Your class loader would get called for all classes which the untrusted code uses, so you can do things like disable access to individual JDK classes. The thing to do is have a white-list of allowed JDK classes.

4. You might want to run the untrusted code in a separate JVM. While the previous steps would make the code safe, there’s one annoying thing the isolated code can still do: allocate as much memory as it can, which causes the visible footprint of the main application to grow.

JSR 121: Application Isolation API Specification was designed to solve this, but unfortunately it doesn’t have an implementation yet.

This is a pretty detailed topic, and I’m mostly writing this all off the top of my head.

But anyway, some imperfect, use-at-your-own-risk, probably buggy (pseudo) code:

ClassLoader

class MyClassLoader extends ClassLoader {
  @Override
  public Class<?> loadClass(String name) throws ClassNotFoundException {
    if (name is white-listed JDK class) return super.loadClass(name);
    return findClass(name);
  }
  @Override
  public Class findClass(String name) {
    byte[] b = loadClassData(name);
    return defineClass(name, b, 0, b.length);
  }
  private byte[] loadClassData(String name) {
    // load the untrusted class data here
  }
}

SecurityManager

class MySecurityManager extends SecurityManager {
  private Object secret;
  public MySecurityManager(Object pass) { secret = pass; }
  private void disable(Object pass) {
    if (pass == secret) secret = null;
  }
  // ... override checkXXX method(s) here.
  // Always allow them to succeed when secret==null
}

Thread

class MyIsolatedThread extends Thread {
  private Object pass = new Object();
  private MyClassLoader loader = new MyClassLoader();
  private MySecurityManager sm = new MySecurityManager(pass);
  public void run() {
    SecurityManager old = System.getSecurityManager();
    System.setSecurityManager(sm);
    runUntrustedCode();
    sm.disable(pass);
    System.setSecurityManager(old);
  }
  private void runUntrustedCode() {
    try {
      // run the custom class's main method for example:
      loader.loadClass("customclassname")
        .getMethod("main", String[].class)
        .invoke(null, new Object[]{...});
    } catch (Throwable t) {}
  }
}