See the customized window appearance and handling in the Ensemble Sample application, which includes source code. Download the source, build it and run it as a standalone application rather than embedded in a browser. It isn’t going to be exactly what you are asking for because stuff like dragging to screen top edge to activate maximizing isn’t going to work I think, but it should be very close and you could always code something yourself which maximized the window when it was dragged near the top edge. The Ensemble window has features like custom resize decorations, minimize, maximize, close icons, an area at the top of the window you can use to drag the window around or double click to maximize or minimize the window – i.e. most of the standard features you would expect from a desktop windowing system.
To get something even closer to what you are asking, perhaps you could hack something together by creating two windows. One, a standard decorated stage window which includes screen borders, the other an undecorated or transparent child stage always displayed on top of the main window and overlaying the borders of the main window with a custom rendering. I think you may run into difficulties trying to implement this approach, so I wouldn’t really recommend it.
You may like to try an UNDECORATED stage style rather than TRANSPARENT and see if you get better native Windows integration with that.
There are some open feature request currently scheduled for JavaFX to be shipped with JDK8, Windows: support Aero Glass effects for top-level windows, Mac: Support NSTexturedBackgroundWindowMask style for windows and The solid white background created in a Stage should be created – if needed – in the Scenegraph, which, when implemented, will likely help you to acheive your goal – vote for them, if such features are important to you.
Also checkout VFXWindows which is an open source windowing framework for JavaFX.
Also related is the Undecorator project which allows you to easily create a JavaFX stage with standard minimize/maximize/close/resize chrome controls that are rendered via the JavaFX engine rather than the OS windowing system. This allows you to achieve the kind of custom control over window rendering that an application like Ensemble displays.