Update for OS X El Capitan

The hack I described in my original answer below is not needed on OS X El Capitan anymore. The NSVisualEffectView’s maskImage should work correctly there, if the NSWindow’s contentView is set to be the NSVisualEffectView (it’s not enough if it is a subview of the contentView).

Here’s a sample project: https://github.com/marcomasser/OverlayTest

---

Original Answer – Only Relevant for OS X Yosemite

I found a way to do this by overriding a private NSWindow method: - (NSImage *)_cornerMask. Simply return an image created by drawing an NSBezierPath with a rounded rect in it to get a look similar to OS X’s volume window.

---

In my testing I found that you need to use a mask image for the NSVisualEffectView and the NSWindow. In your code, you’re using the view’s layer’s cornerRadius property to get the rounded corners, but you can achieve the same by using a mask image. In my code, I generate an NSImage that is used by both the NSVisualEffectView and the NSWindow:

func maskImage(#cornerRadius: CGFloat) -> NSImage {
    let edgeLength = 2.0 * cornerRadius + 1.0
    let maskImage = NSImage(size: NSSize(width: edgeLength, height: edgeLength), flipped: false) { rect in
        let bezierPath = NSBezierPath(roundedRect: rect, xRadius: cornerRadius, yRadius: cornerRadius)
        NSColor.blackColor().set()
        bezierPath.fill()
        return true
    }
    maskImage.capInsets = NSEdgeInsets(top: cornerRadius, left: cornerRadius, bottom: cornerRadius, right: cornerRadius)
    maskImage.resizingMode = .Stretch
    return maskImage
}

I then created an NSWindow subclass that has a setter for the mask image:

class MaskedWindow : NSWindow {

    /// Just in case Apple decides to make `_cornerMask` public and remove the underscore prefix,
    /// we name the property `cornerMask`.
    @objc dynamic var cornerMask: NSImage?

    /// This private method is called by AppKit and should return a mask image that is used to 
    /// specify which parts of the window are transparent. This works much better than letting 
    /// the window figure it out by itself using the content view's shape because the latter
    /// method makes rounded corners appear jagged while using `_cornerMask` respects any
    /// anti-aliasing in the mask image.
    @objc dynamic func _cornerMask() -> NSImage? {
        return cornerMask
    }

}

Then, in my NSWindowController subclass I set up the mask image for the view and the window:

class OverlayWindowController : NSWindowController {

    @IBOutlet weak var visualEffectView: NSVisualEffectView!

    override func windowDidLoad() {
        super.windowDidLoad()

        let maskImage = maskImage(cornerRadius: 18.0)
        visualEffectView.maskImage = maskImage
        if let window = window as? MaskedWindow {
            window.cornerMask = maskImage
        }
    }
}

I don’t know what Apple will do if you submit an app with that code to the App Store. You’re not actually calling any private API, you’re just overriding a method that happens to have the same name as a private method in AppKit. How should you know that there’s a naming conflict? 😉

Besides, this fails gracefully without you having to do anything. If Apple changes the way this works internally and the method just won’t get called, your window does not get the nice rounded corners, but everything still works and looks almost the same.

---

If you’re curious about how I found out about this method:

I knew that the OS X volume indication did what I want to do and I hoped that changing the volume like a madman resulted in noticeable CPU usage by the process that puts that volume indication on screen. I therefore opened Activity Monitor, sorted by CPU usage, activated the filter to only show “My Processes” and hammered my volume up/down keys.

It became clear that coreaudiod and something called BezelUIServer in /System/Library/LoginPlugins/BezelServices.loginPlugin/Contents/Resources/BezelUI/BezelUIServer did something. From looking at the bundle resources for the latter, it was evident that it is responsible for drawing the volume indication. (Note: that process only runs for a short time after it displays something.)

I then used Xcode to attach to that process as soon as it launched (Debug > Attach to Process > By Process Identifier (PID) or Name…, then enter “BezelUIServer”) and changed the volume again. After the debugger was attached, the view debugger let me take a look at the view hierarchy and see that the window was an instance of a NSWindow subclass called BSUIRoundWindow.

Using class-dump on the binary showed that this class is a direct descendant of NSWindow and only implements three methods, whereas one is - (id)_cornerMask, which sounded promising.

Back in Xcode, I used the Object Inspector (right hand side, third tab) to get the address for the window object. Using that pointer I checked what this _cornerMask actually returns by printing its description in lldb:

(lldb) po [0x108500110 _cornerMask]
<NSImage 0x608000070300 Size={37, 37} Reps=(
    "NSCustomImageRep 0x608000082d50 Size={37, 37} ColorSpace=NSCalibratedRGBColorSpace BPS=0 Pixels=0x0 Alpha=NO"
)>

This shows that the return value actually is an NSImage, which is the information I needed to implement _cornerMask.

If you want to take a look at that image, you can write it to a file:

(lldb) e (BOOL)[[[0x108500110 _cornerMask] TIFFRepresentation] writeToFile:(id)[@"~/Desktop/maskImage.tiff" stringByExpandingTildeInPath] atomically:YES]

To dig a bit deeper, you can use Hopper Disassembler to disassemble BezelUIServer and AppKit and generate pseudo code to see how the _cornerMask is implemented and used to get a clearer picture of how the internals work. Unfortunately, everything in regard to this mechanism is private API.