How can a Swing WindowListener veto JFrame closing

The right way is set JFrame.setDefaultCloseOperation to DO_NOTHING_ON_CLOSE when the window is created. And then just calling setVisible(false) or dispose() when your user accepts the close, or doing nothing when the close isn’t accepted.

The whole purpose of JFrame.setDefaultCloseOperation is only to prevent the need to implement WindowListeners for the most simple actions. The actions performed by these default close operations are very simple.

EDIT:

I’ve added the solution I’m describing. This assumes you want the frame to be fully deleted.

frame.setDefaultCloseOperation(setDefaultCloseOperation);
frame.addWindowListener(new SaveOnCloseWindowListener(fileState));
...

public class SaveOnCloseWindowListener extends WindowAdapter {
    private final FileState fileState;

    public SaveOnCloseWindowListener(FileState fileState) {
        this.fileState = fileState;
    }

    public void windowClosing(WindowEvent e) {
        if (fileState.onQuit())
            frame.dispose();
    }
}