I found this solution:
NotificationManager mNotificationManager = (NotificationManager) getSystemService(Context.NOTIFICATION_SERVICE);
mNotificationManager.setInterruptionFilter(NotificationManager.INTERRUPTION_FILTER_NONE);
It requires:
<uses-permission android:name="android.permission.ACCESS_NOTIFICATION_POLICY" />
Which as far as I can tell does not pop up a request dialog when doing requestPermissions(). It has to be granted through the settings menu Settings -> Sound & Notifcation -> Do Not Disturb Access.
This is on SDK 23 Marshmallow.