Listen to volume buttons in background service?

It is possible. Use code below (for newer Android versions, especially Marshmallow, see bottom of the answer):

public class SettingsContentObserver extends ContentObserver {
    int previousVolume;
    Context context;

    public SettingsContentObserver(Context c, Handler handler) {
        super(handler);
        context=c;

        AudioManager audio = (AudioManager) context.getSystemService(Context.AUDIO_SERVICE);
        previousVolume = audio.getStreamVolume(AudioManager.STREAM_MUSIC);
    }

    @Override
    public boolean deliverSelfNotifications() {
        return super.deliverSelfNotifications();
    }

    @Override
    public void onChange(boolean selfChange) {
        super.onChange(selfChange);

        AudioManager audio = (AudioManager) context.getSystemService(Context.AUDIO_SERVICE);
        int currentVolume = audio.getStreamVolume(AudioManager.STREAM_MUSIC);

        int delta=previousVolume-currentVolume;

        if(delta>0)
        {
            Logger.d("Ściszył!"); // volume decreased.
            previousVolume=currentVolume;
        }
        else if(delta<0)
        {
            Logger.d("Zrobił głośniej!"); // volume increased.
            previousVolume=currentVolume;
        }
    }
}

Then in your service onCreate register it with:

mSettingsContentObserver = new SettingsContentObserver(this,new Handler());
getApplicationContext().getContentResolver().registerContentObserver(android.provider.Settings.System.CONTENT_URI, true, mSettingsContentObserver );

Then unregister in onDestroy:

getApplicationContext().getContentResolver().unregisterContentObserver(mSettingsContentObserver);

Note that this example judges by change of media volume, if you want to use other volume, change it!

UPDATE:

Above method supposedly doesn’t work on Marshmallow, BUT there’s much better way now since MediaSession was introduced! So first you have to migrate your code to MediaController/MediaSession pattern and then use this code:

private VolumeProviderCompat myVolumeProvider = null;

myVolumeProvider = new VolumeProviderCompat(VolumeProviderCompat.VOLUME_CONTROL_RELATIVE, maxVolume, currentVolume) {
    @Override
    public void onAdjustVolume(int direction) {
        // <0 volume down
        // >0 volume up

    }
};

mSession.setPlaybackToRemote(myVolumeProvider);

Somehow volume button presses are detected even with screen off (just be sure to register proper media button intent receiver if applicable for your platform!)

UPDATE 2 since GalDude requested some more info on getting media MediaSession/MediaController. Sorry, but since I stopped using Java it will be in Kotlin:

lateinit var mediaSession: MediaSessionCompat // you have to initialize it in your onCreate method
val kontroler: MediaControllerCompat
 get() = mediaSession.controller // in Java it's just getController() on mediaSession

// in your onCreate/start method:
mediaSession = MediaSessionCompat(this, "YourPlayerName", receiver, null)
mediaSession.setFlags(MediaSessionCompat.FLAG_HANDLES_MEDIA_BUTTONS or MediaSessionCompat.FLAG_HANDLES_TRANSPORT_CONTROLS)
mediaSession.isActive = true
if (ratingIsWorking) // note: rating crashes on some machines you have to check it!
    mediaSession.setRatingType(RatingCompat.RATING_5_STARS)

mediaSession.setCallback(object : MediaSessionCompat.Callback() {
...
// here you have to implement what happens with your player when play/pause/stop/ffw etc. is requested - see exaples elsewhere
})

// onDestroy/exit method:
mediaSession.isActive = false
mediaSession.release()