Drag and drop items in RecyclerView with GridLayoutManager

There is actually a better way to achieve this. You can use some of the RecyclerView‘s “companion” classes:

ItemTouchHelper, which is

a utility class to add swipe to dismiss and drag & drop support to RecyclerView.

and its ItemTouchHelper.Callback, which is

the contract between ItemTouchHelper and your application

// Create an `ItemTouchHelper` and attach it to the `RecyclerView`
ItemTouchHelper ith = new ItemTouchHelper(_ithCallback);
ith.attachToRecyclerView(rv);

// Extend the Callback class
ItemTouchHelper.Callback _ithCallback = new ItemTouchHelper.Callback() {
    //and in your imlpementaion of
    public boolean onMove(RecyclerView recyclerView, RecyclerView.ViewHolder viewHolder, RecyclerView.ViewHolder target) {
        // get the viewHolder's and target's positions in your adapter data, swap them
        Collections.swap(/*RecyclerView.Adapter's data collection*/, viewHolder.getAdapterPosition(), target.getAdapterPosition());
        // and notify the adapter that its dataset has changed
        _adapter.notifyItemMoved(viewHolder.getAdapterPosition(), target.getAdapterPosition());
        return true;
    }

    @Override
    public void onSwiped(RecyclerView.ViewHolder viewHolder, int direction) {
        //TODO    
    }

    //defines the enabled move directions in each state (idle, swiping, dragging). 
    @Override
    public int getMovementFlags(RecyclerView recyclerView, RecyclerView.ViewHolder viewHolder) {
        return makeFlag(ItemTouchHelper.ACTION_STATE_DRAG,
                ItemTouchHelper.DOWN | ItemTouchHelper.UP | ItemTouchHelper.START | ItemTouchHelper.END);
    }
};

For more details check their documentation.