Try
H[cbind(seq_len(nrow(H)), P)]
## [1] 0.6733731 0.7396847 0.5953580
Here we are indexing by consecutive rows and columns indicated in P
Regarding your question, so the reason H[, P]
returns a matrix is because you are telling R:
select all rows in columns: 2, 1, 2 from matrix “H”
thus the result that you are getting is a matrix with identical first and third columns.