So I was very close. I had 2 problems, first a bad installation of the data.table package (Data table error could not find function “.”) caused an obscure error.
After having fixed that, I got closer an found that :
dt1[dt2, on=.(sp=sp, dbh>=dbh_min, dbh<=dbh_max), nomatch=0]
gave me what I wanted with a bad dbh column. Inverting the command with:
dt2[dt1, on=.(sp=sp, dbh_min<=dbh, dbh_max>=dbh)]
fixed the problem with only one useless extra column.