It doesn’t return what you expect because grouping happens before ordering, as reflected by the position of the clauses in the SQL statement. You’re unfortunately going to have to get fancier to get the rows you want. Try this:
SELECT *
FROM `table`
WHERE `id` = (
SELECT `id`
FROM `table` as `alt`
WHERE `alt`.`otheridentifier` = `table`.`otheridentifier`
ORDER BY `time` DESC
LIMIT 1
)
ORDER BY `time` DESC
LIMIT 3