How about:
var intersection = listOfLists
.Skip(1)
.Aggregate(
new HashSet<T>(listOfLists.First()),
(h, e) => { h.IntersectWith(e); return h; }
);
That way it’s optimized by using the same HashSet throughout and still in a single statement. Just make sure that the listOfLists always contains at least one list.