You can set an individual font size for the legend by adjusting the prop
keyword.
plot.legend(loc=2, prop={'size': 6})
This takes a dictionary of keywords corresponding to matplotlib.font_manager.FontProperties
properties. See the documentation for legend:
Keyword arguments:
prop: [ None | FontProperties | dict ] A matplotlib.font_manager.FontProperties instance. If prop is a dictionary, a new instance will be created with prop. If None, use rc settings.
It is also possible, as of version 1.2.1, to use the keyword fontsize
.