You can fix the problem on Mac OS X Mavericks and Yosemite by doing the following:
cd ~/.matplotlib/
fc-list # Should take a couple minutes.
After fc-list
finishes running you will be able to import the package using import matplotlib.pyplot as plt
. This is the github issue with the solution. It seems the bug is related to the font cache.