I had this exact problem recently and used
apt-get install python-numpy
This adds numpy to your system python interpreter. I may have had to do the same for matplotlib. To use in a virtualenv, you have to create your environment using the
--system-site-packages
option