Maybe it is too late. But the only solution worked for me is installing from source if you do not want to use rosetta2 or moniconda
python3 -m pip install virtualenv
virtualenv -p python3.8 venv
source venv/bin/activate
pip install --upgrade pip
pip install numpy cython
git clone --depth 1 https://github.com/pandas-dev/pandas.git
cd pandas
python3 setup.py install