Thanks to @Darthbith and this post How to change the default browser used by the ipython/jupyter notebook in Linux? I was able to figure it out:
Step 1: To open Anaconda Prompt from the Start Menu and type
# for old notebook and JupyterLab < 3.0, or
jupyter notebook --generate-config
# for new nbclassic and JupyterLab >= 3.0
jupyter server --generate-config
This will generate the file ~/.jupyter/jupyter_notebook_config.py
(or jupyter_server_config.py
for nbclassic/new JupyterLab)
Step 2: Edit this file and change the following line (chrome is also is also in a local installation)
# for old notebook and JupyterLab < 3.0
c.NotebookApp.browser = u'C:/Home/AppData/Local/Google/Chrome/Application/chrome.exe %s'
# OR for new nbclassic and JupyterLab >= 3.0
c.ServerApp.browser = u'C:/Home/AppData/Local/Google/Chrome/Application/chrome.exe %s'