Python: Unable to download with selenium in webpage

I don’t see any major drawback in your code block as such. But here are a few recommendations through this Solution & the execution of this Automated Test Script:

1. This code works perfect in Off Market Hours. During Market Hours a lot of JavaScript & Ajax Calls are in play and handling those are beyond the scope of this Question.
2. You may consider checking for the the intended download directory first & if not available, create a new one. That code block for this functionality is in Windows style and works perfect on Windows platform.
3. Once you click on “Login” induce some wait for the HTML DOM to render properly.
4. When you want to see off the downloading process, you need to set certain more preferences in the FirefoxProfile as mentioned in my code below.
5. Always consider maximizing the browser window through browser.maximize_window()
6. When you start downloading you need to wait for sufficient amount of time to get the file completely downloaded.
7. If you are using browser.quit() at the end you don’t need to use browser.close()
8. You may consider to replace all the time.sleep() with either of ImplicitlyWait or ExplicitWait or FluentWait.

9. Here is your own code block with some simple tweaks in it:

   #!/usr/bin/python
   print "hello from python 2"
   
   import urllib2
   from selenium import webdriver
   from selenium.webdriver.common.keys import Keys
   import time
   from pyvirtualdisplay import Display
   import requests, zipfile, os    
   
   display = Display(visible=0, size=(800, 600))
   display.start()
   
   newpath="C:\\home\\vivvin\\shKLSE"
   if not os.path.exists(newpath):
       os.makedirs(newpath)    
   
   profile = webdriver.FirefoxProfile()
   profile.set_preference("browser.download.dir",newpath);
   profile.set_preference("browser.download.folderList",2);
   profile.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/zip");
   profile.set_preference("browser.download.manager.showWhenStarting",False);
   profile.set_preference("browser.helperApps.neverAsk.openFile","application/zip");
   profile.set_preference("browser.helperApps.alwaysAsk.force", False);
   profile.set_preference("browser.download.manager.useWindow", False);
   profile.set_preference("browser.download.manager.focusWhenStarting", False);
   profile.set_preference("browser.helperApps.neverAsk.openFile", "");
   profile.set_preference("browser.download.manager.alertOnEXEOpen", False);
   profile.set_preference("browser.download.manager.showAlertOnComplete", False);
   profile.set_preference("browser.download.manager.closeWhenDone", True);
   profile.set_preference("pdfjs.disabled", True);
   
   for retry in range(5):
       try:
           browser = webdriver.Firefox(profile)
           print "firefox"
           break
       except:
           time.sleep(3)
   time.sleep(1)
   
   browser.maximize_window()
   browser.get("https://www.shareinvestor.com/my")
   time.sleep(10)
   login_main = browser.find_element_by_xpath("//*[@href="https://stackoverflow.com/user/login.html"]").click()
   time.sleep(10)
   print browser.current_url
   username = browser.find_element_by_id("sic_login_header_username")
   password = browser.find_element_by_id("sic_login_header_password")
   print "find id done"
   username.send_keys("bkcollection")
   password.send_keys("123456")
   print "log in done"
   login_attempt = browser.find_element_by_xpath("//*[@type="submit"]")
   login_attempt.submit()
   browser.get("https://www.shareinvestor.com/prices/price_download.html#/?type=price_download_all_stocks_bursa")
   print browser.current_url
   time.sleep(20)
   dl = browser.find_element_by_xpath("//*[@href="https://stackoverflow.com/prices/price_download_zip_file.zip?type=history_all&market=bursa"]").click()
   time.sleep(900)
   
   browser.close()
   browser.quit()
   display.stop()
   
   zip_ref = zipfile.ZipFile(/home/vinvin/sh/KLSE, 'r')
   zip_ref.extractall(/home/vinvin/sh/KLSE)
   zip_ref.close()
   os.remove(zip_ref)
   

Let me know if this Answers your Question.