Dynamic dropdown doesn’t populate with auto suggestions on https://www.nseindia.com/ when values are passed using Selenium and Python

by

I executed your test adding a few tweaks and ran the test as follows:

  • Code Block:

    from selenium import webdriver        
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.nseindia.com/companies-listing/corporate-filings-actions')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@id='Corporate_Actions_equity']//input[@placeholder="Company Name or Symbol"]"))).send_keys("Reliance")

  • Observation: Similar to your observation, I have hit the same roadblock with no results as follows:

nseindia

Deep Dive

It seems the click() on the element with text as Get Data does happens. But while inspecting the DOM Tree of the webpage you will find that some of the <script> tag refers to JavaScripts having keyword akam. As an example:

  • <script type="text/javascript" src="https://www.nseindia.com/akam/11/3b383b75" defer=""></script>
  • <noscript><img src="https://www.nseindia.com/akam/11/pixel_3b383b75?a=dD02ZDMxODU2ODk2YTYwODA4M2JlOTlmOGNkZTY3Njg4ZWRmZjE4YmMwJmpzPW9mZg==" style="visibility: hidden; position: absolute; left: -999px; top: -999px;" /></noscript>

Which is a clear indication that the website is protected by Bot Manager an advanced bot detection service provided by Akamai and the response gets blocked.

Bot Manager

As per the article Bot Manager – Foundations:

akamai_detection

Conclusion

So it can be concluded that the request for the data is detected as being performed by Selenium driven WebDriver instance and the response is blocked.

References

A couple of documentations:

tl; dr

A couple of relevant discussions:

More Related Contents:

  1. how to scrape web page that is not written directly using HTML, but is auto-generated using JavaScript? [closed]
  2. WebDriverWait not working as expected
  3. Difference between text and innerHTML using Selenium
  4. selenium with scrapy for dynamic page
  5. How can I download a file on a click event using selenium?
  6. Python: find_element_by_css_selector
  7. ‘list’ object has no attribute ‘get_attribute’ while iterating through WebElements
  8. Scrape website with dynamic mouseover event
  9. How to click a link by text with No Text in Python
  10. Selenium Python: How to web scrape the element text
  11. Selenium-Debugging: Element is not clickable at point (X,Y)
  12. unable to call firefox from selenium in python on AWS machine
  13. Fetch all href link using selenium in python
  14. Python selenium multiprocessing
  15. Scraping dynamic data selenium – Unable to locate element
  16. Python selenium error when trying to launch firefox
  17. Speeding up beautifulsoup
  18. Python: Disable images in Selenium Google ChromeDriver
  19. “Failed to decode response from marionette” message in Python/Firefox headless scraping script
  20. Scraping Data from a website which uses Power BI – retrieving data from Power BI on a website
  21. Capture AJAX response with selenium and python
  22. How can I bypass the Google CAPTCHA with Selenium and Python?
  23. How to run Scrapy from within a Python script
  24. Downloading a file at a specified location through python and selenium using Chrome driver
  25. Switching into new window using Selenium after button click
  26. How to get text from span tag in BeautifulSoup
  27. How to properly stop phantomjs execution
  28. IOError: [Errno 13] Permission denied: ‘geckodriver.log when running Python/Selenium
  29. Passing cookies while logging in
  30. csv.writer writing each character of word in separate column/cell

Leave a Comment