Is there any way to download csv file from “website button click” using Python?

by

First of all, you should understand that HTTP protocol based on requests. Final result of JavaScript execution will be formed HTTP request which let server respond with file content. You need to “reverse” web page, find how to create proper request and repeat it as similar as it can be done.

So, let’s try to do this step by step:

  1. Click right mouse button on element which execute download and press “Inspect element”
    enter image description here
  2. In source code you can see name of JavaScript function this element executes
    enter image description here
  3. Type the name of function in console without parentheses and click button which should appear near console return (This button will open this JavaScript function in source code)
    enter image description here
  4. In source code we see that function execute submit on HTML element which has id frmDownload. So, go back to “Inspector” tab and type this id into search box.
    enter image description here

  5. Now we found that this element is HTML form. This form send POST request to URL https://www.vcsprojectdatabase.org/services/publicViewServices/fetchProjectsExport with next data:

    searchTerm=
country=
sectoral_scope=0
recentProjects=
sort=projectId
dir=DESC
formatType=csv

    This information is enough to try repeat this request in Python.

Let’s write small script which form and send same request and save result into .csv file:

import requests

data = {
    "searchTerm": "",
    "country": "",
    "sectoral_scope": "0",
    "recentProjects": "",
    "sort": "projectId",
    "dir": "DESC",
    "formatType": "csv"
}

file = requests.post("https://www.vcsprojectdatabase.org/services/publicViewServices/fetchProjectsExport", data)

with open("res.csv", "wb+") as f:
    f.write(file.content)

Launch it and it … works. res.csv contains proper result.

BUT THAT’S NOT ALL. Usually everything is not so easy. To let our request look same as sent by browser we should take a look on request headers. To capture HTTP request from browser we can open “Network” tab:

enter image description here

Now let’s press download button on web page and download csv file. In requests table now we can see our post request. Click on it and take a look on “Headers” tab into “Request headers” section.

enter image description here

There’s Cookie header, which mostly in such as requests is not important and can be missed. But if you have some issues with request you should take a look on previous requests, find request with Set-Cookie header in server response and repeat it.

Let’s improve our script and copy important (Host, Content-Length, Connection we don’t include, cause Python requests module will add them automatically; DNT and Upgrade-Insecure-Requests are not necessary at all) headers from browser.

import requests

data = {
    "searchTerm": "",
    "country": "",
    "sectoral_scope": "0",
    "recentProjects": "",
    "sort": "projectId",
    "dir": "DESC",
    "formatType": "csv"
}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language":  "en-US,en;q=0.5",
    "Accept-Encoding": "gzip, deflate, br",
    "Referer": "https://www.vcsprojectdatabase.org/",
    "Content-Type": "application/x-www-form-urlencoded"
}

file = requests.post("https://www.vcsprojectdatabase.org/services/publicViewServices/fetchProjectsExport", data,
                     headers=headers)

with open("res.csv", "wb+") as f:
    f.write(file.content)

P.S. Don’t forget to ask website owner for permission 😉

More Related Contents:

  1. How to download an image with Python 3/Selenium if the URL begins with “blob:”?
  2. Running python on client side
  3. Web-scraping JavaScript page with Python
  4. Display data streamed from a Flask view as it updates
  5. Detect URLs in text with JavaScript
  6. WebDriver click() vs JavaScript click()
  7. JavaScript – Get Portion of URL Path
  8. Getting the return value of Javascript code in Selenium
  9. Convert blob URL to normal URL
  10. Regular expression for URL validation (in JavaScript)
  11. How can I execute a python script from an html button?
  12. Display the contents of a log file as it is updated
  13. Find the longest common starting substring in a set of strings [closed]
  14. Get javascript variable’s value in Django url template tag
  15. How to fetch URL of current Tab in my chrome extension using javascript
  16. Accessing Shadow DOM tree with Selenium
  17. How to escape a JSON string to have it in a URL?
  18. Is there any method to get the URL without query string?
  19. Accessing elements in the shadow DOM
  20. Returning JSON array from a Django view to a template
  21. Javascript: extract URLs from string (inc. querystring) and return array
  22. Call Javascript function from URL/address bar
  23. Multiple download links to one zip file before download javascript
  24. How to send a date directly as text to a calendar control with readonly attribute using Selenium through Python?
  25. Django authentication and Ajax – URLs that require login
  26. JavaScript timestamp to Python datetime conversion
  27. Disable images in Selenium Python
  28. Checking a Url in Jquery/Javascript
  29. FastAPI returns “Error 422: Unprocessable entity” when I send multipart form data with JavaScript Fetch API
  30. Cannot Upload File to FastAPI backend using JavaScript Fetch API in the frontend

Leave a Comment