Scraping Javascript driven web pages with PyQt4 – how to access pages that need authentication?

by

I figured it out. Here’s what I ended up with in case it can help someone else.

#!/usr/bin/python
# -*- coding: latin-1 -*-
import sys
import base64
from PyQt4.QtGui import *
from PyQt4.QtCore import *
from PyQt4.QtWebKit import *
from PyQt4 import QtNetwork

class Render(QWebPage):
  def __init__(self, url):
    self.app = QApplication(sys.argv)

    username="username"
    password = 'password'

    base64string = base64.encodestring('%s:%s' % (username, password))[:-1]
    authheader = "Basic %s" % base64string

    headerKey = QByteArray("Authorization")
    headerValue = QByteArray(authheader)

    url = QUrl(url)
    req = QtNetwork.QNetworkRequest()
    req.setRawHeader(headerKey, headerValue)
    req.setUrl(url)

    QWebPage.__init__(self)
    self.loadFinished.connect(self._loadFinished)


    self.mainFrame().load(req)
    self.app.exec_()

  def _loadFinished(self, result):
    self.frame = self.mainFrame()
    self.app.quit()

def main():
    url="http://www.google.com"
    r = Render(url)
    html = r.frame.toHtml()

More Related Contents:

  1. PyQt showing video stream from opencv
  2. PyQt: No error msg (traceback) on exit
  3. Scrape multiple urls using QWebPage
  4. QLabel is not updated unless the mainWindow is unfocused
  5. How to add a QVideoWidget in Qt Designer?
  6. How to display a Pandas data frame with PyQt5/PySide2
  7. Is there a difference between QFileDialog strings in PyQt4 and PyQt5?
  8. How to add static(html, css, js, etc) files in pyinstaller to create standalone exe file?
  9. Install PyQt5 5.14.1 on Linux
  10. PyQt4 to PyQt5 how?
  11. Customising code of Qt designer widget?
  12. How do I assert the identity of a PyQt5 signal?
  13. Multiple inheritance metaclass conflict
  14. Custom Titlebar with frame in PyQt5
  15. time.sleep() and BackGround Windows PyQt5
  16. Pyqt5 qthread + signal not working + gui freeze
  17. QPushButton.clicked() fires twice when autowired using .ui form
  18. clicked.connect() Error
  19. PyQt4 to PyQt5 -> mainFrame() deprecated, need fix to load web pages
  20. Why do I need to decorate connected slots with pyqtSlot?
  21. QtWidgets.QFileDialog.getOpenFileName returns a Tuple
  22. PyQt5 failing import of QtGui
  23. How can I check if a keyboard modifier is pressed (Shift, Ctrl, or Alt)?
  24. How Can I Update a Qml Object’s Property from my Python file?
  25. PyQt5 – Show virtual keyboard
  26. Access a QLCDNumber object from a external function
  27. Run command with PyQt5 and getting the stdout and stderr
  28. pylint can’t find QWidget and QApplication
  29. ImportError: No module named ‘resource_rc’
  30. PyQt QProgressBar not working when I use it with Selenuim

Leave a Comment