Skip to main content

Simulate browser to solve https 403 error

As more and more website use https instead of http, we'll encounter 403 error when get content with requests modules. To solve it, the easiest way is simulating browser to access.

OS: Debian 9
Python: 3.5.3
Module: requests

setup headers as 1st step:
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3)\
           AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36'}

then add headers as parameter:
res = requests.get(xqurl, headers=headers)

Below is my example to get stock price in dictionary from a https website:
Labels:

Comments

Post a Comment

Popular posts from this blog

Upgrade BeautifulSoup4 to compatible with Python 3.6

After upgraded to Python 3.6 from 3.5, I encountered BeautifulSoup (bs4) error, such as 'ImportError: cannot import name 'HTMLParseError'. Seems there's compatible problem for BeautifulSoup 4 version. Solution: Upgrade beautifulsoup 4 version to 4.6.0 Commands: $ pip3 install --upgrade beautifulsoup4 Deep in: Install pip3 command: $ sudo apt install python3-pip
322 comments
Read more

Atom community packages recommendation for Python development

Below are Atom community packages I installed for Python development. They are covered most of functions u need. Take this post as an backup.  
Post a Comment
Read more

Solved: Pyperclip could not find a copy/paste mechanism for your system.

OS: Debian 9 Python version: 3.5 After import pyperclip module, encounter error message ' Pyperclip could not find a copy/paste mechanism for your system. '. Solution: install xsel utility $ sudo apt install xsel then reboot Python IDLE3, import pyperclip, solved. *************** Install pip/pyperclip: $ sudo apt install python3-pip $ sudo pip3 install pyperclip
7 comments
Read more