写在最前面:以下数据均脱敏
from urllib import request import requests import urllib if __name__ ==
"__main__": # 接口的url session_requests = requests.session() data = {'username':
'11111111', 'password': '11111111'} requrl
='https://xxxxxx.com/xx/login?xxxxxxxxxxxxxxxxxxxxxxx' #登录请求url headers =
{'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:58.0) Gecko/20100101
Firefox/58.0'} # 发送请求 conn=requests.post(requrl,data,headers) #cookies =
conn.cookies.get_dict() print(conn.request.headers) newheaders =
conn.request.headers url = "http://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.htm"
#请求爬虫的url print(type(newheaders)) newheaders = dict(newheaders)
print(type(newheaders)) del newheaders['Accept-Encoding'] print(newheaders) req
= request.Request(url=url,headers=newheaders) rsp = request.urlopen(req) html =
rsp.read().decode("utf-8","ignore") print(html)
因为不把Accepe-Encoding去掉,会报错,或者乱码
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1:
invalid start byte
 所以转成字典,再去掉Accepe-Encoding

下面稍微解释一下吧,首先构造登录请求报文,包含用户名,密码,登录成功后获取cookie,使用cookie再去访问你要爬虫的页面,不然还是会被登录页面给拦截掉

能抓到你想访问的页面,接下来想干什么都可以了

关于cookie,其实你也可以手动F12看一下,Network里,Headers里有一个Request
Headers,其中最重要的就是你的cookie,保存了你本次登录的所有信息,每次重新登录都会改变