基于scrapy+splash的网站爬虫
本示例基于scrapy和splash技术,实现的爬取去哪儿机票价格最优的一个Demo
===========================
docker run -p 8050:8050 scrapinghub/splash
pip install -r requirements.txt
DB_CONNECT = {
'db': 'test',
'user': 'root',
'passwd': '111111',
'host': '127.0.0.1',
'charset': 'utf8',
'use_unicode': True,
}
scrapy crawl qunar
https://doc.scrapy.org/en/1.3/intro/overview.html
https://github.com/scrapy-plugins/scrapy-splash