Python实现抓取百度搜索结果页的网站标题信息
比如,你想采集标题中包含“58同城”的SERP结果,并过滤包含有“北京”或“厦门”等结果数据。
该Python脚本主要是实现以上功能。
其中,使用BeautifulSoup来解析HTML,可以参考我的另外一篇文章:Windows8下安装BeautifulSoup
代码如下:
代码如下:
__author__ = '曾是土木人' # -*- coding: utf-8 -*- #采集SERP搜索结果标题 import urllib2 from bs4 import BeautifulSoup import time #写文件 def WriteFile(fileName,content): try: fp = file(fileName,"a+") fp.write(content + "\r") fp.close() except: pass #获取Html源码 def GetHtml(url): try: req = urllib2.Request(url) response= urllib2.urlopen(req,None,3)#设置超时时间 data = response.read().decode('utf-8','ignore') except:pass return data #提取搜索结果SERP的标题 def FetchTitle(html): try: soup = BeautifulSoup(''.join(html)) for i in soup.findAll("h3"): title = i.text.encode("utf-8") if any(str_ in title for str_ in ("北京","厦门")): continue else: print title WriteFile("Result.txt",title) except: pass keyword = "58同城" if __name__ == "__main__": global keyword start = time.time() for i in range(0,8): url = "http://www.baidu.com/s?wd=intitle:"+keyword+"&rn=100&pn="+str(i*100) html = GetHtml(url) FetchTitle(html) time.sleep(1) c = time.time() - start print('程序运行耗时:%0.2f 秒'%(c))
相关推荐
rojyang 2020-05-20
ELEMENTS爱乐小超 2020-05-07
baynkbtg 2019-11-02
ksjlhy 2019-10-28
geminihr 2015-03-19
chang00 2015-06-17
LinSeeker 2019-09-07
nextwhy 2012-09-09
tongjianru 2011-05-18
Jarvan 2016-09-25
fadacai0 2016-04-18
deusrwvf 2017-06-21
AIOps智能运维 2015-05-25
magvwiz 2013-10-18
AIOps智能运维 2010-08-17
linqiang 2010-02-04
XSxing 2019-06-30