安科网

scrapy爬虫完整实例

Arvinzx

Arvinzx

2018-01-25

关注 关注

本文主要通过实例介绍了scrapy框架的使用，分享了两个例子，爬豆瓣文本例程 douban 和图片例程 douban_imgs ，具体如下。

例程1： douban

目录树

douban
--douban
 --spiders
  --__init__.py
  --bookspider.py
  --douban_comment_spider.py
  --doumailspider.py
 --__init__.py
 --items.py
 --pipelines.py
 --settings.py
--scrapy.cfg

CspidersCinit.py

# This package will contain the spiders of your Scrapy project
#
# Please refer to the documentation for information on how to create and manage
# your spiders.

bookspider.py

# -*- coding:utf-8 -*-
'''by sudo rm -rf http://imchenkun.com'''
import scrapy
from douban.items import DoubanBookItem


class BookSpider(scrapy.Spider):
  name = 'douban-book'
  allowed_domains = ['douban.com']
  start_urls = [
    'https://book.douban.com/top250'
  ]

  def parse(self, response):
    # 请求第一页
    yield scrapy.Request(response.url, callback=self.parse_next)

    # 请求其它页
    for page in response.xpath('//div[@class="paginator"]/a'):
      link = page.xpath('@href').extract()[0]
      yield scrapy.Request(link, callback=self.parse_next)

  def parse_next(self, response):
    for item in response.xpath('//tr[@class="item"]'):
      book = DoubanBookItem()
      book['name'] = item.xpath('td[2]/div[1]/a/@title').extract()[0]
      book['content'] = item.xpath('td[2]/p/text()').extract()[0]
      book['ratings'] = item.xpath('td[2]/div[2]/span[2]/text()').extract()[0]
      yield book

douban_comment_spider.py

# -*- coding:utf-8 -*-
import scrapy
from faker import Factory
from douban.items import DoubanMovieCommentItem
import urlparse
f = Factory.create()


class MailSpider(scrapy.Spider):
  name = 'douban-comment'
  allowed_domains = ['accounts.douban.com', 'douban.com']
  start_urls = [
    'https://www.douban.com/'
  ]

  headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3',
    'Connection': 'keep-alive',
    'Host': 'accounts.douban.com',
    'User-Agent': f.user_agent()
  }

  formdata = {
    'form_email': '你的邮箱',
    'form_password': '你的密码',
    # 'captcha-solution': '',
    # 'captcha-id': '',
    'login': '登录',
    'redir': 'https://www.douban.com/',
    'source': 'None'
  }

  def start_requests(self):
    return [scrapy.Request(url='https://www.douban.com/accounts/login',
                headers=self.headers,
                meta={'cookiejar': 1},
                callback=self.parse_login)]

  def parse_login(self, response):
    # 如果有验证码要人为处理
    if 'captcha_image' in response.body:
      print 'Copy the link:'
      link = response.xpath('//img[@class="captcha_image"]/@src').extract()[0]
      print link
      captcha_solution = raw_input('captcha-solution:')
      captcha_id = urlparse.parse_qs(urlparse.urlparse(link).query, True)['id']
      self.formdata['captcha-solution'] = captcha_solution
      self.formdata['captcha-id'] = captcha_id
    return [scrapy.FormRequest.from_response(response,
                         formdata=self.formdata,
                         headers=self.headers,
                         meta={'cookiejar': response.meta['cookiejar']},
                         callback=self.after_login
                         )]

  def after_login(self, response):
    print response.status
    self.headers['Host'] = "www.douban.com"
    yield scrapy.Request(url='https://movie.douban.com/subject/22266320/reviews',
               meta={'cookiejar': response.meta['cookiejar']},
               headers=self.headers,
               callback=self.parse_comment_url)
    yield scrapy.Request(url='https://movie.douban.com/subject/22266320/reviews',
               meta={'cookiejar': response.meta['cookiejar']},
               headers=self.headers,
               callback=self.parse_next_page,
               dont_filter = True)  #不去重

  def parse_next_page(self, response):
    print response.status
    try:
      next_url = response.urljoin(response.xpath('//span[@class="next"]/a/@href').extract()[0])
      print "下一页"
      print next_url
      yield scrapy.Request(url=next_url,
               meta={'cookiejar': response.meta['cookiejar']},
               headers=self.headers,
               callback=self.parse_comment_url,
               dont_filter = True)
      yield scrapy.Request(url=next_url,
               meta={'cookiejar': response.meta['cookiejar']},
               headers=self.headers,
               callback=self.parse_next_page,
               dont_filter = True)
    except:
      print "Next page Error"
      return

  def parse_comment_url(self, response):
    print response.status
    for item in response.xpath('//div[@class="main review-item"]'):
      comment_url = item.xpath('header/h3[@class="title"]/a/@href').extract()[0]
      comment_title = item.xpath('header/h3[@class="title"]/a/text()').extract()[0]
      print comment_title
      print comment_url
      yield scrapy.Request(url=comment_url,
               meta={'cookiejar': response.meta['cookiejar']},
               headers=self.headers,
               callback=self.parse_comment)

  def parse_comment(self, response):
    print response.status
    for item in response.xpath('//div[@id="content"]'):
      comment = DoubanMovieCommentItem()
      comment['useful_num'] = item.xpath('//div[@class="main-panel-useful"]/button[1]/text()').extract()[0].strip()
      comment['no_help_num'] = item.xpath('//div[@class="main-panel-useful"]/button[2]/text()').extract()[0].strip()
      comment['people'] = item.xpath('//span[@property="v:reviewer"]/text()').extract()[0]
      comment['people_url'] = item.xpath('//header[@class="main-hd"]/a[1]/@href').extract()[0]
      comment['star'] = item.xpath('//header[@class="main-hd"]/span[1]/@title').extract()[0]

      data_type = item.xpath('//div[@id="link-report"]/div/@data-original').extract()[0]
      print "data_type: "+data_type
      if data_type == '0':
        comment['comment'] = "\t#####\t".join(map(lambda x:x.strip(), item.xpath('//div[@id="link-report"]/div/p/text()').extract()))
      elif data_type == '1':
        comment['comment'] = "\t#####\t".join(map(lambda x:x.strip(), item.xpath('//div[@id="link-report"]/div[1]/text()').extract()))
      comment['title'] = item.xpath('//span[@property="v:summary"]/text()').extract()[0]
      comment['comment_page_url'] = response.url
      #print comment
      yield comment

doumailspider.py

# -*- coding:utf-8 -*-
'''by sudo rm -rf http://imchenkun.com'''
import scrapy
from faker import Factory
from douban.items import DoubanMailItem
import urlparse
f = Factory.create()


class MailSpider(scrapy.Spider):
  name = 'douban-mail'
  allowed_domains = ['accounts.douban.com', 'douban.com']
  start_urls = [
    'https://www.douban.com/'
  ]

  headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3',
    'Connection': 'keep-alive',
    'Host': 'accounts.douban.com',
    'User-Agent': f.user_agent()
  }

  formdata = {
    'form_email': '你的邮箱',
    'form_password': '你的密码',
    # 'captcha-solution': '',
    # 'captcha-id': '',
    'login': '登录',
    'redir': 'https://www.douban.com/',
    'source': 'None'
  }

  def start_requests(self):
    return [scrapy.Request(url='https://www.douban.com/accounts/login',
                headers=self.headers,
                meta={'cookiejar': 1},
                callback=self.parse_login)]

  def parse_login(self, response):
    # 如果有验证码要人为处理
    if 'captcha_image' in response.body:
      print 'Copy the link:'
      link = response.xpath('//img[@class="captcha_image"]/@src').extract()[0]
      print link
      captcha_solution = raw_input('captcha-solution:')
      captcha_id = urlparse.parse_qs(urlparse.urlparse(link).query, True)['id']
      self.formdata['captcha-solution'] = captcha_solution
      self.formdata['captcha-id'] = captcha_id
    return [scrapy.FormRequest.from_response(response,
                         formdata=self.formdata,
                         headers=self.headers,
                         meta={'cookiejar': response.meta['cookiejar']},
                         callback=self.after_login
                         )]

  def after_login(self, response):
    print response.status
    self.headers['Host'] = "www.douban.com"
    return scrapy.Request(url='https://www.douban.com/doumail/',
               meta={'cookiejar': response.meta['cookiejar']},
               headers=self.headers,
               callback=self.parse_mail)

  def parse_mail(self, response):
    print response.status
    for item in response.xpath('//div[@class="doumail-list"]/ul/li'):
      mail = DoubanMailItem()
      mail['sender_time'] = item.xpath('div[2]/div/span[1]/text()').extract()[0]
      mail['sender_from'] = item.xpath('div[2]/div/span[2]/text()').extract()[0]
      mail['url'] = item.xpath('div[2]/p/a/@href').extract()[0]
      mail['title'] = item.xpath('div[2]/p/a/text()').extract()[0]
      print mail
      yield mail

init.py

(此文件内无代码)

items.py

# -*- coding: utf-8 -*-
import scrapy


class DoubanBookItem(scrapy.Item):
  name = scrapy.Field()      # 书名
  price = scrapy.Field()      # 价格
  edition_year = scrapy.Field()  # 出版年份
  publisher = scrapy.Field()    # 出版社
  ratings = scrapy.Field()     # 评分
  author = scrapy.Field()     # 作者
  content = scrapy.Field()


class DoubanMailItem(scrapy.Item):
  sender_time = scrapy.Field()   # 发送时间
  sender_from = scrapy.Field()   # 发送人
  url = scrapy.Field()       # 豆邮详细地址
  title = scrapy.Field()      # 豆邮标题

class DoubanMovieCommentItem(scrapy.Item):
  useful_num = scrapy.Field()   # 多少人评论有用
  no_help_num = scrapy.Field()   # 多少人评论无用
  people = scrapy.Field()     # 评论者
  people_url = scrapy.Field()   # 评论者页面
  star = scrapy.Field()      # 评分
  comment = scrapy.Field()     # 评论
  title = scrapy.Field()      # 标题
  comment_page_url = scrapy.Field()# 当前页

pipelines.py

# -*- coding: utf-8 -*-


class DoubanBookPipeline(object):
  def process_item(self, item, spider):
    info = item['content'].split(' / ') # [法] 圣埃克苏佩里 / 马振聘 / 人民文学出版社 / 2003-8 / 22.00元
    item['name'] = item['name']
    item['price'] = info[-1]
    item['edition_year'] = info[-2]
    item['publisher'] = info[-3]
    return item


class DoubanMailPipeline(object):
  def process_item(self, item, spider):
    item['title'] = item['title'].replace(' ', '').replace('\\n', '')
    return item


class DoubanMovieCommentPipeline(object):
  def process_item(self, item, spider):
    return item

settings.py

# -*- coding: utf-8 -*-

# Scrapy settings for douban project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#   http://doc.scrapy.org/en/latest/topics/settings.html
#   http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#   http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'douban'

SPIDER_MODULES = ['douban.spiders']
NEWSPIDER_MODULE = 'douban.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
from faker import Factory
f = Factory.create()
USER_AGENT = f.user_agent()

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
  'Host': 'book.douban.com',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3',
  'Accept-Encoding': 'gzip, deflate, br',
  'Connection': 'keep-alive',
}
#DEFAULT_REQUEST_HEADERS = {
#  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#  'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#  'douban.middlewares.MyCustomSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#  'douban.middlewares.MyCustomDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#  'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
  #'douban.pipelines.DoubanBookPipeline': 300,
  #'douban.pipelines.DoubanMailPipeline': 600,
  'douban.pipelines.DoubanMovieCommentPipeline': 900,
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

scrapy.cfg

# Automatically created by: scrapy startproject
#
# For more information about the [deploy] section see:
# https://scrapyd.readthedocs.org/en/latest/deploy.html

[settings]
default = douban.settings

[deploy]
#url = http://localhost:6800/
project = douban

例程2： douban_imgs

目录树

douban_imgs
--douban
 --spiders
  --__init__.py
  --download_douban.py
 --__init__.py
 --items.py
 --pipelines.py
 --run_spider.py
 --settings.py
--scrapy.cfg

CspidersCinit.py

# This package will contain the spiders of your Scrapy project
#
# Please refer to the documentation for information on how to create and manage
# your spiders.

download_douban.py

# coding=utf-8
from scrapy.spiders import Spider
import re
from scrapy import Request
from douban_imgs.items import DoubanImgsItem


class download_douban(Spider):
  name = 'download_douban'

  default_headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, sdch, br',
    'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6',
    'Cache-Control': 'max-age=0',
    'Connection': 'keep-alive',
    'Host': 'www.douban.com',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36',
  }

  def __init__(self, url='1638835355', *args, **kwargs):
    self.allowed_domains = ['douban.com']
    self.start_urls = [
      'http://www.douban.com/photos/album/%s/' % (url)]
    self.url = url
    # call the father base function

    #super(download_douban, self).__init__(*args, **kwargs)

  def start_requests(self):

    for url in self.start_urls:
      yield Request(url=url, headers=self.default_headers, callback=self.parse)

  def parse(self, response):
    list_imgs = response.xpath('//div[@class="photolst clearfix"]//img/@src').extract()
    if list_imgs:
      item = DoubanImgsItem()
      item['image_urls'] = list_imgs
      yield item

init.py

(此文件内无代码)

items.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

import scrapy
from scrapy import Item, Field


class DoubanImgsItem(scrapy.Item):
  # define the fields for your item here like:
  # name = scrapy.Field()
  image_urls = Field()
  images = Field()
  image_paths = Field()

pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
from scrapy.pipelines.images import ImagesPipeline
from scrapy.exceptions import DropItem
from scrapy import Request
from scrapy import log


class DoubanImgsPipeline(object):
  def process_item(self, item, spider):
    return item


class DoubanImgDownloadPipeline(ImagesPipeline):
  default_headers = {
    'accept': 'image/webp,image/*,*/*;q=0.8',
    'accept-encoding': 'gzip, deflate, sdch, br',
    'accept-language': 'zh-CN,zh;q=0.8,en;q=0.6',
    'cookie': 'bid=yQdC/AzTaCw',
    'referer': 'https://www.douban.com/photos/photo/2370443040/',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36',
  }

  def get_media_requests(self, item, info):
    for image_url in item['image_urls']:
      self.default_headers['referer'] = image_url
      yield Request(image_url, headers=self.default_headers)

  def item_completed(self, results, item, info):
    image_paths = [x['path'] for ok, x in results if ok]
    if not image_paths:
      raise DropItem("Item contains no images")
    item['image_paths'] = image_paths
    return item

run_spider.py

from scrapy import cmdline
cmd_str = 'scrapy crawl download_douban'
cmdline.execute(cmd_str.split(' '))

settings.py

# -*- coding: utf-8 -*-

# Scrapy settings for douban_imgs project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#   http://doc.scrapy.org/en/latest/topics/settings.html
#   http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#   http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'douban_imgs'

SPIDER_MODULES = ['douban_imgs.spiders']
NEWSPIDER_MODULE = 'douban_imgs.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
# USER_AGENT = 'douban_imgs (+http://www.yourdomain.com)'

# Configure maximum concurrent requests performed by Scrapy (default: 16)
# CONCURRENT_REQUESTS=32

# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
# DOWNLOAD_DELAY=3
# The download delay setting will honor only one of:
# CONCURRENT_REQUESTS_PER_DOMAIN=16
# CONCURRENT_REQUESTS_PER_IP=16

# Disable cookies (enabled by default)
# COOKIES_ENABLED=False

# Disable Telnet Console (enabled by default)
# TELNETCONSOLE_ENABLED=False

# Override the default request headers:
# DEFAULT_REQUEST_HEADERS = {
#  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#  'Accept-Language': 'en',
# }

# Enable or disable spider middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
# SPIDER_MIDDLEWARES = {
#  'douban_imgs.middlewares.MyCustomSpiderMiddleware': 543,
# }

# Enable or disable downloader middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
# DOWNLOADER_MIDDLEWARES = {
#  'douban_imgs.middlewares.MyCustomDownloaderMiddleware': 543,
# }

# Enable or disable extensions
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
# EXTENSIONS = {
#  'scrapy.telnet.TelnetConsole': None,
# }

# Configure item pipelines
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
  'douban_imgs.pipelines.DoubanImgDownloadPipeline': 300,
}

IMAGES_STORE = 'D:\\doubanimgs'
#IMAGES_STORE = '/tmp'

IMAGES_EXPIRES = 90


# Enable and configure the AutoThrottle extension (disabled by default)
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html
# NOTE: AutoThrottle will honour the standard settings for concurrency and delay
# AUTOTHROTTLE_ENABLED=True
# The initial download delay
# AUTOTHROTTLE_START_DELAY=5
# The maximum download delay to be set in case of high latencies
# AUTOTHROTTLE_MAX_DELAY=60
# Enable showing throttling stats for every response received:
# AUTOTHROTTLE_DEBUG=False

# Enable and configure HTTP caching (disabled by default)
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
# HTTPCACHE_ENABLED=True
# HTTPCACHE_EXPIRATION_SECS=0
# HTTPCACHE_DIR='httpcache'
# HTTPCACHE_IGNORE_HTTP_CODES=[]
# HTTPCACHE_STORAGE='scrapy.extensions.httpcache.FilesystemCacheStorage'

scrapy.cfg

# Automatically created by: scrapy startproject
#
# For more information about the [deploy] section see:
# https://scrapyd.readthedocs.org/en/latest/deploy.html

[settings]
default = douban_imgs.settings

[deploy]
#url = http://localhost:6800/
project = douban_imgs

总结

以上就是本文关于scrapy爬虫完整实例的全部内容，希望对大家有所帮助。感兴趣的朋友可以继续参阅本站其他相关专题，如有不足之处，欢迎留言指出。感谢朋友们对本站的支持！

scrapy response xpath

Arvinzx

Arvinzx

0 关注 0 粉丝 0 动态

关注 关注

相关推荐

8_3 scrapy模拟登录人人网

在这个方法中发送post请求，没有重写这个方法基类Spider中的方法 start_request()默认是发送get请求。

javaraylu 2020-06-28

Python爬虫 - scrapy

start_requests:可以将遍历start_urls列表，将每一个列表元素进行get请求的发送。def file_path:指定文件路径。# Don‘t forget to add your pipeline to the ITEM_PIPELIN

ZHANGRENXIANG00 2020-06-28

用scrapy爬取图片

detail_list = response.xpath(‘//*[@id="main"]/div[3]/ul/li/a/@href‘).extract(). yield scrapy.Request(detail_url, callb

Catastrophe 2020-06-26

scrapy中使用selenium+webdriver获取网页源码，爬取简书网站

由于简书中一些数据是通过js渲染出来的，所以通过正常的request请求返回的response源码中没有相关数据，# 加载chrome驱动，若chromedriver.exe文件和python.exe 在相同目录下，可以省略executable_path=

andrewwf 2020-05-30

三、Scrapy Shell

　　Scrapy终端是一个交互终端，可以在未启动spider的情况下尝试及调试代码，也可以用来测试XPath或CSS表达式，查看它们的工作方式，方便在爬取的网页中提取数据。　　如果安装了 IPython ，Scrapy终端将使用 IPython 。IPyt

ZHANGRENXIANG00 2020-05-11

python爬虫：scrapy自定义item

item[‘body‘]=response.xpath("//div[@class=‘xx‘]/text()").get()

Catastrophe 2020-05-07

在Scrapy中如何利用Xpath选择器从网页中采集目标数据——详细教程（下篇）

前几天给大家分享了在Scrapy中如何利用Xpath选择器从网页中采集目标数据——详细教程（上篇），没来得及上车的小伙伴可以戳进去看看，今天继续上篇的内容往下进行。至此，关于Xpath表达式的具体应用教程先告一段落。

paleyellow 2020-10-25

在Scrapy中如何利用Xpath选择器从网页中采集目标数据——详细教程（上篇）

上一篇文章我们讲述了网页结构和Xpath表达式语法知识，感兴趣的小伙伴可以戳这篇文章：网页结构的简介和Xpath语法的入门教程。我们了解到Xpath表达式最好是通过自己进行网页分析和针对性的选取唯一性的标签进行定位，可以提高提取效率，而且还不容易出错。

baifanwudi 2020-10-25

Python爬虫 - scrapy框架的基本操作

scrapy异步的爬虫框架。c. 进入下载目录，执行 pip3 install Twisted?只可以将parse方法的返回值存储到指定后缀的文本文件中。# #xpath在进行数据提取时，返回的不再是字符串而是一个Selector对象，想要

fangjack 2020-06-25

Scrapy数据解析和持久化

c. 进入下载目录，执行 pip3 install Twisted?- 5.在管道类中的process_item方法负责接收item对象，然后对item进行任意形式的持久化存储。- 如果管道文件中定义了多个管道类，爬虫类提交的item会给到优先级最高的管道

andrewwf 2020-05-08

如何利用Scrapy爬虫框架抓取网页全部文章信息（上篇）

首先我们理一下爬取思路，大致思想是：当获取到第一个页面的URL之后，尔后将第二页的URL发送给Scrapy，让Scrapy去自动下载该网页的信息，之后通过第二页的URL继续获取第三页的URL，由于每一页的网页结构是一致的，所以通过这种方式如此反复进行迭代，

andrewwf 2020-11-11

一分钟搞定Scrapy分布式爬虫、队列和布隆过滤器

你知道最快的方法是什么吗？一分钟真的能 开发好或者修改出 一个分布式爬虫吗？话不多说，先让我们看看怎么实践，再详细聊聊细节。如果你没有所需要的运行条件，你可以启动两个 Docker 镜像进行测试 :. 如果你有一个现成的爬虫，可以跳过这个 Step，直接到

Arvinzx 2020-10-28

一篇文章教会你理解Scrapy网络爬虫框架的工作原理和数据采集过程

今天小编给大家详细的讲解一下Scrapy爬虫框架，希望对大家的学习有帮助。Scrapy是一个使用Python编程语言编写的爬虫框架，任何人都可以根据自己的需求进行修改，并且使用起来非常的方便。它可以应用在数据采集、数据挖掘、网络异常用户检测、存储数据等方面

CycloneKid 2020-10-27

手把手教你进行Scrapy中item类的实例化操作

首先去parse_detail函数下对其进行实例化，实例化的方法也十分简单，如下图所示。其中，目标字段可以参考items.py中定义的item，这样可以加快填充的速度。

heyboz 2020-10-21

如何改造 Scrapy 从而实现多网站大规模爬取？

Scrapy 框架默认是用来开发定向爬虫的。一般情况下，在 spiders 文件夹下面的一个.py 文件对应了一个网站的爬取。但还有另外一种爬虫，它不会拘泥于提取页面上的特定文字，而是关注如何并行爬取非常多的网站。这种爬虫，一般是从若干个种子网址开始爬。但

wumxiaozhu 2020-10-16

二十六、Scrapy自定义命令

from scrapy.commands import ScrapyCommand??class Command(ScrapyCommand): requires_project = True?

ZHANGRENXIANG00 2020-07-27

scrapy 管理部署的爬虫项目的python类

还有部分api接口没有添加进来,可以参照官方的文档添加.

hilary0 2020-07-05

分布式爬虫部署 基于scrapy和scrapy-redis

安装一个scrapy-redis的组件。原生的scrapy是不可以实现分布式爬虫，必须要让scrapy结合着scrapy-redis组件一起实现分布式。scrapy-redis组件的作用可以给原生的scrapy框架提供可以被共享的管道和调度器。结合配置文件

zhangll00 2020-07-05

Scrapy爬虫

二、Scrapy爬虫框架结构

ZHANGRENXIANG00 2020-06-27

scrapy基本知识

Scrapy爬虫框架主要由5个部分组成，分别是：Scrapy Engine，Scheduler（调度器），Downloader（下载器），Spiders（蜘蛛），Item Pipeline。爬取过程是Scrapy引擎发送请求，之后调度器把初始URL交给下载

Catastrophe 2020-06-26
Arvinzx

Arvinzx

W3CSchool教程
HTML 教程
CSS 教程
Bootstrap 教程
Javascript 教程
jQuery 教程
后端教程
C 教程
Java 教程
PHP 教程
Python 教程
Go 教程
移动开发
Android 教程
Swift 教程
Kotlin 教程
jQuery Mobile 教程
ionic 教程
关于我们
新闻动态
联系方式
招聘英才
安科实验室
帮助与反馈

安科网(Ancii)，中国第一极客网

Copyright © 2013 - 2019 Ancii.com

京ICP备18063983号 京公网安备11010802014868号