把起始URL放到Redis中去
把起始URL放到redis中去 from scrapy_redis.spiders import RedisSpider # 继承RedisSpider class ChoutiSpider(RedisSpider): name = ‘chouti‘ allowed_domains = [‘chouti.com‘] 配置文件: REDIS_START_URLS_KEY = ‘%(name)s:start_urls‘ REDIS_START_URLS_AS_SET = False 下面是源码 # use_set = self.settings.getbool(‘REDIS_START_URLS_AS_SET‘, defaults.START_URLS_AS_SET) # fetch_one = self.server.spop if use_set else self.server.lpop
相关推荐
王道革 2020-11-25
wangdonghello 2020-11-03
Langeldep 2020-11-16
chenhualong0 2020-11-16
聚合室 2020-11-16
koushr 2020-11-12
MRFENGG 2020-11-11
guoyanga 2020-11-10
fackyou00 2020-11-10
Orangesss 2020-11-03
dongCSDN 2020-10-31
rainandtear 2020-10-30
Quietboy 2020-10-30
liuyulong 2020-10-29
fansili 2020-10-29
温攀峰 2020-10-23
jackbon 2020-10-19
kaixinfelix 2020-10-04