Scrapy Tool Greapy and SpiderKeeper

ScrapyToolGreapyandSpiderKeeper

OnmyUbuntuMasterVirtualMachine

CheckPIPVersion

>pip--version

pip18.1from/home/carl/.pyenv/versions/3.6.0/lib/python3.6/site-packages/pip(python3.6)

InstalltheGerapy

>pipinstallgerapy

CheckVersion

>gerapy-version

0.8.5

Intheworkingdirectory

>pwd

/home/carl/work

Initiateandcreateaworkingdirectory/home/carl/work/gerapy

>gerapyinit

Initiatethedatabase

>cdgerapy/

>gerapymigrate

Thenthereisdb.sqlite3

StarttheService

>gerapyrunserver

Performingsystemchecks...

Systemcheckidentifiednoissues(0silenced).

March04,2019-06:44:22

Djangoversion2.1.7,usingsettings'gerapy.server.server.settings'

Startingdevelopmentserverathttp://127.0.0.1:8000/

QuittheserverwithCONTROL-C.

Opentotheworld

>gerapyrunserver0.0.0.0:8000

Performingsystemchecks...

Systemcheckidentifiednoissues(0silenced).

March04,2019-06:45:57

Djangoversion2.1.7,usingsettings'gerapy.server.server.settings'

Startingdevelopmentserverathttp://0.0.0.0:8000/

Thenwecanvisitthepage

http://ubuntu-master:8000/#/client

IntheUIwecanaddclientsanddoingotherthings.

InstallSpiderKeeper

>pipinstallspiderkeeper

>mkdirspiderkeeper

>cdspiderkeeper/

Starttheweb

>spiderkeeper--server=http://localhost:6800

SinceGerapyhavenoAuthentication,weneedtouseSSH

>ssh-L8010:localhost:8010root@ubuntu-master-N

http://localhost:8010

HereisthemajorinformationtosetupDockerService

HereistheDockerfilethathaveallthesteps

#SetupGerapyinDocker

#PrepretheOS

FROMcentos/python-36-centos7

MAINTAINERYiyiKang<[email protected]>

#setuser

USERroot

#installthesoftwarea

#upgradepip

RUNpip3install--upgradepip

#installgerapy

RUNpip3installgerapy

#initgerapy

RUNmkdir-p/tool/

WORKDIR/tool/

RUNgerapyinit

WORKDIR/tool/gerapy/

RUNgerapymigrate

#setuptheapp

EXPOSE8000

RUNmkdir-p/app/

ADDstart.sh/app/

WORKDIR/app/

CMD["./start.sh"]

HereistheMakefilewhichhaveallthesteps

IMAGE=sillycat/gerapy

TAG=sillycat-gerapy-1.0

NAME=sillycat-gerapy-1.0

docker-context:

build:docker-context

dockerbuild-t$(IMAGE):$(TAG).

run:

dockerrun-d-p127.0.0.1:8010:8000--restartalways--name$(NAME)$(IMAGE):$(TAG)

debug:

dockerrun-ti-p8010:8000--name$(NAME)$(IMAGE):$(TAG)/bin/bash

clean:

dockerstop${NAME}

dockerrm${NAME}

logs:

dockerlogs${NAME}

publish:

dockerpush${IMAGE}:${TAG}

fetch:

dockerpull${IMAGE}:${TAG}

Hereisthestartcommandstart.sh

#!/bin/sh-ex

#starttheservice

cd/tool/gerapy/

gerapyrunserver0.0.0.0:8000

HereisthereadmehowtoaccesstheUI

Gerapyusetolistscrapyd.

##howtobuild

>makebuild

##howtorun

>makerun

##howotstop

>makeclean

##WebUI

ssh-L8010:localhost:8010carl@ubuntu-master-N

http://ubuntu-master:8010/

##

References:

https://blog.csdn.net/fengltxx/article/details/79894839

https://www.jianshu.com/p/f3447c90a0ec

https://github.com/Gerapy/Gerapy

https://github.com/DormyMo/SpiderKeeper

https://askubuntu.com/questions/112177/how-do-i-tunnel-and-browse-the-server-webpage-on-my-laptop