Python安装

1
2
3
4
5
6
7
8
9
10
11
12
class XXXXXXXSpider(scrapy.Spider):
	name = "......."
   allowed_domains = ["......com"]
   domains = "https://.......com"
   start_urls = [
        #"http://www.baidu.com"
        "............."
    ]
   user_agent='***************'
   def __init__(self,tag):
     self.tag = tag

使用

1.控制台运行时

scrapy crawl xxxxx -a tag=123

2.py文件运行

python3 xxx.py

1
2
3
4
5
import ...

process = CrawlerProcess(get_project_settings())
process.crawl(XXXXXSpider, 123)
process.start()

2.1

接收键盘输入

input 使用

$python xxx.py

$请选择::

123

1
2
3
4
5
6
import ...
tag = input("请选择::\n")
process = CrawlerProcess(get_project_settings())
process.crawl(XXXXXSpider, tag)
process.start()

2.2运行时传递参数

python 3 xxx.py 123

1
2
3
4
5
6
import sys
import ...
tag = argv[1]
process = CrawlerProcess(get_project_settings())
process.crawl(XXXXXSpider, tag)
process.start()