电商平台价格监控
- 前端：Boostrap+Jquery
- 后端：Python+Flask
- 爬虫：requests+beautifulsoup4+PyExecJS
- 代理池：https://github.com/0xHJK/Proxies 简易实现
$ python server.py
或者
$ nohup python server.py &
soup = BeautifulSoup(response.text, 'html.parser')
name = soup.find(class_='sku-name')
[
{
"cbf": "0",
"id": "J_4311178",
"m": "800.00",
"op": "399.00",
"p": "219.00"
}
]
- 爬取历史价格
http://tool.manmanbuy.com/history.aspx?DA=1&action=gethistory&url=http://item.jd.com/4311178.html&bjid=&spbh=&cxid=&zkid=&w=951&token=hbo5aec0bfd10dad3242ed1e5614d57c9b2f349auz8t
(根据ID从
MMM_GET_TOKEN.js获取token)
{
"datePrice": "[1508774400000,629.00,\"\"],......"
......
}
Goods:
| 属性 | 解释 | 类型 | 备注 |
|---|---|---|---|
| id | 商品编号 | String(20) | primary_key=True |
| dname | 自定义商品名 | TEXT | nullable=True |
| want | 期望价格 | FLOAT | nullable=False |
| status | 运行状态 | BOOLEAN | nullable=False |
修改config.cfg的参数，可以上网搜索怎么申请qq邮箱smtp密码~，记得开放25端口
钉钉token获取方式自行搜索
[mail]
host = smtp.qq.com
port = 25
user = [email protected]
pass = xxxxxxxxxxxxxxxx
sender = [email protected]
[dingding]
webhook = https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxxxxxxx
[select]
# 邮件和钉钉通知二选一
option = dingding
配置文件放在/data/jd-spider/config目录下，配置好后挂载到容器中
docker run --name jdspider -p 5858:5858 -v /data/jd-spider/config:/usr/src/app/web/config -d yimijianfang/jd-spider
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent