Code Monkey home page Code Monkey logo

wechatscope's Introduction

Wechatscope

每日数据存档

由于 Wechatscope(http://wechatscope.jmsc.hku.hk/) 项目只提供七天内的被删数据,设计定时任务每日获取近三天的数据,并去重后合并到同一数据库内。

git clone https://github.com/Terminus2049/Wechatscope.git

设置定时任务:

crontab -e

wechatscope.R 负责抓取最近三天的数据并合并到数据库内。可开启定时任务:

复制 0 */1 * * * cd ~/Wechatscope && R CMD BATCH wechatscope.R 到最后一行。程序会每小时运行一次。

表格化

app.R 利用 shiny 进行表格化。

在 R 语言环境中 shiny::runApp()

在线部署并自动更新

在 Linux 服务器上安装 shiny-server

cd /srv/shiny-server/
git clone https://github.com/Terminus2049/Wechatscope.git

同样开启定时任务:

crontab -e

复制 0 */1 * * * cd /srv/shiny-server/Wechatscope/ && R CMD BATCH wechatscope.R 到定时任务中。

然后就可以在 http://<ip>:3838/Wechatscope/ 查看。

注意

  1. 请安装合适的 R 版本,不要使用默认的 R 版本,但也不要安装最新的 R 3.5,应安装 R 3.4。安装教程:UBUNTU PACKAGES FOR R

  2. 下载并安装 shiny-server

  3. 需要安装 R 包 readrshinyhttrDTdata.table,如果内存过小,会编译失败,可通过 此教程 处理。

下载原始文件

安装 rust,编译后:

./target/release/wechat_scope_dl < ceninfo.csv

wechatscope's People

Contributors

terminusbot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wechatscope's Issues

在自己电脑上实现这个代码?

你好,不好意思打扰一下。刚入门R和Github,看readme有一些不明白的地方想请教一下。
就是如果我想要尝试在自己的电脑上实现这个项目的话,是应该先在linux和R上设置定时任务,然后再运行Wechatscope.R和设定每小时时抓取,再接着用app.R来表格化数据吗?

error 页面事后重新下载

wechatscope 原文经常处理偶尔会出现error,只需要删掉data文件夹中文档大小小一个数量级的文件即可。

先用 ls -Slr |head -n 200 找出出错的文件,然后用以下命令删掉。随后会再次下载。

ls -Sr |head -n 99 | xargs rm

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.