Dugtrio is Object Oriented Web Scraping.

Dugtrio uses

• Puppeteer for scraping,
• Knex as SQL Query Builder.(default RDBMS is MySQL)

npm install dugtrio

Define scraping target page detail:

const Dugtrio = require("dugtrio");
const BaseModel = Dugtrio.BaseModel;

class TwitterUser extends BaseModel {
  constructor() {
    super();
    this.url = 'https://twitter.com/(:username)?lang=ja',
    this.primary = "username"
    this.columns = {
      'followersCount': function() { return Array.from(document.getElementsByTagName('a')).filter(e => e.dataset['nav'] === 'followers')[0].getElementsByClassName('ProfileNav-value')[0].innerText.replace(',', ''); },
      'followingsCount': function() { return Array.from(document.getElementsByTagName('a')).filter(e => e.dataset['nav'] === 'following')[0].getElementsByClassName('ProfileNav-value')[0].innerText.replace(',', ''); },
    }
  }
}

const Diglett = Dugtrio.Diglett;
const username = 'Masayukiii';
(async(username) => {
  instance = new TwitterUser
  diglett = new Diglett(instance);
  await diglett.fetch(username).then(async(data) =>{ console.log(data) });
})(username);

I Strongrly recommend use this template, it supports variaus option.

npm run dugtrio generate

update database(MySQL) config on config/default.js.

{
  "db":{
    "host": "127.0.0.1",
    "user": "root",
    "password": "",
    "charset": "utf8",
    "database": "Dugtrio
  }
}

after fetching, you can save them. (Default data type is JSON.)

(async(username) => {
  instance = new TwitterUser
  diglett = new Diglett(instance);
  await diglett.fetch(username).then(async(data) =>{ await instance.save(data) });
})(username);

dugtrio supports to get seed data. (You know user list is oftern separate. So you have to get them to access page like user profile.)

I will introduce best practice code using dugtrio.

next time

suprisingly, dugtorio supports cli. so you can scrap on your terminal.

with -d, save data with -j, output to data/json/

default time sleep is about 5s. This behaviour is not like human. if you want to change it. you can edit Diglett code.

WARNING: you have to remember that Scraping so many times in short term is evil, I dont have any responsiblities about this.

if you want to login fetch more detail info, be careful to use fetch method many times because many login and logout action in short time is so strange.

dugrorio creates database, and table, columns by your definition automatically If you use BaseModel init method. And data type of primary key is string (with index), the others's data type is text. I guess you think it is bad, I agree you opinion. But it is so tired to define data type of each elements and scraping contens is text base.