viniciusgava / itauscraper Goto Github PK

View Code? Open in Web Editor NEW

25.0 3.0 9.0 627 KB

Download Itaú exportable files using node and Puppeteer. OFX, OFC and TXT(CSV)

License: MIT License

JavaScript 80.50% Dockerfile 17.64% Makefile 1.86%

node puppeteer itau ofx statement extrato ofc csv txt

itauscraper's Introduction

Itau Scraper

Download Itaú exportable files using node and Puppeteer. Available file formats:

PDF
TXT - It's a CSV with semi-colon (DEFAULT)
OFX - Money 2000
OFC 1.0 - Money 1995 a Money 1999
OFC 1.06 - Money
OFC 1.06 - Quicken

Usage

node run.js --branch=0000 --account=00000-0 --password=000000 --days 5

Usage - Docker

Download this seccomp for chrome on docker. It will be used in the docker run:

wget https://raw.githubusercontent.com/jessfraz/dotfiles/master/etc/docker/seccomp/chrome.json

Execute:

docker run -v $(pwd):/home/node/itauscrapper/download \
    -rm \
    -u $UID:$GID \ 
    --security-opt seccomp=./chrome.json \
    -e BRANCH='0000' \
    -e ACCOUNT='00000-0' \
    -e PASSWORD='000000' \
    -e DAYS='000000' \
    viniciusgava/itauscraper:latest

Details about the dockerfile

This dockerfile has several fixes to be able to executed headless and safe. If you wanna know more, check the links bellow:

fixuid to fix user privileges inside the container when using volumes:
- https://github.com/boxboat/fixuid
fixing user privileges to enable sandbox and keep safety:

Help

Usage: node run.js [options]

Options:
      --help         Show help                                         [boolean]
      --version      Show version number                               [boolean]
  -b, --branch       Itaú branch number, format: 0000        [string] [required]
  -c, --account      Itaú account number, format: 00000-0    [string] [required]
  -n, --name         Itaú account name, format: Joao                    [string]
  -p, --password     Itaú account digital password(6 digits) [number] [required]
  -d, --days         Transaction log days
                          [number] [required] [choices: 3, 5, 7, 15, 30, 60, 90]
  -f, --file_format  File format to export
    [choices: "pdf", "txt", "ofx", "ofc10", "ofc106", "ofc106quicken"] [default:
                                                                          "txt"]
      --node_env     Node environment
        [choices: "development", "production", "docker"] [default: "production"]

In order to use multiple account holders names(Conta Conjunta), please, inform the option -name. Otherwise, it will not work.

Crontab

Create bash like this:

#!/bin/bash
SCRIPTPATH="$( cd "$(dirname "$0")" ; pwd -P )"

# print current date for debuging proposes
date

# try 5 times
n=0
until [ $n -ge 2 ]
do
    echo "trying $n"
    /usr/bin/docker run -v $SCRIPTPATH/download:/home/node/itauscrapper/download \ 
    --env-file "$SCRIPTPATH/env-configs" \
    --rm \
    -u $UID:$GID \
    --security-opt seccomp=./chrome.json \
    viniciusgava/viniciusgava/itauscraper:latest 2>&1 && break
    n=$[$n+1]
    sleep 15
done

Mac tip: You must pass docker full path to works at crontab /usr/local/bin/docker

add all env variables at env-configs. Example:

BRANCH=0000
ACCOUNT=00000-0
PASSWORD=000000
DAYS=5

DO NOT use quotation to define values on env files.

run crontab -e and add the follow cron. Example:

0 */4 * * * sh /home/username/automate/itauscraper/run.sh  >> /home/username/automate/itauscraper/log.log

The example bellow runs every 4 hours of everyday

You can generate a different crontab config on https://crontab-generator.org

itauscraper's People

Contributors

Stargazers

Watchers

Forkers

pheliperocha phsantiago32 ameisehaufen rtelesc williansmartins joaoalvarenga marcusvx gruberdev

itauscraper's Issues

Caixa Econômica Scraper

Hello, I love your project. In the past I helped with the implementation of a joint account. 1e5122d

Based entirely on this project, I created the scraper for Caixa Econômica. When you can, take a look and let me know what you think.

https://github.com/ludufre/caixascraper

[Bug] Puppeteer fails to find selector on headless mode

Running the itauscraper script results in a failure at the login step. Puppeteer is unable to locate the selector #agencia, throwing an Error: No node found for selector: #agencia.

Environment:

Node.js version: v18.17.1
Project version: v1.4.2

Steps to Reproduce:

Clone the repository and navigate to the project directory.
Install dependencies via npm install or yarn install.
Run the script

Observe the error:

/home/deck/Projects/itauscraper/node_modules/puppeteer/lib/cjs/puppeteer/common/assert.js:26
        throw new Error(message);
              ^

Error: No node found for selector: #agencia
    at assert (/home/deck/Projects/itauscraper/node_modules/puppeteer/lib/cjs/puppeteer/common/assert.js:26:15)
    at DOMWorld.type (/home/deck/Projects/itauscraper/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:315:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async stepLogin (/home/deck/Projects/itauscraper/itauscraper.js:16:3)
    at async scraper (/home/deck/Projects/itauscraper/itauscraper.js:199:3)

Additional information

Running on non-headless mode, it works as expected. This might be due to Itaú checking for automated tools like Pupeteer, and it seems an user-agent can be used to fix this error.

No node found for selector

Olá Vini, estou vendo aqui que deve ter mudado o HTML do site do Itau.
Eu gostaria de contribuir atualizando o código para corrigir este erro, olha abaixo:

Error: No node found for selector: #varejo > header > div.container > nav > ul > li > div > div > div:nth-child(1) > ul:nth-child(1) > li:nth-child(2) > a
at assert (/Users/macadmin/dev/projetos/itauscraper/node_modules/puppeteer/lib/helper.js:283:11)
at DOMWorld.hover (/Users/macadmin/dev/projetos/itauscraper/node_modules/puppeteer/lib/DOMWorld.js:386:5)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async stepExport (/Users/macadmin/dev/projetos/itauscraper/itauscraper.js:45:3)
at async scraper (/Users/macadmin/dev/projetos/itauscraper/itauscraper.js:171:3)
-- ASYNC --
at Frame. (/Users/macadmin/dev/projetos/itauscraper/node_modules/puppeteer/lib/helper.js:111:15)
at Page.hover (/Users/macadmin/dev/projetos/itauscraper/node_modules/puppeteer/lib/Page.js:1078:29)
at stepExport (/Users/macadmin/dev/projetos/itauscraper/itauscraper.js:45:14)
at async scraper (/Users/macadmin/dev/projetos/itauscraper/itauscraper.js:171:3)