adizafri2000 / billscraper Goto Github PK

(personal problem) I don't want to have to develop using 2 different Python versions in my laptop ...

The section 'Installment' appears in the generated whatsapp message, even though there are no active installments counted for that month. To be edited to make that section not appear when there are no active installments to be counted.

When the automation executes, it should check whether the installment is going to be paid for its last month on the current run. If yes, after calculation or anywhere suitable, update the row of the installment to have its is_active column set to FALSE.

Suggested algo:

1. Retrieve all active installments.
2. Calculate projected end date/month by start_date + period.
3. If the result in 2 in falls in the current month, update the installment's record by setting the is_active column to FALSE.

Automation failing after supabase DB connection strings updated. Probably due to new kind of popup appearing on the TNB login page

Currently, when bill statements are calculated and generated, some of the items are actually charges for the previous month while the others are for the current month. For example:

• Electricity, water, unifi are all previous-month charged. This means that for a bill on December 2023, the calculated total for the mentioned items are actually charges for November 2023.
• Sewa (Rent) and washing machine installment are current-month charged. This means that for a bill on December 2023, the calculated total for the mentioned items are for December 2023.

This column is important especially when a resident leaves on the start of a month, it can be confusing to split the bills as we have to remember and find out which items of the bill are previous-month charges and which are current-month charges.

Suggested solution:

1. Update tables to have a new column is_previous_month_charge (or current month) to contain boolean values.
2. Previous-month charges like electricity, water & unifi will have its field set to TRUE while the others set to FALSE.

There is no application or use for this feature in the automation's calculations yet, but for manual checking it is very helpful to have such records.

In the case of elements not found etc., webdriver will save the page source via driver.page_source and upload to a bucket on Supabase. This is just to log current HTML state of the visited webpages on error encounters

1. Update the utility table to contain a nullable column to contain the fixed price of a utility item
2. Change 'automation' to become data retrieval from DB to entirely remove fixed price numbers in the code

After executing workflows numerous times and still failing, I resorted to saving screenshots to my Supabase bucket (that of which the screenshot logic has problems initially in the workflow, but now somewhat fixed). The reason the webdriver was not able to find any popups or login inputs was due to the cloudflare prompt page shown upon reaching TNB Portal.

Aim to release v1.0.0. Must contain at least:

• Web scraping automation for tnb & air selangor
• Database CRUD to supabase DB
• Screenshot/Image CRUD to supabase bucket
• Test scripts
• Whatsapp API service (whatsmeow + docker + render)
• Github actions to trigger monthly bill and keep-alives (supabase & render)

After several runs on Actions, it is safe to say that the automation will fail due to encountering 403 Forbidden error. Try this out.

Study and test:

1. Try to understand and setup whatsmeow
2. Build Docker image, test, and push to DockerHub if successful.
3. Create Render web service and deploy the Docker container and test
4. Change API URLs in current automation script and test on GitHub actions.
5. If all successful, create a GitHub actions to keep the Render service from stopping to prevent cold starts. May need to create a dummy API endpoint e.g api/health

Remove unnecessary steps from all workflow files after checking