robbrad / ukbincollectiondata Goto Github PK

Add issue templates to help better manage the adding of councils, I have provided below an example

Example

Code

name: Council Request
description: Request for a council to be added to the repository
labels: ['Class: enhancement']

body:
    - type: input
      id: council
      attributes:
          label: Name of Council
          description: What council are you wishing to be added
          placeholder: e.g. Huntingdon District Council
      validations:
          required: true
    - type: input
      id: postcode
      attributes:
          label: Example Postcode
          description: Please provide and example postcode for the area
          placeholder: e.g. PE7 3YQ
      validations:
          required: true
    - type: textarea
      id: extra
      attributes:
          label: Additional Information
          description: Add any other information here
          placeholder: Links to the councils site, information you have already gathered

https://www.eastriding.gov.uk/environment/bins-rubbish-recycling/bins-and-collections/bin-collection-dates/

https://github.com/Tmp2k/wigan-council-api

GET /bins?postcode=WN1+2RW

please add torbay, Thanks!

https://www.torbay.gov.uk/recycling/bin-collections/

http://echo2.ectrecycling.co.uk/EchoServiceLookup/lookup.aspx?cid=33

I have linked below my postman collections. This will need to use soup as far as I can see there is no API

Postcode - PE73GD
UPRN     - 10012048674

Hi there, i've been wanting to do this for a while but I don't think my skills are up to it, any help with Kingston Upon Thames would be greatly appreciated!

https://www.kingston.gov.uk/when-is-my-collection
They are also listed on here: https://maps.kingston.gov.uk/propertyServices/InMyArea

Thanks!

Interesting one, doesn't look like they have an API they require a log in to view your details or at least I thought
https://my.crawley.gov.uk/
looking at the page source you don't have to be logged in once you have a customer info via the page source, I tested this in an incognito window so it should be able to be scraped if someone with the knowhow wants to work with me to get this one working, I just don't want to publicly post my customer info

https://www.northlincs.gov.uk/bins-waste-and-recycling/bin-collection-dates/

Enter Post Code: DN17 2TP

General code coverage improvement for the common.py file

Hi, I have a working script for Charnwood Borough Council and would like to raise a PR to get this into the repo, however I have no access to create a branch. Could you look at the access policies?

Thanks

Name of Council

North East Lincolnshire

Example Postcode

DN36 4DQ

Additional Information

https://www.nelincs.gov.uk/?s=dn36+4dq

I used to be able to scrape the wesbite but they've made it a little more difficult and my know-how is lacking.

It'd be great to be able to have the latest data from Fenland please.

https://www.fenland.gov.uk/article/13114/Find?uprn=200002981143&lat=52.665569590474&lng=0.177905443639&postcode=PE13+3SL&line1=20&rad=5m&layers=2%2C3%2C1

Thanks in advance.

As an end user I would like to consume the repo as a pip package potentially integrated with Home Assistant

@dp247 : what are your thoughts on including the common functions in the abstract class instead of a separate file?

This would be in keeping with the design pattern mentioned in https://github.com/robbrad/UKBinCollectionData/blob/master/CONTRIBUTING.md

Maybe it makes sense to rename the get_bin_data.py

Name of Council

Broadland & South Norfolk District Council

Example Postcode

NR18 0HA

Additional Information

https://www.southnorfolkandbroadland.gov.uk/rubbish-recycling/south-norfolk-bin-collection-day-finder

Seems like the page provides an iframe to render the results... recording the network requests once you have selected the address, it sends a post request to the following endpoint:

https://collections-southnorfolk.azurewebsites.net/WebService2.asmx
Content-Type must be set to text/xml; charset=UTF-8 text/xml; charset=UTF-8 for it to work

Post body is XML:

<?xml version="1.0" encoding="utf-8" ?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <getRoundCalendarForUPRN xmlns="http://tempuri2.org/">
            <council>SNO</council>
            <UPRN><redacted for privacy></UPRN>
            <PW>wax01653</PW>
        </getRoundCalendarForUPRN>
    </soap:Body>
</soap:Envelope>

I have a feeling changing "SNO" in the above to the shortcode for broadland district council (not sure what it is however) will allow the use of the same endpoint for both (both councils have formed a partnership and somewhat merged recently)

I'm not sure what that PW element is for in the post body, changing it gave me the following response:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <soap:Body>
        <getRoundCalendarForUPRNResponse xmlns="http://tempuri2.org/">
            <getRoundCalendarForUPRNResult>UnableToPRocess--Code-IVUsPsRl</getRoundCalendarForUPRNResult>
        </getRoundCalendarForUPRNResponse>
    </soap:Body>
</soap:Envelope>

Sample response on pastebin:

https://pastebin.com/CC6u1FS5

Could you try Durham CC?

They seem to provide a unique page for each house, for example

http://mydurham.durham.gov.uk/article/12690?uprn=10091173263

(UPRN, Unique Property Reference Number)

Which has the bin collections as text at the top.

Thanks!

name: Council Request
description: Request for a council to be added to the repository
labels: ['Class: enhancement']

body:
- type: input
id: council
attributes:
label: Buckinghamshire
description: Buckinghamshire - Chiltern Area
placeholder: e.g. Huntingdon District Council
validations:
required: true
- type: input
id: postcode
attributes:
label: Example Postcode
description: Please provide and example postcode for the area
placeholder: e.g. HP14 3NL
validations:
required: true
- type: textarea
id: extra
attributes:
label: Additional Information
description: The actual site is a Chiltern.gov site which is a child of the main Buckinghamshire site.
placeholder: (https://www.chiltern.gov.uk/collection-dates) - It seems to take a form submission with a session ID, which is redirected and provides results back in an HTML format. The table at the bottom of the returned page lists the next collection date for each bin type. Thank you!

Could you add North Somerset:

https://forms.n-somerset.gov.uk/Waste/CollectionSchedule
POST:
Content-Type: application/x-www-form-urlencoded

PreviousHouse :
PreviousPostcode : x
Postcode : x
SelectedUprn: 24034376

for example.

I'll have a going at doing it myself, but if you can do it, it might be quicker !

The idea here would be to create a GitHub action that runs the library for a set of given inputs and confirms against the output files

This could then validate if the council website has changed

Hi, any chance of the Peterborough Council code? Web page is here
https://www.peterborough.gov.uk/residents/rubbish-and-recycling/bins
You have to put in a postcode (try PE46PE), any number house. I would try myself but wouldn't know where to start!
Many thanks

Anyone got any objections moving to Poetry ?

https://python-poetry.org/docs/pyproject/

Name of Council

Castle Point District Council

Example Postcode

SS8 8AB

Additional Information

This is how far I have got at my first attempt at Python. The council site is https://apps.castlepoint.gov.uk/cpapps/index.cfm?fa=wastecalendar&a=1 and they use a 4 digit road code that I obtained through the url once I had selected my road. My road being Korndyke Avenue = 2757

I did adapt the code to work with Pyscript, that was fine until July's HA update and I'm now getting SSL errors with the URL.

Not sure how this could be adapted into your code if at all but thought I'd post it here incase its of any help.

import datetime
import requests
import calendar
from bs4 import BeautifulSoup


def ord(n):  # returns st, nd, rd and th
    return str(n) + (
        "th" if 4 <= n % 100 <= 20 else {1: "st", 2: "nd", 3: "rd"}.get(n % 10, "th")
    )


def todays_day():
    print(f"Today {(ord(todays_date.day))}")
    if todays_date.day in pink_sack:
        print("is pink")
    elif todays_date.day in black_sack:
        print("is black")


URL = "https://apps.castlepoint.gov.uk/cpapps/index.cfm?roadID=2757&fa=wastecalendar.displayDetails"
raw_html = requests.get(URL)
data = BeautifulSoup(raw_html.text, "lxml")


pink = data.find_all("td", class_="pink", limit=4)
black = data.find_all("td", class_="normal", limit=4)
month = data.find("div", class_="calMonthCurrent")

todays_date = datetime.date.today()

pink_sack = []
for div in pink:
    n = div.text
    pink_sack.append(n)
pink_sack = list(map(int, pink_sack))

black_sack = []
for div in black:
    n = div.text
    black_sack.append(n)
black_sack = list(map(int, black_sack))

color_sack = []
color_sack = [None] * (len(pink_sack) + len(black_sack))
if pink_sack[0] < black_sack[0]:
    color_sack[::2] = pink_sack
    color_sack[1::2] = black_sack
else:
    color_sack[::2] = black_sack
    color_sack[1::2] = pink_sack

second_month_start = [
    i for i in range(len(color_sack) - 1) if (color_sack[i] > color_sack[i + 1])
]

second_month_start = int("".join(map(str, second_month_start))) + 1

current_month_list = color_sack[:(second_month_start)]

next_month_list = color_sack[(second_month_start):]

for n in current_month_list:
    if n > (todays_date.day):
        nrd = n
        break
else:
    nrd = next_month_list[0]


if todays_date.day in current_month_list:
    todays_day()
elif todays_date.month == 12 and nrd == next_month_list[0]:
    todaysmonth = todays_date.month % 12 + 1
    todaysyear = todays_date.year + 1
    day = calendar.weekday((todaysyear), (todaysmonth), (nrd))
    print(
        f"Next rubbish day is {(calendar.day_name[day])} the {(ord(nrd))}"
        + (" and is Pink" if nrd() in pink_sack else " and is Black")
    )
elif todays_date.month != 12 and nrd == next_month_list[0]:
    todaysmonth = todays_date.month + 1
    todaysyear = todays_date.year
    day = calendar.weekday((todaysyear), (todaysmonth), (nrd))
    print(
        f"Next rubbish day is {(calendar.day_name[day])} the {(ord(nrd))}"
        + (" and is Pink" if nrd in pink_sack else " and is Black")
    )
else:
    day = calendar.weekday((todays_date.year), (todays_date.month), (nrd))

    print(
        f"Next rubbish day is {(calendar.day_name[day])} the {(ord(nrd))}"
        + (" and is Pink" if nrd in pink_sack else " and is Black")
    )

Hi - Great project you have here - shame more councils don't just open up a public API in this day and age.

I wondered what your take on Wealden Council was: https://www.wealden.gov.uk/recycling-and-waste/bin-search/?uprn=10033413624

I can see there's a call to a service at https://www.wealden.gov.uk/wp-admin/admin-ajax.php but I think its behind an authorisation layer. So is the data scrapeable instead?

Thanks

A friend pointed me towards this project a while ago and since I've started learning/using Python in a professional capacity, I thought I'd take a crack at adding my local council to the repo. Unfortunately, however, it seems that their website has Incapsula as a firewall, which returns a bot detection response (like the one here).

We do have an actual bin app though that's not made by the council and seems to be getting its data from Data Mill North via two CSV files. I thought about taking the same approach with this, but thought I'd ask permission first, since I know this project is more gained towards using BeautifulSoup.

I've started in my own repo using the CouncilClass like the other scripts - the only things that differ so far is the passed URL doesn't do anything, though I'll need to do something to handle PAON/postcode entry (either adding input prompts or passing as command line arguments somehow).

Is this a viable approach, or would I be better contributing by trying the outstanding requests?

Name of Council

Test

Example Postcode

TESTPOSTCODE

Additional Information

TEST

https://www.southlanarkshire.gov.uk/directory_record/577332/knights_gate_bothwell

Name of Council

Mid Sussex District Council

Example Postcode

RH10 3XD

Additional Information

https://www.midsussex.gov.uk/waste-recycling/bin-collection/

When a matrix build happens on multiple versions of python they clash publishing the allure reports

Behave runs on one version of python and then on the other - causing a git race condition

Using the Poetry restructure the project so it support proper project references and paths

Hey @robbrad, just wondered if it was possible to give contributors write permissions to the wiki (totally understandable if not, I don't mind submitting a PR instead 😁). I'd like to use it to provide a list of supported councils, as well as the necessary commands to run the script (as opposed to another markdown file!)

Hi, I think I have this working from a Home Assistant / Python point of view, however when I run I get a Python Syntax error

Syntax errors detected :

Line 26:
data[f'{binType}'] = f'{binCollection}'
^
SyntaxError: invalid syntax

I know next to nothing about coding so I tried checking here https://extendsclass.com/python-tester.html and it shows the same error. I checked the other bin collection scripts and the one that is similar to this throws the same error, but the ones that are completely different doesn't. I checked pythonbuddy.com and it said "Format strings are only supported in Python 3.6 and greater (, line 26)" however I have 3.7 installed as Home Assistant required it and the venv was set up using it. I tried to shebang the specific version 3.7.3 and it still threw the syntax error. Any idea?

Originally posted by @Mod74 in #3 (comment)

Name of Council

Royal Borough of Windsor and Maidenhead

Example Postcode

TW19 5NF

Additional Information

Other council sites I have scraped before auto update, on this one it seems the current date is included in the URL aswell just to confuse things. https://my.rbwm.gov.uk/special/find-your-collection-dates

Each council basically needs:

• A json schema for the output generated
• A line item in the above feature
• A change to input.json to include a working scrape url

Add instructions for the following councils:

• York
• East Riding
• Windsor and Maidenhead (once complete)
• Castlepoint

Please could East northants be added?

https://mapsenc.east-northamptonshire.gov.uk/waste-and-recycling/bin-collection-day.html

Example post code: NN14 4AY

Name of Council

City of York Council

Example Postcode

YO24 4PB

Additional Information

This would be a good first issue for anyone new to Python/scraping APIs, as York provide their own API for scraping bin data easily. Requires a UPRN rather than a postcode.

Link to the API documentation:

https://storage.googleapis.com/dx-york-production/ckan/resources/91b2852b-c779-4f4d-9ef4-3c2fdfa2a565/waste-services-api-v-3.0.0.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Credential=GOOG1EZ6DEPSOK7DRKXPDSBDQNJKTDCVUD65PV6GN5JNEA76IMWPISTE5BIHY%2F20220723%2Feurope-west1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20220723T231428Z&X-Amz-Signature=4dd34cb7ad4a80d79c4cad9238b708aac66afcbebabf87cb733c05d0974e098e

Link to the waste collection date checker:

https://myaccount.york.gov.uk/bin-collections

I have linked below my postman collections, however this should be really easy to implement.

Postcode - PE15RY
UPRN     - 10008051981

Hi Rob,

I'm having a hack around with the scripts, but am having line ending issues, specifically adding the shebang #!/usr/bin/env python3 to collect_data.py in develop. It looks like the master branch uses LF endings, but the develop has switched to CRLF. Was this a deliberate choice? (If so, HuntingdonDistrictCouncil.py in develop appears to be LF endings).

I'm totally unfamiliar with git/github's line end mangling and the expected formatting in repo and working copy, except on *nix files must be LF for shebangs to work. If it wasn't intentional, I think I've sorted it in aejh/UKBinCollectionData@ 46e00c9 (from here: https://www.aleksandrhovhannisyan.com/blog/crlf-vs-lf-normalizing-line-endings-in-git/#a-simple-gitattributes-config), can make a pull request if you think I'm barking up the right tree!

Cheers, Al.

Hi Could you add Sheffield to this?

https://wasteservices.sheffield.gov.uk/

Thanks in advance!

Create suite of unit tests for the common classes.

Hi there,

Now I'm 99.9% sure this is a user error issue but thought I'd ask anyways.

I've copied the repo to a ubuntu WSL vm thing on my pc and updated/installed the scripts according to the poetry file, but when running the script for my council that you guys kindly created the other week I get a missing module error

Traceback (most recent call last):
File "/root/UKBinCollectionData/uk_bin_collection/uk_bin_collection/collect_data.py", line 6, in
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
ModuleNotFoundError: No module named 'uk_bin_collection'

My HA install is running on a separate machine running a Proxmox server

any help/guidance would be smashing

Name of Council

Rushcliffe Borough Council

Example Postcode

ng13 8tz

Additional Information

No response

Root folder in unit test breaks the module import

ref: 4173300

Moving back to
from uk_bin_collection.common import *

from recent commit

from uk_bin_collection.uk_bin_collection.common import *

Seams to work fine

Unit tests run with : poetry run coverage run -m pytest

and also work on VS Code with the venv set

https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/Pages/default.aspx

Example postcode - CF5 3SX

Please could South Tyneside be added?

https://www.southtyneside.gov.uk/article/33352/Bin-collection-dates

Example post code: NE33 3LH

Name of Council

Welwyn Hatfield Council

Example Postcode

AL7 4TT

Additional Information

https://services.welhat.gov.uk/ebase/Next_bin_collection.eb?CALLING_FORM=Search&ebd=0&ebp=10&ebz=7_1641940471960

Its use two step, first a valid postcode and then the select door number. Please use is postcode mentioned above.

The webpage is using jquery, may need use selenium library with python to extract the Bin collection dates.

Thanks in Advance.

As an end user I would like to have the ability to learn how to run the library/debug/develop

consider read the docs (RST) or readme.md / build.md etc

Hi

Any chance of getting Wakefield Council added please ?

Thanks

Martyn

Please could Bournemouth Poole Council be added?

https://www.poole.gov.uk/bins-waste-and-recycling/bins/bin-collection-day-finder/

Example post code: BH14 0PD

Name of Council

South Ayrshire Council

Example Postcode

KA197AD

Additional Information

https://www.south-ayrshire.gov.uk/bin-days?pageSessionId=09dec6f4-9646-408d-9779-fe8063f0805b&fsn=a4c11bc8-de4d-4f9b-be62-eb9a3b38ca8f

Thank you :)