robbrad / ukbincollectiondata Goto Github PK
View Code? Open in Web Editor NEWUK Council Bin Collection Data Parser Outputting Bin Data as a JSON
License: MIT License
UK Council Bin Collection Data Parser Outputting Bin Data as a JSON
License: MIT License
Add issue templates to help better manage the adding of councils, I have provided below an example
name: Council Request
description: Request for a council to be added to the repository
labels: ['Class: enhancement']
body:
- type: input
id: council
attributes:
label: Name of Council
description: What council are you wishing to be added
placeholder: e.g. Huntingdon District Council
validations:
required: true
- type: input
id: postcode
attributes:
label: Example Postcode
description: Please provide and example postcode for the area
placeholder: e.g. PE7 3YQ
validations:
required: true
- type: textarea
id: extra
attributes:
label: Additional Information
description: Add any other information here
placeholder: Links to the councils site, information you have already gathered
https://github.com/Tmp2k/wigan-council-api
GET /bins?postcode=WN1+2RW
Hi there, i've been wanting to do this for a while but I don't think my skills are up to it, any help with Kingston Upon Thames would be greatly appreciated!
https://www.kingston.gov.uk/when-is-my-collection
They are also listed on here: https://maps.kingston.gov.uk/propertyServices/InMyArea
Thanks!
Interesting one, doesn't look like they have an API they require a log in to view your details or at least I thought
https://my.crawley.gov.uk/
looking at the page source you don't have to be logged in once you have a customer info via the page source, I tested this in an incognito window so it should be able to be scraped if someone with the knowhow wants to work with me to get this one working, I just don't want to publicly post my customer info
https://www.northlincs.gov.uk/bins-waste-and-recycling/bin-collection-dates/
Enter Post Code: DN17 2TP
General code coverage improvement for the common.py file
Hi, I have a working script for Charnwood Borough Council and would like to raise a PR to get this into the repo, however I have no access to create a branch. Could you look at the access policies?
Thanks
North East Lincolnshire
DN36 4DQ
I used to be able to scrape the wesbite but they've made it a little more difficult and my know-how is lacking.
It'd be great to be able to have the latest data from Fenland please.
Thanks in advance.
As an end user I would like to consume the repo as a pip package potentially integrated with Home Assistant
@dp247 : what are your thoughts on including the common functions in the abstract class instead of a separate file?
This would be in keeping with the design pattern mentioned in https://github.com/robbrad/UKBinCollectionData/blob/master/CONTRIBUTING.md
Maybe it makes sense to rename the get_bin_data.py
Broadland & South Norfolk District Council
NR18 0HA
Seems like the page provides an iframe to render the results... recording the network requests once you have selected the address, it sends a post request to the following endpoint:
https://collections-southnorfolk.azurewebsites.net/WebService2.asmx
Content-Type must be set to text/xml; charset=UTF-8 text/xml; charset=UTF-8
for it to work
Post body is XML:
<?xml version="1.0" encoding="utf-8" ?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<getRoundCalendarForUPRN xmlns="http://tempuri2.org/">
<council>SNO</council>
<UPRN><redacted for privacy></UPRN>
<PW>wax01653</PW>
</getRoundCalendarForUPRN>
</soap:Body>
</soap:Envelope>
I have a feeling changing "SNO" in the above to the shortcode for broadland district council (not sure what it is however) will allow the use of the same endpoint for both (both councils have formed a partnership and somewhat merged recently)
I'm not sure what that PW element is for in the post body, changing it gave me the following response:
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<getRoundCalendarForUPRNResponse xmlns="http://tempuri2.org/">
<getRoundCalendarForUPRNResult>UnableToPRocess--Code-IVUsPsRl</getRoundCalendarForUPRNResult>
</getRoundCalendarForUPRNResponse>
</soap:Body>
</soap:Envelope>
Sample response on pastebin:
Could you try Durham CC?
They seem to provide a unique page for each house, for example
http://mydurham.durham.gov.uk/article/12690?uprn=10091173263
(UPRN, Unique Property Reference Number)
Which has the bin collections as text at the top.
Thanks!
name: Council Request
description: Request for a council to be added to the repository
labels: ['Class: enhancement']
body:
- type: input
id: council
attributes:
label: Buckinghamshire
description: Buckinghamshire - Chiltern Area
placeholder: e.g. Huntingdon District Council
validations:
required: true
- type: input
id: postcode
attributes:
label: Example Postcode
description: Please provide and example postcode for the area
placeholder: e.g. HP14 3NL
validations:
required: true
- type: textarea
id: extra
attributes:
label: Additional Information
description: The actual site is a Chiltern.gov site which is a child of the main Buckinghamshire site.
placeholder: (https://www.chiltern.gov.uk/collection-dates) - It seems to take a form submission with a session ID, which is redirected and provides results back in an HTML format. The table at the bottom of the returned page lists the next collection date for each bin type. Thank you!
Could you add North Somerset:
https://forms.n-somerset.gov.uk/Waste/CollectionSchedule
POST:
Content-Type: application/x-www-form-urlencoded
PreviousHouse :
PreviousPostcode : x
Postcode : x
SelectedUprn: 24034376
for example.
I'll have a going at doing it myself, but if you can do it, it might be quicker !
The idea here would be to create a GitHub action that runs the library for a set of given inputs and confirms against the output files
This could then validate if the council website has changed
Hi, any chance of the Peterborough Council code? Web page is here
https://www.peterborough.gov.uk/residents/rubbish-and-recycling/bins
You have to put in a postcode (try PE46PE), any number house. I would try myself but wouldn't know where to start!
Many thanks
Anyone got any objections moving to Poetry ?
Castle Point District Council
SS8 8AB
This is how far I have got at my first attempt at Python. The council site is https://apps.castlepoint.gov.uk/cpapps/index.cfm?fa=wastecalendar&a=1 and they use a 4 digit road code that I obtained through the url once I had selected my road. My road being Korndyke Avenue = 2757
I did adapt the code to work with Pyscript, that was fine until July's HA update and I'm now getting SSL errors with the URL.
Not sure how this could be adapted into your code if at all but thought I'd post it here incase its of any help.
import datetime
import requests
import calendar
from bs4 import BeautifulSoup
def ord(n): # returns st, nd, rd and th
return str(n) + (
"th" if 4 <= n % 100 <= 20 else {1: "st", 2: "nd", 3: "rd"}.get(n % 10, "th")
)
def todays_day():
print(f"Today {(ord(todays_date.day))}")
if todays_date.day in pink_sack:
print("is pink")
elif todays_date.day in black_sack:
print("is black")
URL = "https://apps.castlepoint.gov.uk/cpapps/index.cfm?roadID=2757&fa=wastecalendar.displayDetails"
raw_html = requests.get(URL)
data = BeautifulSoup(raw_html.text, "lxml")
pink = data.find_all("td", class_="pink", limit=4)
black = data.find_all("td", class_="normal", limit=4)
month = data.find("div", class_="calMonthCurrent")
todays_date = datetime.date.today()
pink_sack = []
for div in pink:
n = div.text
pink_sack.append(n)
pink_sack = list(map(int, pink_sack))
black_sack = []
for div in black:
n = div.text
black_sack.append(n)
black_sack = list(map(int, black_sack))
color_sack = []
color_sack = [None] * (len(pink_sack) + len(black_sack))
if pink_sack[0] < black_sack[0]:
color_sack[::2] = pink_sack
color_sack[1::2] = black_sack
else:
color_sack[::2] = black_sack
color_sack[1::2] = pink_sack
second_month_start = [
i for i in range(len(color_sack) - 1) if (color_sack[i] > color_sack[i + 1])
]
second_month_start = int("".join(map(str, second_month_start))) + 1
current_month_list = color_sack[:(second_month_start)]
next_month_list = color_sack[(second_month_start):]
for n in current_month_list:
if n > (todays_date.day):
nrd = n
break
else:
nrd = next_month_list[0]
if todays_date.day in current_month_list:
todays_day()
elif todays_date.month == 12 and nrd == next_month_list[0]:
todaysmonth = todays_date.month % 12 + 1
todaysyear = todays_date.year + 1
day = calendar.weekday((todaysyear), (todaysmonth), (nrd))
print(
f"Next rubbish day is {(calendar.day_name[day])} the {(ord(nrd))}"
+ (" and is Pink" if nrd() in pink_sack else " and is Black")
)
elif todays_date.month != 12 and nrd == next_month_list[0]:
todaysmonth = todays_date.month + 1
todaysyear = todays_date.year
day = calendar.weekday((todaysyear), (todaysmonth), (nrd))
print(
f"Next rubbish day is {(calendar.day_name[day])} the {(ord(nrd))}"
+ (" and is Pink" if nrd in pink_sack else " and is Black")
)
else:
day = calendar.weekday((todays_date.year), (todays_date.month), (nrd))
print(
f"Next rubbish day is {(calendar.day_name[day])} the {(ord(nrd))}"
+ (" and is Pink" if nrd in pink_sack else " and is Black")
)
Hi - Great project you have here - shame more councils don't just open up a public API in this day and age.
I wondered what your take on Wealden Council was: https://www.wealden.gov.uk/recycling-and-waste/bin-search/?uprn=10033413624
I can see there's a call to a service at https://www.wealden.gov.uk/wp-admin/admin-ajax.php but I think its behind an authorisation layer. So is the data scrapeable instead?
Thanks
A friend pointed me towards this project a while ago and since I've started learning/using Python in a professional capacity, I thought I'd take a crack at adding my local council to the repo. Unfortunately, however, it seems that their website has Incapsula as a firewall, which returns a bot detection response (like the one here).
We do have an actual bin app though that's not made by the council and seems to be getting its data from Data Mill North via two CSV files. I thought about taking the same approach with this, but thought I'd ask permission first, since I know this project is more gained towards using BeautifulSoup.
I've started in my own repo using the CouncilClass like the other scripts - the only things that differ so far is the passed URL doesn't do anything, though I'll need to do something to handle PAON/postcode entry (either adding input prompts or passing as command line arguments somehow).
Is this a viable approach, or would I be better contributing by trying the outstanding requests?
Test
TESTPOSTCODE
TEST
Mid Sussex District Council
RH10 3XD
https://www.midsussex.gov.uk/waste-recycling/bin-collection/
When a matrix build happens on multiple versions of python they clash publishing the allure reports
Behave runs on one version of python and then on the other - causing a git race condition
Using the Poetry restructure the project so it support proper project references and paths
Hey @robbrad, just wondered if it was possible to give contributors write permissions to the wiki (totally understandable if not, I don't mind submitting a PR instead ๐). I'd like to use it to provide a list of supported councils, as well as the necessary commands to run the script (as opposed to another markdown file!)
Hi, I think I have this working from a Home Assistant / Python point of view, however when I run I get a Python Syntax error
Syntax errors detected :
Line 26:
data[f'{binType}'] = f'{binCollection}'
^
SyntaxError: invalid syntax
I know next to nothing about coding so I tried checking here https://extendsclass.com/python-tester.html and it shows the same error. I checked the other bin collection scripts and the one that is similar to this throws the same error, but the ones that are completely different doesn't. I checked pythonbuddy.com and it said "Format strings are only supported in Python 3.6 and greater (, line 26)" however I have 3.7 installed as Home Assistant required it and the venv was set up using it. I tried to shebang the specific version 3.7.3 and it still threw the syntax error. Any idea?
Originally posted by @Mod74 in #3 (comment)
Royal Borough of Windsor and Maidenhead
TW19 5NF
Other council sites I have scraped before auto update, on this one it seems the current date is included in the URL aswell just to confuse things. https://my.rbwm.gov.uk/special/find-your-collection-dates
Each council basically needs:
Add instructions for the following councils:
Please could East northants be added?
https://mapsenc.east-northamptonshire.gov.uk/waste-and-recycling/bin-collection-day.html
Example post code: NN14 4AY
City of York Council
YO24 4PB
This would be a good first issue for anyone new to Python/scraping APIs, as York provide their own API for scraping bin data easily. Requires a UPRN rather than a postcode.
Link to the API documentation:
Link to the waste collection date checker:
Hi Rob,
I'm having a hack around with the scripts, but am having line ending issues, specifically adding the shebang #!/usr/bin/env python3
to collect_data.py in develop. It looks like the master branch uses LF endings, but the develop has switched to CRLF. Was this a deliberate choice? (If so, HuntingdonDistrictCouncil.py in develop appears to be LF endings).
I'm totally unfamiliar with git/github's line end mangling and the expected formatting in repo and working copy, except on *nix files must be LF for shebangs to work. If it wasn't intentional, I think I've sorted it in aejh/UKBinCollectionData@ 46e00c9 (from here: https://www.aleksandrhovhannisyan.com/blog/crlf-vs-lf-normalizing-line-endings-in-git/#a-simple-gitattributes-config), can make a pull request if you think I'm barking up the right tree!
Cheers, Al.
Create suite of unit tests for the common classes.
Hi there,
Now I'm 99.9% sure this is a user error issue but thought I'd ask anyways.
I've copied the repo to a ubuntu WSL vm thing on my pc and updated/installed the scripts according to the poetry file, but when running the script for my council that you guys kindly created the other week I get a missing module error
Traceback (most recent call last):
File "/root/UKBinCollectionData/uk_bin_collection/uk_bin_collection/collect_data.py", line 6, in
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
ModuleNotFoundError: No module named 'uk_bin_collection'
My HA install is running on a separate machine running a Proxmox server
any help/guidance would be smashing
Rushcliffe Borough Council
ng13 8tz
No response
Root folder in unit test breaks the module import
ref: 4173300
Moving back to
from uk_bin_collection.common import *
from recent commit
from uk_bin_collection.uk_bin_collection.common import *
Seams to work fine
Unit tests run with : poetry run coverage run -m pytest
and also work on VS Code with the venv set
Example postcode - CF5 3SX
Please could South Tyneside be added?
https://www.southtyneside.gov.uk/article/33352/Bin-collection-dates
Example post code: NE33 3LH
Welwyn Hatfield Council
AL7 4TT
Its use two step, first a valid postcode and then the select door number. Please use is postcode mentioned above.
The webpage is using jquery, may need use selenium library with python to extract the Bin collection dates.
Thanks in Advance.
As an end user I would like to have the ability to learn how to run the library/debug/develop
consider read the docs (RST) or readme.md / build.md etc
Hi
Any chance of getting Wakefield Council added please ?
Thanks
Martyn
Please could Bournemouth Poole Council be added?
https://www.poole.gov.uk/bins-waste-and-recycling/bins/bin-collection-day-finder/
Example post code: BH14 0PD
South Ayrshire Council
KA197AD
Thank you :)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.