Code Monkey home page Code Monkey logo

parker's Introduction

CarParker

CarParker is a web based searching engine to facilitate drivers to quickly find a parking with acceptable rates. It crawls the web to build up an up-to-date database of parking rates which is then used to serve search results to end users.

Getting started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisities

You need to have Docker Engine and docker-compose installed.

Installing

Create a new file parker/settings.py.

Copy the contents from parker/settings.default.py into the new file.

Alternatively, on Unix/Linux systems you can simply create a symlink by running.

ln -s parker/settings/default.py parker/settings.py

Start docker containers

docker-compose up -d

Attaching to app container

In order to run any commands and test your code, you need to be attached to the main app container. In order to do so, execute

docker exec -i -t carparker_parker_1 /bin/bash

Downloading rates

Run the following command in order to download HTML cache for all known carparks.

python3 scripts/downloadParkingRatesHTML.py

Run the following command to process and store the cached rates.

python3 scripts/loadParkingRates.py

Running the tests

Tests are located in parker/tests/.py*.

To execute test run:

./runtests.sh

Backing up database

If any changes has been made to the data in database, they need to be backed up manually in order to not get lost. It can be done by running a backup_db.sh script from the main app container.

Versioning

Used SemVer for versioning. For the versions available, see the tags on this repository.

Authors

  • Maxim Pak - Initial idea & work - zucler
  • Stepan Tsymbal - Greatest contributor - ouvtk

License

Copyright (C) Maxim Pak - All Rights Reserved

parker's People

Contributors

zucler avatar

Watchers

James Cloos avatar  avatar Stepan Tsymbal avatar

parker's Issues

Geo-search JS optimisations

The following JS tweaks are required on the geo-search page:

  • Markers are being refreshed each time as the map is moved. It should not refresh markers already added.
  • InfoWindow objects are created for each marker. Should only be 1.

Profile and improve front-end database requests

The code below is not optimised and can cause long delays and broad results.

 parkings = Parking.objects.filter(lat__gte=min_lat, lat__lte=max_lat, long__gte=min_long, long__lte=max_long).all()
    rtype = RateType.objects.filter(parkingID__in=parkings).all()
    rprice = RatePrice.objects.filter(rateID__in=rtype).all()

    # To cache whole querysets
    # FIXME: We need workaround where no extra work done
    bool(parkings)
    bool(rtype)
    bool(rprice)

    parkings = list(geosearch['parkings'])     # Here we have list of dicts
    rtype = geosearch['ratetype']
    rprice = geosearch['rateprice']

        for p_item in parkings:
            sub_ratetype = rtype.filter(parkingID=p_item.parkingID)
            for rt_item in sub_ratetype:
                rt_item.rateprice = list(rprice.filter(rateID=rt_item.rateID))
            p_item.ratetype = sub_ratetype

In general, we need to limit the maximum number of rows returned (e.g. to not return carparks for the entire world) by either:

a) Limiting the difference between min and max lat/long
b) Limiting max count of rows returned

Refactor unit tests to not rely on hardcoded expected results

Background:
Generating hardcoded set of expected results is not a long-term solution. Instead, we should have a generic validation logic.

Requirements:
Refactor existing unit tests code to validate each field / attribute of the parking rates object generically based on the data type expected (e.g. string, integer, double) and/or mandatory vs optional value assigned to a field..

Store any non-recognisable information as notices

Background
The parking rates page is usually a free form and administrators are regularly place information that is not recognisable by a parser. In some cases it can be a short, seasonal notice (e.g. public holidays surcharge).

Requirements:
All notices that parses can not recognise must be stored as notes for each of the carparks.

Examples:

  • Early bird parking only available if you book online via Secure a Spot (Secure Parking)
  • Have ticket validated by Attendant in the morning (Wilson)

Build website UI

Background:
Need to build a website UI which users are going to use to run searches

Pre-requisites:

  • Graphic designs
  • Approved user workflows (i.e. user stories)

Requirements:

  • User friendliness
  • Intuitiveness
  • UX should be straight - forward, so that a user can get information required in up to 3 clicks.

Research machine learning

Background:
Machine learning approach may become useful if the platform grows big and need to support multiple different cities / countries. We need to practice it ASAP and compare ML driven rates retriever vs traditional OOP driven one (Wilson Parkings).

Requirements:

  • Research into Machine Learning realm and find an algorithm that would suit our needs
  • Document the approach
  • Build POC by prototyping Secure Parking rates retriever

Parse & Store carparks operating hours

Background:
Some parkings are not working during public holidays, others only work for a few hours during the weekend.

Requirements:

  • Need to parse working hours section of each of the carparks
  • Store the information received in the DB
  • Account for the working hours when running a user search

Create a script to pull down list of car parks available

Background:
Current approach when a person needs to manually hardcode a URL for a carpark to process is very tedious and is not working well. We need to start from a higher level and first write a script to pull down list of all carparks available per carpark website (i.e. a single interpreter for wilson and a single for secure parking at first).

This would allow to have access to a much larger pool of data and enable to apply ML techniques.

Task:

  • Create a script that downloads HTML with all carparks available
  • Download & update details of each of the carparks and populate data such as opening hours, potentially geo data, etc.

Store rates processing audit log for every car park

Background:
To enable us to troubleshoot issues historically for a certain carpark, rates processing script need to report its' actions to a temporary audit logs for all carparks in the system.

Requirements:

  • Log storage should be scalable
  • Log entries should expire after 30 days, to maintain disk space usage
  • Storage system must be robust, highly accessible and ideally open-source.

Potential candidates:

  • Redis
  • Splunk
  • Kibana

This is an R&D task.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.