unownhash / fletchling Goto Github PK

View Code? Open in Web Editor NEW

10.0 5.0 3.0 218 KB

Dockerfile 0.48% Go 98.96% Shell 0.33% Makefile 0.23%

fletchling's Introduction

Fletchling

Fletchling is a Golbat webhook receiver that processes pokemon webhooks and computes nesting pokemon.

Features

Receives and processes pokemon on the fly via webhook from Golbat
Uses global spawn data to not confuse event spawns (CD, Spotlight, etc) with nesting pokemon.
No reliance on external sites for event or current nesting mon data.
Tool for importing nests from overpass (fletchling-osm-importer)
API to pull stats, purge stats, reload config, etc.

Configuration

Copy configs/fletchling.toml.example to configs/fletchling.toml and edit it.
In golbat, add a new webhook entry to its config. It should look like this:

[[webhooks]]
url = "http://FLETCHLING-HOSTNAME:9042/webhook"
types = ["pokemon_iv"]

Replace 'FLETCHLING-HOSTNAME' with your host that is running Fletchling. If you are using pm2 and not using docker, this should likely be '127.0.0.1'. If you are using docker, your fletchling hostname will likely be 'fletchling'.

After adding the config to Golbat, restart Golbat to have it take effect.

If you plan to use docker-compose, copy docker-compose.yml.example to docker-compose.yml and edit.

Running

docker

There is an included docker-compose.yml.example file. Copy it to docker-compose.yml and edit it, if needed.
docker-compose up -d

pm2

Requirements

You will need to have at least golang 1.22.1 installed. It is rather new as of the time of this writing. You may need to install it manually. See the instructions and download links.

Building and starting

make
pm2 start ./fletchling --name fletchling -o /dev/null

Verifying it is working

curl http://localhost:9042/api/config -- this should give you the current processor configuration
Check the logs in logs/ (or the log_dir that you configured in fletchling.toml) for errors.
- Every minute, a log message will appear saying how many pokemon were processed. If this is 0, it means that Fletchling is not getting any webhooks. Check your Golbat webhooks configuration. Check the address Fletchling is listening on (http section in config).

Migrating from other nest processors

nestcollector to Fletching using existing Golbat DB for nests (SIMPLEST)

Gather your Golbat DB info.
Create configs/fletchling.toml as per 'Configuration' instructions above.
- configure your existing golbat db in configs/fletchling.toml in BOTH 'nests_db' and 'golbat_db' sections.
- fix the listen address in 'http' section, if necessary.
nuke your cronjob.
start up fletchling per 'Running' section above.

Others

I would just start over, personally. :)

Importing OSM data

Importing docs can be found here.

API

API docs can be found here.

FAQ

FAQ can be found here.

Credits

Thanks to the folks who put a lot of time and effort into nestcollector! Most of the importing, nest-activation filtering, and so forth were either copied/ported from nestcollector or used as a guide to achieve some amount of compatibility.

Enjoy!

All your nest are belong to us.

fletchling's People

Contributors

Stargazers

Watchers

Forkers

sabregreen cronick reuschelcgn

fletchling's Issues

Add option to disable nests with low nest spawn counts

Some stuff from nc:

 echo "update nests set discarded=NULL where pokemon_avg >= $nest_min_spawn_hr and active=1 and discarded = 'spawnhr_warn';" >&3
 echo "update nests set discarded='spawnhr_ban', active=0, pokemon_id=NULL, pokemon_form=NULL where pokemon_avg < $nest_min_spawn_hr and active=1 and discarded = 'spawnhr_warn';" >&3
 echo "update nests set discarded='spawnhr_warn' where pokemon_avg < $nest_min_spawn_hr and active=1 and (discarded <> 'spawnhr_ban' or discarded is NULL);" >&3

overlap query should unset pokemon_id

Fletchling makes sure to unset pokemon_id when disabling a nest due to area/spawnpoints, however it doesn't unset if disabled due to overlap and it should.

Writes logs into pm2 error stream

It writes the out into the wrong error stream ,we initially had that issue with Golbat as well, should be an easy fix to write to right stream :)

Option to Merge/Ignore Polygons Encased Entirely by Another

Most of these inner Polygons are not "real" Nests and don't need to be included.

Failed HTTP server startup missing error log

When HTTP server can't be spawned on provided IP:PORT error should be logged. Looks like it's not and Fletchling is shutting down.

Ignore unsupported shapes on import

osm data can return linestring routes. we should ignore these.

Webhook area matching

Need to do something about webhook area matching. There's code for this but disabled at the moment.

I think this should work like Golbat, but the problem is if you have parents on your areas.

DB only has "area_name" right now. Not sure if people have "parent/area" in this column or just "area".

Inaccuracies when areas span multiple timezones.

If one has areas in multiple time zones, the global spawn data is all mixed together. If only 1 small area is in a completely separate timezone, it's effectively comparing against global spawn data that contains mostly data for another time zone. This would mess up the logic that prevents CD/Spotlight hour mons from being counted as nesting.

I wondered if anyone would have this. Apparently at least 1 person does.

Add option for koji `replace` param

When I use /api/v1/geofence/feature-collection/Nest?replace=%22-nest%22 I'm getting error:

INFO 2024-03-24 12:40:02 Reloading areas from koji project 'Nest'
ERRO 2024-03-24 12:40:02 Failed to load areas: invalid character '<' looking for beginning of value.

That's actually a big issue for me cause I can't duplicate some areas with same names (eg. new york -> New York) and that would require rethink my whole koji setup and rename lots of areas :(

To fix it I simply opened my url and saved content to a file that I used in config instead of koji. It works but it requires manual updates each time I change anything in koji :/

Missing nameless nests from OSM import

Nameless nests are not saved.

Example case: https://www.openstreetmap.org/way/251449254

WARN 2024-02-08 14:22:20 skipping osm feature id '251449254' at 52.71359,16.37914 with no name, leisure=''

PS. Unsure about the ones that have name = "Unknown " + mapping as all of them got rejected in my case.

Importer - optimize contained

Run a contanied check only within the same area name. Otherwise, the end of great amount of areas will be heavy as hell :)

fletchling-importer: process all areas flag

Having a looong list of areas, I think it makes sense to process all areas from an input project. Maybe a switch like -all-input-areas that would allow to process without -overpass-area

./bin/fletchling-importer/fletchling-importer -overpass-areas-src koji -overpass-koji-project 'EnabledAreas' -overpass-area 'Warszawa' overpass db

Reload API endpoint does not refresh/re-run spawnpoint count

I don't know if I am doing something wrong or skipping any step. I have some Nests that are filled with 0 spawnpoints, but I have checked in RM and Koji that in fact there are spawnpoints in there...

I have tried to reload fletchling, with refresh=1 and with refresh=1&spawnpoints=all and it continues counting 0 spawnpoints.

EDIT:
I had to manually remove the nest from db and import it again. Then using refresh=1&spawnpoints=all spawnpoints have been counted. It seems that this api endpoint http://localhost:9042/api/config/reload?refresh=1&spawnpoints=all doesn't rerun/refresh spawnpoint count if it has a value. It only refreshes if it is NULL.

EDIT 2:
Okey, I think I have been able to reproduce it
If a nest is activated and you force spawnpoints to 0, if you reload with spawnpoint=all, spawnpoints are populated.
If a nest is not activated and it has 0 spawnpoints, if you reload it, it wont update spawnpoints

./fletchling-osm-importer -all-areas -only-new

That would fetch all unique area_name and process only unmatched areas.

Keep unset Pokemon ID as NULL

Don't set it to zero in databse. Previous tools had no active afaik and things might break on pokemon_id = 0 (like in RM).