This is a new message allowing to bulk request raid lobby counts.

Messages

message GetRaidLobbyCounterOutProto {
	enum Result {
		UNSET = 0;
		SUCCESS = 1;
		ERROR_PLAYER_BELOW_MINIMUM_LEVEL = 2;
	}

	Result result = 1;
	repeated RaidLobbyPlayerCountProto raid_lobbies = 2;  // name TBD
}

message RaidLobbyPlayerCountProto {
	string gym_id = 2;
	int32 player_count = 3;
	int64 lobby_join_until_ms = 4;
}

Example response

GetRaidLobbyCounterOutProto {
    result Result.SUCCESS:1
    
    raid_lobbies {  // no active lobby
        gym_id "45a76c2b88ba4529a2f9379de6f88d30.16"
    }
    
    raid_lobbies {
        gym_id "00f9208d76a86794b2178181c304e41d.11"
        player_count 1
        lobby_join_until_ms 1682090628489
    }
}

This will require two new DB fields: lobby_count and lobby_join_end (names TBD) and two new webhook fields. If there's no active lobby, the fields should be null.

there is no point. it's only clutter

my suggestion:

[pvp]
enabled = true
include_hundos_under_cap = false
level_caps = [50, 51]
process_ultra = true
process_great = true
process_little = true

old:

[pvp]
enabled = true
include_hundos_under_cap = false
level_caps = [50, 51]

[[pvp.leagues]]
name = "little"
cap = 500
little = false

[[pvp.leagues]]
name = "great"
cap = 1500
little = false

[[pvp.leagues]]
name = "ultra"
cap = 2500
little = false

some code like this needed (from RDM)

if self.cellId == nil {
  let centerCoord = LocationCoordinate2D(latitude: encounter.pokemon.latitude, longitude: encounter.pokemon.longitude)
  let cellID = S2CellId(latlng: S2LatLng(coord: centerCoord)).parent(level: 15)
  self.cellId = cellID
}

As description.

As title. Move webhook settings from the config to the database and add an API endpoint to hot reload.

The last modified time for lures and invasions is incorrect

I miss the MAD feature to only send e.g. quests to a specific endpoint rather than sending everything and creating unwanted traffic.

Calculate quest end time based on lat/lon and store

If we have a scanner which always encounters everything; and it understands it needs to re-encounter pokemon on weather change then writing wilds to the DB is inefficient

Provide an option to ignore wilds and work on encounters only
This should not impact despawn time discovery

(nearbys unaffected)

I call a Ditto 0P if it is in partly cloudy weather whose disguise is unboosted. In this state, the seen IV is boosted but the caught IV is unboosted. Currently, the only way to know the unboosted IV is if we have encountered the same spawn before in a different state (having different PokemonDisplay and/or under different weather).

Another way to scan IV is to actually catch the Pokemon. Two things could happen:

1. Pokemon was caught. In that case, we would be able to read out its IV from CatchPokemonOutProto.
2. Since the bcr of Ditto is pretty low (20%), it is likely that we will not be able to catch it. However, even in this case, it appears that we can still get some information. In particular, CatchPokemonLogEntry.CombatPoints will have the correct CP set.

For example, we have a 0P Ditto disguised as Snubbull as below, encountered as Lv32 and CP 1050.

Catching the Pokemon will give a Ditto at Lv27 14/4/13. However, on a different account where this spawn fled, we can see that the CP in the log was set to an anomalous 889, which is the CP of Snubbull at Lv27 14/4/13.

Using this correct CP + level, we can in principle reverse engineer the IV. While this in principle will not tell you a unique set of IVs, when both the level and the IV are high, it is in principle possible to identify the unique correct IV.

Clear weather

https://github.com/Dragohash/Golbat/blob/f0f35099791334e6ca7ea9ed26ae815f2be740f9/decoder/pokemon.go#L312

https://github.com/Dragohash/Golbat/blob/f0f35099791334e6ca7ea9ed26ae815f2be740f9/decoder/pokemon.go#L327

Golden lure is id 506

eg. Voltdb

When I added logic in RDM to support the same processing as Golbat I also "fixed" the ditto with weatherboosts.

it happened that 100% Dittos were stored in DB, that's wrong when those ditto was weatherboosted

https://github.com/RealDeviceMap/RealDeviceMap/pull/386/files#diff-7883d9d04c6a4d57b65ab97bfa4b48390a245e2fe5ce68e9719cd2d27fd2418cR1218

Never enough, I know...

Won't be as significant as others optimizations, but we might flatten database load a bit.
For a starter, we should randomize cache TTL and rise overall cache time (second open for discussion).

Compared

Less cache hits

More cache hits

Load might be different, but describes a problem well.

Pokemon 114 CP225"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xa61c09]

goroutine 244420 [running]:
golbat/decoder.UpdateGymRecordWithGymInfoProto(0xc001c66050?, 0xc0000f3c20)
/home/dkmur/progs/Golbat/decoder/gym.go:444 +0x29
main.decodeGetGymInfo({0xc001c66050, 0x9, 0x9})
/home/dkmur/progs/Golbat/main.go:176 +0x12e
main.decode(0x9c, 0xc00059bef0)
/home/dkmur/progs/Golbat/main.go:108 +0x26a
created by main.Raw
/home/dkmur/progs/Golbat/main.go:358 +0x8f8
time="2022-07-24T00:04:02+02:00" level=info msg="Connected to database"
time="2022-07-24T00:04:02+02:00" level=info msg="Golbat started"

Example grunt - showing line up and reward

OpenInvasionCombatSessionProto          incident_lookup { incident_id: "5477255916644222015" fort_id: "d36fb83acc5f4516bcf2e3c083145f65.16" fort_lat: 54.090941 fort_lng: 13.397025 } step: 1 attacking_pokemon_id: 6253405486787881069 attacking_pokemon_id: 7051996254806144273 attacking_pokemon_id: 4049703279889415729 lobby_join_time_ms: 1659221762713
    Response | OpenInvasionCombatSessionOutProto      status: SUCCESS combat { combat_state: ACTIVE combat_id: "COMBAT_1659221782019_-5139414284709935119" player { public_profile { name: "Flux4218" level: 30 avatar { avatar: 1 avatar_hair: "AVATAR_f_hair_default_0" avatar_shirt: "AVATAR_f_shirt_default_5" avatar_pants: "AVATAR_f_pants_default_4" avatar_hat: "AVATAR_f_hat_default_A_5" avatar_shoes: "AVATAR_f_shoes_default_1" avatar_eyes: "AVATAR_f_eyes_0" avatar_backpack: "AVATAR_f_backpack_default_0" avatar_gloves: "AVATAR_f_gloves_default_0" avatar_socks: "AVATAR_f_socks_default_0" avatar_belt: "AVATAR_f_belt_default_4" avatar_necklace: "AVATAR_f_necklace_default_0" } caught_pokemon: 20 experience: 2104205 } active_pokemon { pokemon_id: 6253405486787881069 pokedex_id: ANORITH cp: 1094 cp_multiplier: 0.7068842 stamina: 96 max_stamina: 96 move1: SCRATCH_FAST move2: ANCIENT_POWER pokemon_display { gender: FEMALE form: ANORITH_NORMAL display_id: -2902952500726692630 } individual_attack: 10 individual_defense: 2 individual_stamina: 8 pokeball: ITEM_ULTRA_BALL } reserve_pokemon { pokemon_id: 7051996254806144273 pokedex_id: AIPOM cp: 995 cp_multiplier: 0.7068842 stamina: 108 max_stamina: 108 move1: SCRATCH_FAST move2: AERIAL_ACE pokemon_display { gender: FEMALE form: AIPOM_NORMAL display_id: 2480148140930035634 } individual_attack: 15 individual_defense: 1 individual_stamina: 8 pokeball: ITEM_ULTRA_BALL } reserve_pokemon { pokemon_id: 4049703279889415729 pokedex_id: RAMPARDOS cp: 2054 cp_multiplier: 0.65443563 stamina: 146 max_stamina: 146 move1: ZEN_HEADBUTT_FAST move2: FLAMETHROWER pokemon_display { gender: MALE display_id: 4961293038146386647 } individual_attack: 12 individual_stamina: 5 pokeball: ITEM_ULTRA_BALL } minigame_defense_chances_left: 2 lobby_join_time_ms: 1659221762713 } opponent { public_profile { name: "CHARACTER_FIGHTING_GRUNT_FEMALE" } active_pokemon { pokedex_id: MAKUHITA cp: 1754 cp_multiplier: 1.0204883 stamina: 116 max_stamina: 116 move1: ROCK_SMASH_FAST move2: CROSS_CHOP pokemon_display { gender: MALE alignment: SHADOW } individual_attack: 91 individual_defense: 15 individual_stamina: 9 } reserve_pokemon { pokedex_id: HITMONCHAN cp: 5004 cp_multiplier: 1.0204883 stamina: 92 max_stamina: 92 move1: BULLET_PUNCH_FAST move2: CLOSE_COMBAT pokemon_display { gender: MALE alignment: SHADOW } individual_attack: 153 individual_defense: 15 individual_stamina: 9 } reserve_pokemon { pokedex_id: MACHOKE cp: 4373 cp_multiplier: 1.0204883 stamina: 125 max_stamina: 125 move1: LOW_KICK_FAST move2: BRICK_BREAK pokemon_display { gender: MALE alignment: SHADOW } individual_attack: 143 individual_defense: 15 individual_stamina: 9 } minigame_defense_chances_left: 2 combat_npc_personality_id: "TRAINER_PERSONALITY_EASY" } combat_start_ms: 1659221792019 combat_end_ms: 1659222062019 server_ms: 1659221782019 }

1. Implement RDM-style cell checking where golbat will notice if forts removed.
2. Ensure there is a pokestop / gym webhook sent to indicate a fort is removed

while playing with fortwatcher I notice missing webhooks for fort_update.
for me, the flow for new forts is often:
1 added
2 edit name+image
3 edit description

Example, added is send(step 1), the name edit (step 2) never received

eecf6e240f6831eeae2f165b363ea3a5.16
[20230516 21:08:36] Send added pokestop id: eecf6e240f6831eeae2f165b363ea3a5.16 location: 51.628082,4.926408 fence: Dongen name: "Unknown" time: 1.096 s
[20230516 23:12:24] Send edit pokestop description id: eecf6e240f6831eeae2f165b363ea3a5.16 fence: Dongen name: "Broederpad Links" time: 0.768 s
first_seen_timestamp for eecf6e240f6831eeae2f165b363ea3a5.16 2023-05-16 21:08:34

Example, added was never received but I do get an edit name(step2)

eec54b8f48d53bb98b008b720a43291e.16
[20230515 21:45:10] noHook edit pokestop name id: eec54b8f48d53bb98b008b720a43291e.16 fence: unfenced name: "Fietsroutenetwerk Drechtsteden" oldname: "Unknown" time: 0.355 s
[20230515 23:16:04] noHook edit pokestop description id: eec54b8f48d53bb98b008b720a43291e.16 fence: unfenced name: "Fietsroutenetwerk Drechtsteden" time: 0.456 s
first_seen_timestamp for eec54b8f48d53bb98b008b720a43291e.16 2023-05-15 20:50:05

We should define them somehow :)

#72

we removed capture_1, capture_2, capture_3 from processing because those values are not relevant for users. also worker can influence the capture rate with catch medals, so it's not always the base catch rate

we will need a way to keep track of the base catch rate, and then we can do some calculations in the frontends or poracle :)

Does old Pokémon still work?

What do you think about adding Prometheus pull-type metrics to Golbat?

That would allow monitoring almost everything, store in TSDB, display using Grafana and even create alerts. Quite powerful stack used (mostly) by big companies.

Prometheus is open source (and written in Go). Here's very basic example of implementation in Go https://prometheus.io/docs/tutorials/instrumenting_http_server_in_go/

It's similar to @dkmur (great!) solution, except Golbat would handle counters inside and expose them to Prometheus server through a single HTTP endpoint. No need for queries or crons.

Clear incident table when archive mode is set

why?

To support misleading characterisation of spawn-points

if pokestop transitioned to gym this cache entry should expire if not raw processed anymore, so imo we could keep the whole cache in that case?! :)

Ability to switch off data processing based on context sent along with data. Would allow us to select what type of data should be ignored in processing depending on what context worker sends and Golbat configuration.

Implementation #90

Another possibility to save CPU time would be to optimize golbat/geo.MatchGeofences

It's worth checking the approach with indexing bbox of polygons inside RTree index. Depending on results, we might as well move MatchGeofences higher to check all incoming locations.

Building a Geospatial cache in Go

As we preserve IV values now on weather change when mon is still boosted we don't send pvp value in webhook anymore.

This is happening due to the pokemon processing rework / ditto detection improvement of @Mygod. As we don't store pvp value in Cache (to reduce memory usage) we should either recreate pvp value on the fly or find another way :)

https://github.com/Dragohash/Chansey/blob/7cb4e4ab0e5eda31bd901f07f27083ddacbae007/default_files/procedures.sql.default#L3

Golbat shouldn't write to disk to make stats, it should count into buckets by itself

So in memory / db do not get out of sync

I have a gym with an empty id and lat/lon = 0 in my db, assuming the game sent that wrong for whatever reason. I think this happened on a GymGetInfoProto.

Poffins would be nice to have I guess.

Probably retrospectively categorising spawnpoints created during event times

they're super straight-forward and come in very handy. This must be done if we want to use AirScout in production.

message GetMapFortsOutProto {
	enum Status {
		UNSET = 0;
		SUCCESS = 1;
		ERROR = 2;
	}

	message FortProto {
		string id = 1;
		string name = 2;
		double latitude = 3;
		double longitude = 4;
		repeated Image image = 5;
	}

	message Image {
		string url = 1;
		string id = 2;
	}

	repeated FortProto fort = 1;
	Status status = 2;
}

Use timezone database
https://github.com/evansiroky/timezone-boundary-builder

For future peeps keep the in_memory option set to disabled for now. MyGod added a bunch of ditto detection code and someone, cough Turtle cough, promises to get this added.

If you see the following error messages that's why.

ERRO 2023-03-30 22:42:02 Error pokemon [XXXXXXXXXXXX]: no such column: iv_inactive
ERRO 2023-03-30 22:42:02 getOrCreatePokemonRecord: no such column: iv_inactive
ERRO 2023-03-30 22:42:02 Error pokemon [XXXXXXXXXXXX]: no such column: iv_inactive
ERRO 2023-03-30 22:42:02 getOrCreatePokemonRecord: no such column: iv_inactive
ERRO 2023-03-30 22:42:02 getOrCreatePokemonRecord: no such column: iv_inactive
ERRO 2023-03-30 22:42:02 getOrCreatePokemonRecord: no such column: iv_inactive

Provide an api to allow dragonite to indicate that a quest area is being rescanned, and so should be cleared

https://github.com/RealDeviceMap/RealDeviceMap/pull/410/files

spotted by fabio in his swift conversion

I would like to discuss how to process invasion teams & confirmed Giovanni

Refer to #12 for real invasion data examples.

Giovanni

Raw Data

This data is best extracted from a StartIncidentOutProto.

StartIncidentOutProto.Incident.step is an array of ClientIncidentStepProto. In it, the confirmed grunt character is in both pokestop_dialogue.dialogue_line.character (ClientPokestopNpcDialogueStepProto) and invasion_battle.character (ClientInvasionBattleStepProto)

A confirmed Giovanni is type 44. Male Decoy is 45 and Female Decoy is 46 (used by the game). Note that "unconfirmed" Giovanni is also 44

Processing

• Add a new bool column to the incident table. I suggest calling it character_confirmed
• After processing this, send a new webhook. This can then also have the same character_confirmed bool field.

Notes

• There's a relatively fresh _u flag for grunt UICONs. This is part of the standard but afaik, no uicon repo supports it. Reference PR: UIcons/UIcons#34

Teams

Raw Data

This data is provided in a OpenInvasionCombatSessionOutProto.

OpenInvasionCombatSessionOutProto.combat.opponent.active_pokemon is the first slot, OpenInvasionCombatSessionOutProto.combat.opponent.reserve_pokemon is an array containing the second and third slot (in order).

These Pokemon are of type CombatPokemonProto. Refer to #12 for details on this.

You may want to abort processing if the array size != 2. This should never happen. Also only process OpenInvasionCombatSessionOutProtos that have teh SUCCESS status. It's not uncommon these can fail.

Processing

I'm not sure how to best tackle this.

Pokemon Data

• Useful: Pokedex ID and Form Id
• Possibly useful: Gender (not sure if same across players)
• Possibly useful: Moves (not sure if same across players), possibly interesting for hardcore players, but most likely not worth storing
• Likely useless: CP/CPM/IVs. Most likely different across players

Storing

• If we only decide to store pokedexId and formId, we would get away with adding 6 new columns to the incidents table: slot_x_pokemon_id and slot_x_form where x is 0, 1, 2 (or 1, 2, 3)
• If we decide to store more, we would probably have to make a new table. This will become quite unperformant and kind of ugly
• When a team is detected, send out a incident webhook with a field that should look like this:

team: [
    {
        "pokemon_id": 0,
        "form": 0,
        ...
    },
    ...
]

Use real account names for updating account name fields

Probably query:

update pokestop join gym on gym.id=pokestop.id set pokestop.deleted = 1 where pokestop.deleted = 0 and gym.deleted = 0 and gym.updated > pokestop.updated;
update gym join pokestop on gym.id=pokestop.id set gym.deleted = 1 where pokestop.deleted = 0 and gym.deleted = 0 and gym.updated < pokestop.updated;

-36.84929156444202,174.76839760477938
That's too accurate, we should cut them on pokemon

https://github.com/Dragohash/Golbat/blob/00ff7df59d2afb72bdbe46af6b6ce17b64e12d7e/decoder/gym.go#L189