A python bot (powered by computer vision) used to play City Island 5.

Access a more comprehensive guide in this article here: https://paulonteri.com/thoughts/play-game-with-computer-vision.

Here's a screen recording of the game play:

I've been playing strategy + city building + simulation? games like TownsMen 6, Clash of the Clans, SimCity for the last 10 years.

On trying out City Island 5 I found it mildly irritating that my collectibles could not accumulate while I was outside the game. I might have had the best businesses, strategy, etc but I had to be in the game to ensure I collect the cash/keys/gold overtime. For example if my bakery makes €100 per minute I would only earn €100 after leaving the game and coming back 24 hours later.

This became especially tiresome while trying to accumulate €5,000,000 required to buy the island shown below. This would take me roughly two weeks of gameplay if I don't spend any money - it's not worth it.

I needed a way to capture the live game feed.

The easiest way to to capture an in-game screenshot and pass it to the next steps in the script.

We need a way to detect a valuable in the game's feed then return its coordinates.

OpenCv's Template Matching algorithms are perfect for this.

They are used for searching and finding the location of a template image (like a valuable) in a larger image (like the game's feed). It simply slides the template image over the input image (as in 2D convolution) and compares the template and patch of input image under the template image. Several comparison methods are implemented in OpenCV. (You can check docs for more details). We use it in the method: cv2.matchTemplate(... )

To achieve this, I needed the template images. I took screenshots by hand then cropped off the cash, star and key:

Once we have the coordinates of an item we can try to click on it.

pyautogui's .click(x,y) function works like magic for this. It clicks the screen on the coordinates x and y where our valuable is lying. Learn more about it here.

Our clicks above may result in popups when we are being given a reward, levelling up, etc.

We need to close it before attempting to collect valuables again. We use the same logic used in finding and clicking on valuables.

To achieve this, I needed the template images for the popups' close buttons so that they can be clicked. I took screenshots by hand then cropped off the various close buttons:

We do the steps above repeatedly to collect the valuables while the script is running.

I started the game with €316,415.

The following morning I had €6,463,870.

I made €6,147,455 overnight!

I then proceeded to buy the Island I wanted:

• This is cheating.
• Why get a game if you're not the one playing it?

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Current maintainers:

• Paul Onteri - https://paulonteri.com

Apache License 2.0

Access the full guide in this article here: https://paulonteri.com/thoughts/play-game-with-computer-vision

This was fun!