Now that we've finished #127, it's worth doing a bit of work on the type: ignores in our mypy annotations, trying to fix upstream in typeshed anything where that's possible.

I don't want this issue to remain open forever, so we should happily close this as soon as we've done a sweep through our type: ignore entries.

@neiljp I feel like you'd probably be pretty effective at doing this.

Example:

    c = zulip.Client(
        email="[email protected]",
        site="https://zulip.com/api",
        api_key="key")
    c.call_on_each_event(
        lambda msg: print(str(msg) + "\n"))

returns:

Server returned error:
event_types is not a list

The current argument for call_on_each_event's event_types is defined as None, but it's never set to a list within the function body. Passing an empty list for event_types works as expected.

See here for context:

zulip/zulip#4975

I'm not sure the current code for the John bot is worth maintaining, unless somebody is excited about fixing it to do AI kind of stuff.

To this point, several bots have shown difficulties to cope with empty messages, leading to crashes.

As discussed in #54 , this issue could be addressed by letting a bot set a state in its class whether they want to receive empty messages.

Steps involved:

• Modify lib.py to check a bot's ACCEPT_EMTPY_MESSAGES state and conditionally send messages.
• Inform the user in case of missing empty message support.
• (add tests for lib.py to verify the feature's functionality? Probably worth another PR)

With the code from #27 I see this detail of the failure using python3 -m unittest discover:

AssertionError: Calls to send_reply do not match those expected
MESSAGE:
{'content': 'New York'}
ACTUAL CALLS:
[call({'content': 'New York'}, 'Weather in New York, US:\n71.32999999999998 F / 21.850000000000023 C\nMist')]
EXPECTED:
[call({'content': 'New York'}, 'Weather in New York, US:\n71.33 F / 21.85 C\nMist')]

The test passes with python2 (2.7.13), whereas python3 here is 3.5.3 currently.

I'm guessing this is some kind of rounding error, and will look into it.

zulip-api-examples is a little script that prints the path to the examples directory of the installed zulip package.

We should document it in our API docs. At the creation time of this issue, the proper place to add such a documentation (/api/) did not have the right layout adding a section like this (see the discussion here. Once that's resolved, someone can claim this.

see zulip/zulip#5359

from vabs22:

Add supervisord configuration for running the server persistently and simplifying the server setup.
e.g. install-zulip-flask-supervisord <path_to_flaskbotrc>

UPDATE Scroll down to the latest comments to find a list of the remaining bot files not at 100% coverage.

NOTE: All striked-through bots have been deprecated, all checked bots have been given tests. See the comments for more information on how to proceed.

We have a couple bots in zulip_bots/zulip_bots/bots that are untested:

- [ ] commute
- [ ] foursquare
- [ ] git_hub_comment
- [ ] github

• github_issues

- [ ] googlesearch
- [ ] howdoi
- [ ] john
- [ ] youtube

For each bot, we should either

• add tests. You can find help in our guide for bot testing.

• deprecate the bot. If a bot seems to be bugged or completely useless, we should discuss it on chat.zulip.org and decide whether it is worth keeping.

The Zulip server repo's docs/THIRDPARTY file has the following information about the files that were moved from there into this repo:

Files: api/*
Copyright: 2012-2017 Dropbox, Inc., Kandra Labs, Inc., and contributors
License: Expat

Files: api/integrations/perforce/git_p4.py
Copyright: 2007 Simon Hausmann <[email protected]>,
 2007 Trolltech ASA
License: Expat
Comment: https://raw.github.com/git/git/34022ba/git-p4.py

Files: bots/jabber_mirror_backend.py
Copyright: 2013 Permabit, Inc., 2013-2014 Dropbox, Inc.
License: Expat

Now that this code is in its own repo, we should move that information here, and also add a LICENSE file with the Expat license to help make that part clear.

See zulip/zulip#5152.

Currently, it is possible for multiple bots to run simultaneously using the same config file. We should find a way to ensure that only one bot can use one zuliprc file at a time.

Example chat log:

Query: @Bot One #7529

Reply:
beep boop // By hello world bot
from [email protected]: #7529 // By followup bot
zulip/zulip#7529 - Bugdown images // By github_detail bot
Created by aero31aero
Status - Open

    Fixes #7324.

The doc should be added in the zulip repo, next to the other integration docs.

Points worth mentioning in the doc:

• how to run log2zulip
• adding a cron job for log2zulip
• sample config file for log2zulip

zulip/zulip#2771 is a PR for adding a Google Calendar bot.
The PR is big, and has potentially the most comments any PR in the Zulip project ever had.

We should:

• Copy the PR's code and create a new Google Calendar bot in this repo.
• Test the bot. If it works, we should just merge it. Then, people can collaborate on code-style issues, adding tests, etc.

The flask botserver currently creates a new ExternalBotHandler instance on each call to handle message.

Once #143 is merged, this will lead to StateHandler being reinstantiated over and over, fetching a bot's full state every time. We should make the flaskserver only keep one instance of each bot's handler for it's runtime.

Ideally, zulip_bots/MANIFEST.in should be automatically generated by setup.py and it should be ignored in .gitignore. There are a few things to keep in mind:

• We could add this as a part of a separate build step or command in setup.py.
• We need to make sure that this generation of zulip_bots/MANIFEST.in can be triggered from our main repo when installing the zulip_bots package, which might be a bit of a workaround. To see why this is a priority, see this commit.

On a more general note, I feel like I'd like to get to a point where we have a script that completely automates the different release processes for our development repo vs. the actual package PyPA release, since PyPA packages shouldn't be shipped with fixtures and logos, but the repo should have them for the reasons described here.

I'll probably pick this up eventually!

@timabbott: FYI

Thanks,
Eeshan

We have dumped Python 2 support for the zulip_bots and zulip_botserver packages. I am opening this issue to track progress on converting remaining code shreds from Python 2 legacy support to Python 3 only.

• - move all shebangs from #!/usr/bin/env python to #!/usr/bin/env python3
• - move remaining six imports
• - remove any python 2-related checks.

Feel free to comment additional steps to be taken here.

We should figure out how to fix this warning.

test_bot (zulip_bots.bots.encrypt.test_encrypt.TestEncryptBot) ... ok
test_bot (zulip_bots.bots.thesaurus.test_thesaurus.TestThesaurusBot) ... /srv/zulip-venv/local/lib/python2.7/site-packages/bs4/__init__.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 82 of the file ./tools/test-bots. To get rid of this warning, change code that looks like this:

 BeautifulSoup(YOUR_MARKUP})

to this:

 BeautifulSoup(YOUR_MARKUP, "lxml")

  markup_type=markup_type))
ok
test_404 (zulip_bots.bots.github_detail.test_github_detail.TestGithubDetailBot) ... ok

What I don't get is where we're calling BeautifulSoup the wrong way, given this:

$ git grep Beautif
zulip_bots/zulip_bots/bots/youtube/youtube.py:from bs4 import BeautifulSoup
zulip_bots/zulip_bots/bots/youtube/youtube.py:            soup = BeautifulSoup(r.text, 'lxml')

I'd love to see this debugged, since we actually have a similar problem in the server project with some code using BeautifulSoup...

For certain advanced API applications, it will be useful to set out a timer for server requests. For example, I may want to only wait 5 seconds to get the next message in the stream, and if the message doesn't come, I want to do something else in real time like send a message myself or go to sleep for a bit, etc.

This would be a fairly technical project for somebody who understands long polling and sockets. The main file to modify would be https://github.com/zulip/python-zulip-api/blob/master/zulip/__init__.py

Once we had low-level support in our API bindings, we would probably add stuff to our BotHandler API to support things like scheduler bots and real-time game bots.

ping @cory2067

Because of the move to pip install zulip (instead of having people clone the repo or download a tarball), it's now very easy to not find the API examples that come with the package. We should probably do some combination of:

• Adding something to our API docs that point to the example code (E.g. just a link to the directory on GitHub)
• Having a recommended command that you can run after you've done pip install zulip that will find the example code. This is tricky because the files will be in some crazy path under the virtualenv's share/ directory (if you're using one) or maybe /usr/local/share if you installed as root or something.

(Also, I actually can't find the examples in our virtualenvs generated by tools/provision; are they not included in the package?)

See zulip/zulip#249

Moving the issue here, let's have a discussion on what parts of the original one are still relevant.

The current tests do not accurately simulate the real world scenarios for the bot. It would be great if we use the test responses supplied on the baremetrics API documentation as fixtures. These contain actual data for "list-plans", etc., so these tests would verify that the bot works correctly with "real" data.

This was on suggestion of @roberthoenig

Context

Soon, our mock StateHandler for external bots is going to interact with a real database on a Zulip server, instead of locally storing values at runtime.

Problem

While bots may access the information stored in the database several times when handling a message, we should keep the number of queries made to the server at a minimum.

Possible solutions

• Do a single query fetch for the dict that is the state for a given key (e.g. for our tic-tac-toe bot, key might be the thread) and then when we're done, save it back to the server at the end; not do a query for every small state change. Caching and accumulating all requests made this way, we could reduce the number of queries to one at the beginning of handle_message and one at the end, when the message is send.

• Fetch the entire state at the bot's startup and use it as a cache. For every get operation, retrieve the value directly from the cache. Accumulate all put operations and send a state change request at the end of handle_message. This approach needs only one query when the message is send. It works with the caveat that a bot can only access its own state (which is the current system).

Once agreed on a procedure, we should document it, implement it, and provide exemplary bots using it.

See zulip/zulip#4960

INFO:urllib3.connectionpool:Starting new HTTPS connection (1): owlbot.info
ERROR:root:Expecting value: line 2 column 1 (char 1)
Traceback (most recent call last):
File "bots/define/define.py", line 49, in get_bot_define_response
definitions = api_result.json()
File "/usr/lib/python2.7/dist-packages/requests/models.py", line 793, in json
return json.loads(self.text, **kwargs)
File "/usr/lib/python2.7/dist-packages/simplejson/init.py", line 505, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Expecting value: line 2 column 1 (char 1)

For bots like giphy, it might make sense to validate API keys right in the initialize function after loading config info, just by sending a quick request to the third party server. This can prevent the embarrassing thing of launching a bot and have people test it before you realize your API key is wrong or out of date, or whatever.

I think it's fine to just do this in on bot-by-bot basis.

Some bots will want access to the entire message, others to just content after the mention. We should respect this by just providing both the raw content of the message as well as the content after the mention, probably as content and content_after_mention. That way, bots can interact with either or both according to what makes sense for the bot.

(As a sidenote, we probably want content_after_mention to be all content after the first mention, with no requirement for the mention to be at the start of the message)

See discussion on Zulip: https://chat.zulip.org/#narrow/stream/bots/subject/xkcd/near/317963

The initially proposed dockerfile with instructions can be found here:
https://gist.github.com/jmandel/9f3aeabf8b17292e66582657b46ed497

The steps would be:

• Build a Dockerfile that loads and automatically refreshes the flask botserver on file changes. (The file above can be used as a starting point).
• Add the Dockerfile to zulip_botserver/Dockerfile
• Add a section to zulip/docs/running-bots-guide.md explaining the setup.

Background

We have recently added mypy support to this repo, but many files are corrupted and throw errors on mypy.

Task

#126 updates the exclude_list in tools/run-mypy Each file in this list errors on mypy. We should fix each file and remove it from the list. zulip_bots/zulip_bots/bots is an exception, we don't enforce mypy annotations in this directory.

This task is great for beginners or/and those who want to get comfortable with mypy annotations.

Currently, there are three stateful bots (bots using permanent storage): virtual_fs, incrementor, and tictactoe. Some of these might not use an optimal model for handling the storage. For example, the tictactoe bot currently stores one running game per user, whereas it might be better to store games on a topic/stream basis.

Since these bots serve as examples for new stateful bots, we should:

• decide on and implement, and document an optimal storage value for virtual_fs, incrementor and tictactoe.
• document general recommendations in the writing bots guide.

This will help a lot for working towards having a complete test suite in this project.

I think we can borrow most of the logic from tools/test-backend in the server repo.

See zulip/zulip#5360

Currently, the followup bot expects to have a stream #followup with write access. This parameter should be modifiable via either a config file like in github_detail bot or some other method.

Pseudo code for the logic:

try:
    load config
    defaultsttream = config.stream
except:
    defaultstream = "followup"

The API tests actually live in our server repo.

We want to improve them a bit:

See zulip/zulip#3736

And we also might want to move them over to this repo.

ErrBot is a framework for writing bots, which I think it would be worth doing the work to integrate with the Zulip API. The code probably should go into their project, but it'll use these API bindings, so I'm opening an issue here to prioritize in relation to other issues here.

Here's their docs on how to add support:

http://errbot.io/en/latest/user_guide/backend_development/

See also errbotio/errbot#845

We need to figure out ways to run these kinds of things on our newly split out repo

• ./tools/lint (from server repo, should run on api repo and borrow code somehow)
• bots automated tests (should run on new repo ideally)
• api tests (these should actually stay on the server repo for the short term, but we could move them here if we had a way to set up a test server)

This ticket is not for beginners.

originally here: zulip/zulip#3365

from dehnert:

The Python API examples use generate_option_group and init_from_options to support a common set of options, but (mostly?) hardcode their usage messages. They should ideally display a more complete usage message as well (or instead) that reflects the other options, especially --config-file (though others are also occasionally useful, certainly).

We should set User-Agent so it is easy for the Zulip server's logging to track usage of the various bots.

The pattern we've established for this sort of thing is e.g. ZulipTrelloWebhook for our webhooks. So I think we probably want e.g. ZulipConverterBot as the naming scheme for these. Ideally, the framework would generate the client name, so there's no need to make any individual per-bot changes.

We would pass this as an extra option to zulip.Client or zulip.init_from_options (depending what we're using).

We have at least three bots related to two-player games: chess, tictactoe, connectfour.

We should try to consolidate some of the code related to two-player games, so that it's easy to create future two-player games without duplicating logic for inviting players to games, managing message flow, etc., that is really the same for all two-player games, apart from small details that could be delegated to game-specific classes.

The best example we have so far in terms of high level design is the connect-four game. The connect-four game is nice, too, because it's pretty moderate in terms of programming difficulty and fun to play.

The connect-four game has two flaws that I know of:

• It seems to have a model that there's only one active game at any time.
• It plays all games privately, instead of using streams.

If you take this ticket, you'll want to discuss ideas on the "integrations" stream.

Currently, when you deploy a bot, it silently listens on the server for incoming messages. You are not aware of the bot's activity unless the bot's owner has told you.

The idea here is to announce the bots activity, e.g. startup and shutdown.

This can be implemented in various way and flavors, details should be discussed before making progress on this. Some ways could be:

• Have the BotHandler send a message on startup/shutdown.
• Have the previous idea with an additional flag --announce
• Display active bots in the user sidebar.

Currently we have only Python 2.7 in the .travis.yml in this repo, leaving us open to regressions in our Python 3 support. We should add 3.4 to the list of Python versions, like we have in the Zulip server repo, and fix the bugs that show up when we do that.

At a quick test on my own fork of the repo, there look to be at least a couple of Python 3-related bugs we can fix.

This tweet: https://twitter.com/zulip/status/895473932278431744

shows up as the following on https://chat.zulip.org/#narrow/stream/twitter/subject/zulip/near/265158

Namely, the chat.zulip.org version doesn't have the docs.google.com link.
I'm also not sure why it has the https://twitter.com/i/web/status/895473932278431744 link .. is that the link to the attached image?

see zulip/zulip#5395 for context

We want to double check that:

• bots use StateHandler for saving state
• bots don't directly access the file system for config info

This project may already be done, and the best person to address this issue is somebody fairly involved with the project already, so they know what to check for.

In my quick test of Python 3 support, tools/test-bots printed failures but then exited with status 0, meaning success -- which means that if the python -m unittest run hadn't also failed, the Travis build would have cheerfully passed even though there were errors.

We should fix tools/test-bots to make sure it returns an exit status that reflects the success or failure of the tests. The stdlib unittest docs aren't super clear on this subject, but I suspect TextTestRunner.run(), which we call at the end of that script, returns some kind of result object, which we can inspect that to find out the result. Then we can either sys.exit(0) for success or sys.exit(1) for failure.

See zulip/zulip#4509

We've been using twitter-search-bot on chat.zulip.org for a long time, but with the Python 3 upgrade, it seems to be broken.

The last time I ran it, it seems to have lost its since_id value (And the rest of its config), because of a Python 3 compatibility exception trying to update its config file.

We should fix this (and its twitter-bot friend) to use a separate file for storing since_id.

We should not have bot_lib ever read third-party configuration data out of source-controlled files. We should instead require users to make their own config files that they can put somewhere outside the source tree, most likely in their home directory.

What we can do is provide a template file for the users, like giphy.conf.template.

Unlike most other bots, followup bot doesn't supply a help text on running @botname help.

On sending help, we should respond only with the help text of the bot stop the control flow there.

The API should avoid throwing RuntimeError exceptions for things where more specific exceptions are appropriate. At a bare minimum, we should change RuntimeError -> ZulipError (and create the latter). We may also want to make sure we throw more custom exceptions for common problems like not specifying a config file, and they could inherit from ZulipError.

If you run one of our API examples with an invalid argument, like this:

(zulip-api-py-venv) tabbott@zaset:~/python-zulip-api$ ./zulip/examples/send-message --foo
usage: send-message [options] <recipients>

Sends a test message to the specified recipients.

Example: send-message --type=stream commits --subject="my subject" --message="test message"
Example: send-message [email protected] [email protected]
send-message: error: unrecognized arguments: --foo

This doesn't explain what options do work, and thus requires that the user know that --help is the magic option for getting help). Ideally, we'd figure out some argparse configuration option that makes this work properly.

See more discussion in #1 for details on the motivation.

It is theoretically possible to run the bot like:

@botname #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 #48 #49 #50 

but this operation would take a lot of time to complete, and is not a real world scenario. Thus, we should rate limit this bot to 5 issues or PRs per message.