naschorr / hawking Goto Github PK

Build up a docker container for simple setup and spinup of the bot.

Some considerations:

• Is there a best practice for passing sensitive keys to the container?
• Is there a nice way of passing production configuration files to the container, or is a volume binding the best way? Maybe environment variables that override the default config.json properties?
• What about additional phrases? Normally those are git cloned into an adjacent directory that the bot knows to pull from. I guess this could be another volume binding, or maybe it could automatically pull the latest phrases on initialization?

Currently, the bot doesn't indicate that it's a permissions issue when it can't connect to the user's voice channel.

Should probably have sections on:

• Module skeleton
• Describing how discover_modules works
• The ModuleInitializationStruct
• Implementing dependencies for your new module
• Practical example(s?)

Currently, commands are all in lowercase, but it really shouldn't matter if the command was typed fully or partially in uppercase.

For example, \say, \Say, and \SAY should all invoke the 'say' command.

If the bot's path has a space in it, say for example D:\Google Drive\hawking, then the bot will fail to save the generated .wav file as expected (because the temp/ folder for storing the generated .wav files falls under it). It'll end up returning an error:

'D:\Google' is not recognized as an internal or external command, operable program or batch file.
Exception in say(): Unable to save a proper .wav file.

Clearly it's just an issue with the command prompt not handling spaces in path names properly, but I messed around with adding in double-quotes around the path, and it still doesn't work properly.

System: Windows 10
Version: 0.1.0

ex: \say test works, but \say this is a test will break.

Right now, the find command either says the found command, or lets the user know that nothing was found. It should also let the user know which command was found, for later use.

Ideally I want to be able to stick some extra command line options to tweak values in config.json when executing Hawking in a development environment. Kinda like those Java VM args in IntelliJ, but for python obviously. I don't think Python (or VS Code) supports anything like that, but maybe an extra config option that would point to another config file which would have precedence?

Really I'm just tired of lowering the channel_timeout_seconds value when I'm making changes quickly.

Edit: Actually just implement this: https://github.com/theskumar/python-dotenv

When repeatedly spinning up and down Hawking during development, the logger doesn't actually shutdown properly. This causes the log file to get locked and be unable to be edited later on. Additionally, it also spams the console with some pretty massive tracebacks.

When a user mistypes a phrase command (ex. \maden instead of \madden), the bot should not only alert the user that their command wasn't valid, but also attempt to figure out what command they wanted to invoke.

Theoretically this should just involve copying this code from my other bot, Clipster.

With the Discord API refresh being built around Clipster, I didn't end up adding the TTSController.delete method back in as a callback.

It's got several references to Hawking, yet really it should be able to be dropped into Clipster without fuss.

With the number of preset phrases growing, users should be able to search for them by typing in the bits that they remember, and the bot will attempt to find what they're looking for.

This will also require adding an extra property to each phrase that holds the text content of the phrase in plain english, without any phonemes.

Realistically, this should just require copying this code from my other bot, Clipster.

I had a question on the discord about this, and I realized that it wasn't super obvious what to do. Basically there wasn't much documentation, and it doesn't really give any direction about how to get a Reddit bot authenticated.

• Updated admin skip logic to reuse the audio_player implementation, but with some flavor of guard/check
• Properly check if any users in the voice channel are bots or not
• Remove the weird non percentage based skip system, that's always been not great

Might be worthwhile, might not be. Will it be relevant to Hawking, and how do the slashes work with the existing activation string?

This doesn't really make sense anymore now that a proper logging implementation exists

Right now it does a multireddit search for the top 250 posts of the last month, which leads to the questions being a bit too smart. I think dropping that down to a week might produce some better results. Or at least the results were a bit better in my quick scan on the app.

Right now it just fails silently, and that's not great.

Basically, the syntax would be \preset @username. This would send Hawking to the voice channel that @username is connected to.

They're alphabetized right now, which doesn't really make sense for new users. This should really be sorted based on usage priority.

A potential order:

• say
• skip
• find
• random
• help
• fortune
• invite
• speech_config

The skip feature needs to take into account bots (and potentially afk people), as well as go back to a percentage only system.

Sometimes, Discord will have issues that make the bot unreliable, leading to command errors. When the on_command_error method is invoked, it should never suggest the same command that it was given.

Bot says "crack" as "cracker" (krakər). That's some racist-ass shit.

The file hasn't been touched in ages, so let's change that.

Actually, it didn't work several months ago, when I firsts noticed them, but it still doesn't now. Ya know?

I've been working on a fix for the core audio playing logic with my other bot, Clipster. Now that I'm happy with those changes, I can import that logic in and it should be a simple job of connecting the bits to get Hawking back up and running.

I'm planning on implementing a way for the bot to notify all users of downtime, restarts and any issues that may be encountered. To do this, the bot needs a way to send messages to all the connected voice clients.

I've been noticing that Hawking (and Clipster) will seem to come online and offline more often than they should. After taking a look at the logs, I've noticed that there are some instances of the socket closing, and the service having to reconnect it automatically. websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason. Upon further inspection, there is a constant type error that seems to cause this issue:

Traceback (most recent call last):
  File "/usr/local/bin/hawking/code/hawking.py", line 206, in <module>
    hawking.run()
  File "/usr/local/bin/hawking/code/hawking.py", line 198, in run
    self.bot.run(utilities.load_json(self.token_file_path)["token"])
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/client.py", line 640, in run
    return future.result()
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/client.py", line 621, in runner
    await self.start(*args, **kwargs)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/client.py", line 585, in start
    await self.connect(reconnect=reconnect)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/client.py", line 499, in connect
    await self._connect()
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/client.py", line 463, in _connect
    await self.ws.poll_event()
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/gateway.py", line 471, in poll_event
    await self.received_message(msg)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/gateway.py", line 425, in received_message
    func(data)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/state.py", line 750, in parse_guild_create
    guild = self._get_create_guild(data)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/state.py", line 725, in _get_create_guild
    guild._from_data(data)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/guild.py", line 296, in _from_data
    self._sync(guild)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/guild.py", line 323, in _sync
    self._add_channel(TextChannel(guild=self, data=c, state=self._state))
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/channel.py", line 107, in __init__
    self._update(guild, data)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/channel.py", line 131, in _update
    self._fill_overwrites(data)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/abc.py", line 294, in _fill_overwrites
    self._overwrites.append(_Overwrites(id=overridden_id, **overridden))
TypeError: __new__() got an unexpected keyword argument 'deny_new'

Fortunately, I'm not the only one to experience this (see: Rapptz/discord.py#5109). The recommended course of action seems to be to update discord.py to the latest version (1.3.4), as it's currently on 1.3.2

Everything's just tossed into the same folder, which was tolerable back when the bot was much smaller, but nowadays it needs some organization.

The message in question:
\say EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST EST PST CST MST

Takes more than 10 minutes to be functional again and accept commands

Occasionally a user will invoke a command for the bot, and nothing will happen. This is usually because discord isn't playing nice for whatever reason, so the bot should make an attempt to let the user know.

Looks to be mostly compatible, though not 100%.

The big one seems to be that queuing up multiple play_audio requests doesn't result in them being processed sequentially, but rather they're immediately executed when the command is sent.

I noticed some pretty aggressive error logging happening due to failed database insert operations (new dev machine, hadn't added credentials back). I was also seeing tons of errors from failed logger output as well. However, after addressing those issues that queuing issue doesn't seem to be presenting itself any more.

This will also allow for an aiohttp bump to version 3.7.4, which fixes a minor vulnerability. (See #72)

Stats are fun, so it might be neat to cache some info about how many guilds Hawking is in, how many users have used it, and how many commands have been invoked thus far. Maybe it could be invoked by anyone with \info or similar

Slightly more complicated, as this'll affect other dependencies too

Sorry, new to Github.

Was working earlier today, then stopped responding to the \say command. Still responds to \help command. Wondering if this is an issue for my server or just something with the bot.

Currently, the ModuleEntry and ModuleManager classes are located inside the hawking.py file. They really shouldn't be in there.

Seeing this error: socket.gaierror: [Errno 11001] getaddrinfo failed which smells like it's time to update discord.py

Clipster and Hawking both share this code, and in the future I'd like to have a set of core modules that can be maintained in a single place and distributed to both bots. During the Discord API refresh, I made it mostly generic, but during the porting process to get Hawking back online, I noticed a few places that still need some work.

• Genericize all instances of "clips" to be audio
• Migrate the \say command out and into own module
• Ensure the play_audio methods just handle taking an audio file and getting it injected into the audio player event loop
• Rewrite the disconnect message logic to take a method to handle prepping the phrase/clip
• Start removing unneeded class members
• See #41

Via https://discord.com/channels/521474080340180998/769086411080073227/835344431766044732

It's not really obvious how to customize the speech, beyond looking through the phrase sources, which is kind of terrible. I'm thinking that there should be a \help voices command or something similar that describes how to use different voice actors, as well as customizing those voices. This wouldn't be super complex, just simple things like how to change the pitch, gender, head size, speed, etc.

Since the help sub commands (ex. 'internet' in '=help internet') are commands themselves, they can be suggested when a user makes an invalid command. This leads to a useless cycle where someone doing the '=internet' command would be suggested to do a '=internet' command instead.

Basically just a copy of naschorr/clipster#1. Should be the same easy fix, but with some extra logic for the extra modules. The command list that the error command is compared to should probably be generated/updated when the modules are updated.

Kudos to the user that reported this in the discord. The bug itself only seems to happen on a per server basis, as Hawking will work fine on another server created by the owners of the affected server. Also, kicking and re-inviting Hawking to the affected server won't fix the issue. It only seems to resolve itself upon reboot.

Here's the relevant redacted logs that should correspond to the issue:

2021-03-06 21:00:01,371 - audio_player - audio_player_loop - ERROR - CancelledError during audio_player_loop, ignoring and continuing loop.
Traceback (most recent call last):
  File "/usr/local/bin/hawking/code/audio_player.py", line 205, in audio_player_loop
    self.active_play_request = await self.audio_play_queue.get()
  File "/usr/lib/python3.6/asyncio/queues.py", line 167, in get
    yield from getter
concurrent.futures._base.CancelledError
2021-03-06 21:00:01,374 - audio_player - audio_player_loop - ERROR - CancelledError during audio_player_loop, ignoring and continuing loop.
Traceback (most recent call last):
  File "/usr/local/bin/hawking/code/audio_player.py", line 205, in audio_player_loop
    self.active_play_request = await self.audio_play_queue.get()
  File "/usr/lib/python3.6/asyncio/queues.py", line 167, in get
    yield from getter
concurrent.futures._base.CancelledError
2021-03-06 21:00:01,374 - audio_player - audio_player_loop - ERROR - CancelledError during audio_player_loop, ignoring and continuing loop.
Traceback (most recent call last):
  File "/usr/local/bin/hawking/code/audio_player.py", line 205, in audio_player_loop
    self.active_play_request = await self.audio_play_queue.get()
  File "/usr/lib/python3.6/asyncio/queues.py", line 167, in get
    yield from getter
concurrent.futures._base.CancelledError
2021-03-06 21:00:01,374 - audio_player - audio_player_loop - ERROR - CancelledError during audio_player_loop, ignoring and continuing loop.
Traceback (most recent call last):
  File "/usr/local/bin/hawking/code/audio_player.py", line 205, in audio_player_loop
    self.active_play_request = await self.audio_play_queue.get()
  File "/usr/lib/python3.6/asyncio/queues.py", line 167, in get
    yield from getter
concurrent.futures._base.CancelledError
2021-03-06 21:00:01,381 - audio_player - audio_player_loop - ERROR - CancelledError during audio_player_loop, ignoring and continuing loop.
Traceback (most recent call last):
  File "/usr/local/bin/hawking/code/audio_player.py", line 205, in audio_player_loop
    self.active_play_request = await self.audio_play_queue.get()
  File "/usr/lib/python3.6/asyncio/queues.py", line 167, in get
    yield from getter
concurrent.futures._base.CancelledError
2021-03-06 21:00:01,382 - audio_player - audio_player_loop - ERROR - Exception inside audio player event loop
Traceback (most recent call last):
  File "/usr/local/bin/hawking/code/audio_player.py", line 218, in audio_player_loop
    voice_client = await self.get_voice_client(self.active_play_request.channel)
  File "/usr/local/bin/hawking/code/audio_player.py", line 141, in get_voice_client
    return await channel.connect()
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/abc.py", line 1101, in connect
    await voice.connect(reconnect=reconnect)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/voice_client.py", line 252, in connect
    await self.ws.poll_event()
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/gateway.py", line 822, in poll_event
    await self.received_message(json.loads(msg.data))
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/gateway.py", line 753, in received_message
    await self.initial_connection(data)
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/gateway.py", line 776, in initial_connection
    recv = await self.loop.sock_recv(state.socket, 70)
concurrent.futures._base.CancelledError
2021-03-06 21:00:01,397 - audio_player - audio_player_loop - DEBUG - Playing file at: /usr/local/bin/hawking/temp/1615033724883.wav, in channel: <REDACTED>, in server: <REDACTED>, for user: <REDACTED>
2021-03-06 21:00:01,397 - audio_player - audio_player_loop - ERROR - Exception inside audio player event loop
Traceback (most recent call last):
  File "/usr/local/bin/hawking/code/audio_player.py", line 278, in audio_player_loop
    voice_client.play(self.active_play_request.audio, after=after_play_callback_builder())
  File "/usr/local/bin/hawking/lib/python3.6/site-packages/discord/voice_client.py", line 402, in play
    raise ClientException('Not connected to voice.')
discord.errors.ClientException: Not connected to voice.

It'll probably just be a bunch of unit tests at this point. However, I think there are some decent candidates for integration testing, namely:

• The data flow from between Hawking receiving a \say command, and the audio_play_queue processing that audio, and popping it from the queue
• How \skip commands affect the audio_play_queue flow
• The data flow between a user requesting their request data to be deleted, and the deletion process actually happening
• Phrase scanning, ingesting, command building, and finally registering those commands with Hawking
• Module discovery, validation checking, importing, and accessibility

Additionally, should modules be treated differently for testing, or do they fit into the same harness as everything else? What's the industry standard Python testing framework? I've got experience with the built-in unittest framework, but I don't know how that stands up these days.

That way it makes more sense for other modules to depend on it, rather than building their own interface

Right now, module data is really just thrown into the modules folder. This was fine back when modules didn't really have any external configuration, but the StupidQuestions module has thrown that out the window. I think that for organizational purposes, modules should probably exist inside their own subfolders inside the modules directory, wherein they can keep whatever additional files close at hand. Theoretically this would also allow modules to become fully fledged, cohesive python programs themselves that are built using a sensible structure.

• Discord requires that users can request that their data be deleted, and perform that deletion within a week of them making the request, so...
  • Switch from a rotating log to a timed rotation, that'll automatically purge all logs after a week.
  • Add a self service command so that user's can request that their data be deleted. To keep Amazon happy (since I'm using AWS's DynamoDB as my DB), this should probably queue up the delete requests, then run them all at once, once a week.
• Adopt an age-out policy for stored data.
  • Currently, data stored in the DB is there forever, and it's not really necessary, other than for me to look at how many rows there are and feel smug about it. I'm thinking that rows older than a year should probably just age out automatically. DynamoDB has got a nice Time to live attribute built in, so this should be easy. I would still like to track total usages of Hawking, however this can really just be a single number. (See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStarted.Python.03.html#GettingStarted.Python.03.04)
• It'd also be nice to have a Discord guild set up for Hawking, so people could interact with me in a less cumbersome way, so I should probably do that to.