Code Monkey home page Code Monkey logo

twilight's Introduction

twilight

codecov badge discord badge github badge license badge rust badge

project logo

twilight is a powerful, flexible, and scalable ecosystem of Rust libraries for the Discord API.

The ecosystem of first-class crates includes twilight-cache-inmemory, twilight-gateway, twilight-http, twilight-model, and more. These are explained in detail below.

The main twilight crate is purely an advertisement crate: it has no functionality. Please use the individual crates listed below instead!

Installation

Twilight supports a MSRV of Rust 1.67.

We recommend that most users start out with these crates:

If you need any other functionality that Twilight provides, you can just add that dependency in.

Core Crates

These are essential crates that most users will use together for a full development experience. You may not need all of these - such as twilight-cache-inmemory - but they are often used together to accomplish most of what you need.

Models defining structures, enums, and bitflags for the entirety of the Discord API. It is split into a number of sub-modules, such as gateway for containing the WebSocket gateway types, guild for containing types owned by guilds (servers), voice containing the types used by the Voice WebSocket API, and more.

These are all in a single crate so that you can use gateway models without depending on twilight-gateway. One use case is if you write your own WebSocket gateway implementation.

In-process-memory based cache over objects received from the gateway. It's responsible for holding and managing information about things like guilds, channels, role information, voice states, and any other events that come from Discord.

Implementation of Discord's sharding gateway sessions. This is responsible for receiving stateful events in real-time from Discord and sending some stateful information.

HTTP client supporting all of the Discord REST API. It is based on hyper. It meets Discord's ratelimiting requirements and supports proxying.

Event processor that allows for tasks to wait for an event to come in. This is useful, for example, when you have a reaction menu and want to wait for a specific reaction on it to come in.

Additional Crates

These are crates that are officially supported by Twilight, but aren't considered core crates due to being vendor-specific or non-essential for most users.

Client for Lavalink as part of the twilight ecosystem.

It includes support for managing multiple nodes, a player manager for conveniently using players to send events and retrieve information for each guild, and an HTTP module for creating requests using the http crate and providing models to deserialize their responses.

Create display formatters for various model types that format mentions. For example, it can create formatters for mentioning a channel or emoji, or pinging a role or user.

Utility crate that adds utilities to the twilight ecosystem that do not fit in any other crate. Currently, it contains:

  • A trait to make extracting data from Discord identifiers (Snowflakes) easier;
  • A calculator to calculate the permissions of a member in a guild or channel.

A trait and some implementations that are used by the gateway to ratelimit identify calls. Developers should prefer to use the re-exports of these crates through the gateway.

Examples

The following example is a template for bootstrapping a new bot using Twilight's HTTP and gateway clients with its in-memory cache. In order to run this, replace the contents of a new project's main.rs file with the following. Be sure to set the DISCORD_TOKEN environment variable to your bot's token. You must also depend on tokio, twilight-cache-inmemory, twilight-gateway, twilight-http, and twilight-model in your Cargo.toml.

use std::{env, error::Error, sync::Arc};
use twilight_cache_inmemory::{DefaultInMemoryCache, ResourceType};
use twilight_gateway::{Event, EventTypeFlags, Intents, Shard, ShardId, StreamExt as _};
use twilight_http::Client as HttpClient;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Initialize the tracing subscriber.
    tracing_subscriber::fmt::init();

    let token = env::var("DISCORD_TOKEN")?;

    // Use intents to only receive guild message events.
    let mut shard = Shard::new(
        ShardId::ONE,
        token.clone(),
        Intents::GUILD_MESSAGES | Intents::MESSAGE_CONTENT,
    );

    // HTTP is separate from the gateway, so create a new client.
    let http = Arc::new(HttpClient::new(token));

    // Since we only care about new messages, make the cache only
    // cache new messages.
    let cache = DefaultInMemoryCache::builder()
        .resource_types(ResourceType::MESSAGE)
        .build();

    // Process each event as they come in.
    while let Some(item) = shard.next_event(EventTypeFlags::all()).await {
        let Ok(event) = item else {
            tracing::warn!(source = ?item.unwrap_err(), "error receiving event");

            continue;
        };

        // Update the cache with the event.
        cache.update(&event);

        tokio::spawn(handle_event(event, Arc::clone(&http)));
    }

    Ok(())
}

async fn handle_event(
    event: Event,
    http: Arc<HttpClient>,
) -> Result<(), Box<dyn Error + Send + Sync>> {
    match event {
        Event::MessageCreate(msg) if msg.content == "!ping" => {
            http.create_message(msg.channel_id)
                .content("Pong!")
                .await?;
        }
        // Other events here...
        _ => {}
    }

    Ok(())
}

License

All first-party crates are licensed under ISC

twilight's People

Contributors

7596ff avatar aenterprise avatar arzte avatar asianintel avatar baptiste0928 avatar blackholefox avatar cldfire avatar coadler avatar dependabot[bot] avatar dusterthefirst avatar erk- avatar gelbpunkt avatar hirunya avatar htgazurex1212 avatar itohatweb avatar james7132 avatar laralove143 avatar mathspy avatar maxohn avatar mikeplus64 avatar minndevelopment avatar nickelc avatar noxime avatar pyrotechniac avatar randomairborne avatar sam-kirby avatar suneettipirneni avatar tbnritzdoge avatar vilgotf avatar zeylahellyer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

twilight's Issues

[gateway] Add new gateway event

The reconnect event is dispatched when a client should reconnect to the gateway (and resume their existing session, if they have one). This event usually occurs during deploys to migrate sessions gracefully off old hosts.

relevant commit

[gateway] Implement Cluster events

The stream returned by Cluster::events and Cluster::some_events is currently unimplemented. We basically just need to merge all of the streams dynamically from all of the shards as new shard sessions are up.

[gateway] Make compression optional

This is the first thing we need to do before getting on #14 since discord explicitly says

When using ETF, the client must not send compressed messages to the server.

[model] member::guild_id can be non-optional

The model::guild::Member::guild_id structfield doesn't have to be optional.

Currently most types in the model crate are naive serde (de)serialization derives, and this type isn't any different. The gateway provides members with the guild_id, but the HTTP API doesn't. Since we already know the guild ID when using the HTTP API (we need it as a reference for where to get members from), we can use it as a serde::de::DeserializeSeed seed implementation for Member, avoiding the need for the structfield to be optional.

Right now, we get unfortunate real-world code like this:

use anyhow::Result;
use crate::State;
use dawn::model::gateway::payload::MemberAdd;

pub async fn handler(state: State, event: MemberAdd) -> Result<()> {
    // An unfortunate consequence of combining the HTTP and gateway models for
    // the Member struct -- we should change this.
    if let Some(guild_id) = event.guild_id {
        // work with the guild id here
    }

    Ok(())
}

Every other resume fails

Every other resume fails because the id fails to be set correctly after reconnecting to a new session.

ShardConfig::builder issues with queue configuration function

This could be a simple mistake on my part or something counter-intuitive but looking at the function I should be able to just provide something simple like Box::new(Queue::new()). Except this doesn't work.

A bare bones example is importing dawn-gateway and only dawn-gateway like so dawn-gateway = { git = "https://github.com/dawn-rs/dawn" } then providing the built-in LocalQueue from dawn as a reproductive example.

use dawn_gateway::{Shard, ShardConfig, queue::LocalQueue};

let token = env::var("DISCORD_TOKEN")?;
let config = ShardConfig::builder(&token);
config.queue(Box::new(LocalQueue::new()));

I would assume this would work and be how it's performed as per this line doing this exact thing, just outside the function https://github.com/dawn-rs/dawn/blob/cc35c8532171ab1e02f7a5cf664599e50465107f/gateway/src/shard/config.rs#L105

[http] Parse error messages

Parse the error messages that Discord provides into error structs. An example of an error struct is this one, sent when a message embed's field name's length is 0:

{
    "code": 50035,
    "errors": {
        "embed": {
            "fields": {
                "0": {
                    "name": {
                        "_errors": [
                            {
                                "code": "BASE_TYPE_REQUIRED",
                                "message": "This field is required"
                            }
                        ]
                    }
                }
            }
        }
    },
    "message": "Invalid Form Body"
}

[gateway] Request from queue on connects

When attempting to start a connection, a Shard needs to go through the queue already given to it, to ensure that only one shard is attempting to connect in a given 5 second period.

Naming consistency across Config structs

ShardConfig builds into a Config, but then you start with a Config in the Command module. I would suggest explicit XConfig and XConfigBuilder names across the board, but being consistent in scheme is a minimum I would ask for in this issue.

[command-parser] Command parser is not deterministic and may chose wrong commands.

So at the moment the code below will sometimes fail and other times pass, it depends on the order
of the elements in the HashSet Commands and as that order is not deterministic it will sometime fail.

There are a couple of ways we could resolve this:

  • Force a separator after commands
  • Use a ordered set like a vector instead, but this will not solve that some commands would shadow others.
fn double_config() -> Parser<'static> {
    let mut config = Config::new();
    config.add_prefix("!");
    config.add_command("echo");
    config.add_command("echolocate");
    
    Parser::new(config)
}

#[test]
fn test_double_config() {
    let parser = double_config();
    
    match parser.parse("!echolocate") {
        Some(Command { name: "echo", .. }) => {
            panic!("Wrong command")
        },
        Some(Command { name: "echolocate", .. }) => {
        },
        // Ignore all other commands.
        Some(_) => panic!("This should not happen (1)"),
        None => panic!("This should not happen (2)"),
    }
}

[project] add 'cache/in-memory'

Add the cache/in-memory crate, named dawn-cache-inmemory. This is an in-memory, (mostly) immutable cache with no public locking.

[gateway] Fix reconnection logic

On reconnects we're creating a new session but aren't propagating it to the heartbeater, so a heartbeat doesn't seem to be sent out, which then causes a close shortly after.

[gateway] Opt-in to receive raw payloads

In the gateway, add a variant of the Event enum which contains the raw bytes of the payload. By default, this should be disabled and opt-in through the existing Cluster::some_events and Shard::some_events methods. This isn't costly because we don't consume the bytes when deserializing into a GatewayEvent, so no cloning is needed here.

[http] Expose raw client requests

Expose the functionality in the HTTP client to perform raw requests, still backed by the ratelimiter but avoiding convenience methods (such as Client::create_message).

[http] add 'reason' to all moderation requests

Add the 'X-Audit-Log-Reason` header to all moderation requests that can be audited.

This is already in the CreateBan request, but as a query parameter: https://github.com/dawn-rs/dawn/blob/ddfe693cb112ed8ceebb83af17ae02067bc6ab17/http/src/request/create_ban.rs#L35

We'll probably need to add better handling of UTF-8. A description of reasons is at the top of this page: https://discordapp.com/developers/docs/resources/audit-log#audit-logs

What this mostly means is manually creating the Request in the start method of requests and specifying the ratelimit header in the header map: https://github.com/dawn-rs/dawn/blob/ddfe693cb112ed8ceebb83af17ae02067bc6ab17/http/src/request/create_ban.rs#L42

[http] Validate request parameters

Validate that request parameters and body values are within the accepted ranges. For example, getting some members from a guild can only get at most 1000 at a time. This will help prevent as many 400s as we can.

[http] Add Path parsing

Add parsing for http::routing::Path to parse from URLs or URI paths. For example, the string channels/123/messages should parse to the path Path::ChannelsIdMessages(123).

[gateway] Request gateway URI via HTTP

On every shard connection, request the gateway URI via the given HTTP client in the config rather than hardcoding it, which is discouraged behaviour.

[model] Add custom statuses

Add support for activity type 4, custom statuses.

The payload looks like this:

{
    "t": "PRESENCE_UPDATE",
    "s": 23,
    "op": 0,
    "d": {
        "user": {
            "id": "114941315417899012"
        },
        "status": "dnd",
        "roles": [
            "620984892410429470",
            "635845304947703808"
            "621037383416283137",
            "621124153361170452",
            "621273850138591262"
        ],
        "guild_id": "620980184606048276",
        "game": {
            "type": 4,
            "state": "Rainy",
            "name": "Custom Status",
            "id": "custom",
            "emoji": {
                "name": "๐ŸŒง"
            },
            "created_at": 1572196553812
        },
        "client_status": {
            "desktop": "dnd"
        },
        "activities": [
            {
                "type": 4,
                "state": "Rainy",
                "name": "Custom Status",
                "id": "custom",
                "emoji": {
                    "name": "๐ŸŒง"
                },
                "created_at": 1572196553812
            }
        ]
    }
}

It looks like we just need to add the activity.emoji key - which has the same payload as channel::ReactionType::Custom - and add a test for it like with other models.

See discord/discord-api-docs#1162

[meta] Standardize logging scheme

Standardize logging to use a single structure for messages. Right now it's stuff like "Error connecting: {:?}", "[SocketForwarder] Starting driving loop", and with #36 "[ShardProcessor {:?}] message" with {:?} being the shard ID/total.

I think the last one is the best.

[project] add 'cache/base'

Add the cache/base library, named dawn-cache. This is a library exporting the types from dawn-cache-trait (Cache and CacheUpdate), as well as exporting the dawn-cache-inmemory library.

Blocks on #5.

[gateway, http] Add serde-ignored feature

Add an optional opt-in feature for the gateway and http crates to use serde_ignored to deserialize payloads and log to INFO if there are any keys we didn't catch.

This should also log unknown keys directly in the custom deserializers where applicable.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.