Slack, Discord, and WhatsApp have proven that responsive, real-time chat is an essential communication technology. This challenge is an opportunity to build your own real-time chat platform. Watch our short intro video on YouTube. We also made a 30 min challenge overview video that you can also watch on YouTube.

A chat application is both extremely satisfying to build and easy to explain to the judges. There's plenty of features you can build to express your creativity. And the skills you'll learn in building a real-time app will help you throughout your software engineering career.

You can use any server technology you like, but for the back-end database, we'd like you to use Redis. Yes, Redis is often used as an ephemeral caching layer, but Redis is also a real-time database with persistence, strong consistency (soon), an extensive array of data structures, and full-text search.

Redis is also the most-loved database. It's true!

What's nice about this challenge is we're going to give you a lot of help in building the data layer. There are several interesting data modeling questions, including:

• How do you store chat messages?
• How do you represent chat rooms and users?
• How do you publish real-time updates? H* ow do you search for messages?

We explain all of this in the Redis data modeling guide below. We also provide some suggestions on how to implement more complex features.

Although you can use any server stack you like, we'll assume in the guide that you're going to build a Node.js API layer that talks to Redis, and a front end using your preferred front-end framework.

Some use cases, like notifications, will require data from the back end to be pushed to the front end. For this, consider websockets or server sent events.

We've created a channel (#remotebase-hackathon) on the official Redis Discord just for this hackathon, and we'll be around to answer your questions!

You should also check out some of the docs and videos below:

Docs:

• redis.io: Redis documentation, including a complete command reference.
• ioredis: a Redis client for Node.js
• Node.js Redis crash course on Redis Developer
• Introduction to Redis Streams
• Official RediSearch Tutorial

Videos:

• ioredis Up and Running video
• Redis Sets Explained video
• Redis Hashes Explained video
• Redis Streams Explained video
• Querying, Indexing and Full Text Search in Redis video
• ioredis Argument and Reply Transformers video (example of working with Redis Streams using ioredis and Node.js)

Tools:

• Node.js (we recommend the latest LTS version where possible)
• redis-cli documentation
• RedisInsight: a graphical tool for managing and editing data in a Redis database.
• You may need to install Docker Desktop if you choose to run Redis in Docker (we also provide a cloud option that doesn't require Docker)
• Your favorite IDE (we like VSCode but anything you are comfortable and can be productive with will be perfect for this challenge)

To take part in this challenge, you'll need a Redis instance with the RediSearch module installed. We've provided two ways to get up and running… Docker, and Redis Enterprise Cloud.

We've made a repository available for this challenge on GitHub. To start Redis in a Docker container, you'll need to install Docker Desktop then clone the repository and start Redis using the Docker compose file provided:

$ git clone https://github.com/redis-developer/redis-chat-challenge.git
$ cd redis-chat-challenge
$ docker-compose up -d

This starts a Redis server instance on port 6379. You can connect to it using redis-cli as follows:

$ docker exec -it redischatapp redis-cli
127.0.0.1:6379>

You may also want to install RedisInsight instead of using redis-cli for administration. Once installed, configure it to find Redis at localhost port 6379 with no password.

When you're done using Redis, stop the container:

$ cd redis-chat-challenge
$ docker-compose down

Redis persists your data to a folder named "redisdata" inside the "redis-chat-challenge" folder. You can stop and restart the container without losing data.

Redis provides a free 30Mb Redis Instance as part of its Redis Enterprise Cloud service. These databases can also be configured to include the RediSearch module, which is required to complete this challenge. To sign up for Redis Enterprise Cloud and create a database, follow the instructions here.

Note: Be sure to select the RediSearch module when choosing which module to install.

Once you have a database hostname, port and password, note them down somewhere safe. You'll need them later when setting up RedisInsight and configuring your Node.js application.

Next, follow these instructions to set up RedisInsight and connect to your database using the hostname, port and password.

Now you're up and running with a Redis instance, it's time to load some sample data and create the search indices that RediSearch needs.

If you haven't already cloned the GitHub repository for this challenge, do so now:

$ git clone https://github.com/redis-developer/redis-chat-challenge.git
$ cd redis-chat-challenge

We've provided a sample data loader, use it to populate your Redis instance with a small dataset:

$ cd data_loader
$ npm install

If you're using Redis Enterprise Cloud, set your cloud database credentials using environment variables. If using Docker, you can omit this step.

$ export REDIS_HOST=<hostname for Redis Enterprise>
$ export REDIS_PORT=<port number for Redis Enterprise>
$ export REDIS_PASSWORD=<password for Redis Enterprise>

Note: these three values are secrets, don't include them directly in your code, or commit them to source control!

Now, run the data loader tool:

$ npm start

You should expect to see the following output:

$ npm start

> [email protected] start
> node dataloader.js

Loading user wasim.
Loading user simon.
Loading user antirez.
Loading user yaz.
Loading user woz.
Loading user ada.
User verification OK.
Posting message by wasim to channel cricket.
Posting message by ada to channel tech.
Posting message by wasim to channel tech.
Posting message by woz to channel random.
Posting message by yaz to channel cricket.
Posting message by simon to channel cricket.
Message verification successful.
Search verification successful.

Note: the data loader first erases all data from Redis, so if you run it a second time you'll lose any additional data that you might have added.

Use RedisInsight or redis-cli to browse the contents of your Redis instance. When using redis-cli you will need to use the KEYS or SCAN commands to discover key names. Use the browser view on RedisInsight to see all of the keys.

To use ioredis in your own Node.js project, install it:

$ npm install ioredis

added 13 packages, and audited 14 packages in 710ms

1 package is looking for funding
  run `npm fund` for details

found 0 vulnerabilities

If you're comfortable reading code, take a look at demo.js which contains an example of how to create a new user and post a new chat message to a channel. To run this code:

$ npm install
$ npm start

Connect to a Redis instance by creating a new Redis client, specifying the host, port and password to connect to:

const Redis = require('ioredis');

const redisClient = new Redis({
  host: process.env.REDIS_HOST || 'localhost',
  port: process.env.REDIS_PORT || 6379,
  password: process.env.REDIS_PASSWORD
});

For Redis Enterprise Cloud, these values were generated when you created your database. For Docker, the default values localhost / 6379 / no password are what you need.

Use your Redis client instance to send commands to Redis. ioredis exposes a function named for each Redis command. These functions generally take the name of a Redis key as the first argument, followed by other parameters that the command requires - passed as strings.

For example, the HMGET command gets multiple, named fields from a Redis Hash. To get the firstname and city fields for a user, we'd use it like this:

const values = await redisClient.hmget('user:ada', 'firstname', 'city');

values then contains an array of the field values, in the order that they were requested:

[ 'Ada', 'Ravenshead' ]

The HSET command for setting name/value pairs in a Redis hash can take an object as its argument:

const user = {
  username: 'demo',
  firstname: 'Demo',
  lastname: 'User',
  city: 'Demoville',
  country: 'Demoland'
};

const response = await redisClient.hset('user:demo', user);

which creates a hash in Redis:

127.0.0.1:6379> hgetall user:demo
 1) "username"
 2) "demo"
 3) "firstname"
 4) "Demo"
 5) "lastname"
 6) "User"
 7) "city"
 8) "Demoville"
 9) "country"
10) "Demoland"

For this challenge, you'll also need to work with commands for the RediSearch module. RediSearch is an optional module for Redis, so its commands aren't built into ioredis as functions. To call a RediSearch command, use the ioredis "call" function, which sends an arbitrary command to Redis for execution.

For example if we wanted to run this command to find users with first name "Simon" who are in Nottingham:

FT.SEARCH idx:users "@firstname:Simon @city:{Nottingham}"

Our code would look like this:

const response = await redisClient.call('FT.SEARCH', 'idx:users', '@firstname:Simon @city:{Nottingham}', 'LIMIT', '0', '999999');

You should implement some or all of these use cases in your solution.

Check if a specific username is available. For example, let's check if user "wasim" is available:

Usernames are kept in a Redis set whose key is "usernames".

127.0.0.1:6379> sismember usernames wasim
(integer) 0

• 0 = yes it is (username isn't currently in the set at key usernames)
• 1 = no it isn't (username is in the set at key usernames)

Create a new user as a Redis hash for the user's details. We'll give these hashes keys using the pattern "user:":

127.0.0.1:6379> hset user:wasim firstname Wasim lastname Akram city Lahore country Pakistan username wasim
(integer) 5

User schema:

Add the new user's username to the Redis set used to track all usernames:

127.0.0.1:6379> sadd usernames wasim
(integer) 1

• 1 = number of new members added to the set

Let's get the user profile for the user with username "wasim":

127.0.0.1:6379> hgetall user:wasim
 1) "firstname"
 2) "Wasim"
 3) "lastname"
 4) "Akhram"
 5) "city"
 6) "Lahore"
 7) "country"
 8) "Pakistan"
 9) "username"
10) "wasim"

Channel names are kept in a Redis set whose key name is "channels".

127.0.0.1:6379> smembers channels
1) "general"
2) "cricket"
3) "random"
4) "tech"
5) "dance"
6) "cycling"

Check if a specific channel already exists e.g. let's check if channel "cricket" is available:

127.0.0.1:6379> sismember channels cricket
(integer) 1
127.0.0.1:6379> sismember channels sewing
(integer) 0

• 0 = channel doesn't exist, go ahead and make it
• 1 = channel exists already, nothing else to do

Create a new channel if it didn't already exist:

To create a new channel, just add its name to the set of channels, whose key is "channels". Redis will create this set for you if it doesn't already exist:

127.0.0.1:6379> sadd channels sewing
(integer) 1

• 1 = number of new members added to set

Create a new entry in the channel's stream in Redis, and get back a message ID. Use the Redis XADD command for this. If this is the first message, Redis will create the stream for you. We'll use the pattern "channel:<channel name>" for the stream key and set a single name/value pair as the payload for each entry:

127.0.0.1:6379> xadd channel:cricket * type message
"1631566566616-0"

Here, "*" tells Redis to generate a new message ID for you. This takes the form of a timestamp, see the XADD documentation for more information.

Then, create a new message hash in Redis for the message using the HSET command. We'll store the message at the key "message:<message ID>":

127.0.0.1:6379> hset message:1631566566616-0 channel cricket username wasim message "Did you see the game today?"
(integer) 3

Message schema:

Get the last 2 message IDs in the "cricket" channel:

The XREVRANGE command retrieves entries from a stream:

127.0.0.1:6379> xrevrange channel:cricket + - count 2
1) 1) "1631567261312-0"
   2) 1) "type"
      2) "message"
2) 1) "1631567227135-0"
   2) 1) "type"
      2) "message"

Note: here, the most recent messages are returned first.

Get all message IDs in the "cricket" channel:

We can use the XRANGE command for this:

127.0.0.1:6379> xrange channel:cricket - +
1) 1) "1631566566616-0"
   2) 1) "type"
      2) "message"
2) 1) "1631567199836-0"
   2) 1) "type"
      2) "message"
3) 1) "1631567227135-0"
   2) 1) "type"
      2) "message"
4) 1) "1631567261312-0"
   2) 1) "type"
      2) "message"

Note: here, the most recent messages are returned last.

Once you have the message IDs, get the message details for each message by retrieving the associated Redis hash from the key "message:" using the HGETALL command:

127.0.0.1:6379> hgetall message:1631567261312-0
1) "channel"
2) "cricket"
3) "username"
4) "simon"
5) "message"
6) "A couple of nice catches too!"

Show the user "new" messages (i.e., those posted since the user last looked at the channel) by storing the highest message ID that the user has seen in a given channel in their profile hash. For this, you'll need the HSET command:

For example, let's remember that user "wasim" has seen messages up to and including "1631567261312-0" in the "cricket" channel:

127.0.0.1:6379> hset user:wasim channel:cricket 1631567261312-0
(integer) 1

Get new messages for user "wasim" in the "cricket" channel.

First, get the last message ID that the user saw for this channel from their profile hash in Redis:

We can use the HGET command to retrieve selected fields from a Redis hash:

127.0.0.1:6379> hget user:wasim channel:cricket
"1631567261312-0"

If there is no last message ID for this channel in the user's profile hash, the above will return "nil":

127.0.0.1:6379> hget users:wasim channel:cricket
(nil)

In this case, use the ID 0 to start at the first message in the channel's stream.

Next, read messages in that channel since the last message ID (or 0), using the XREAD command:

127.0.0.1:6379> xread streams channel:cricket 1631567261312-0
1) 1) "channel:cricket"
   2) 1) 1) "1631568209450-0"
         2) 1) "type"
            2) "message"
      2) 1) "1631568216171-0"
         2) 1) "type"
            2) "message"

Then, get the Redis hash for each message ID returned using HGETALL:

127.0.0.1:6379> hgetall message:1631568216171-0
1) "channel"
2) "cricket"
3) "username"
4) "simon"
5) "message"
6) "I am looking forward to the Test match."

And update the user's hash to store a new latest message ID, using the highest message ID returned. Use the HSET command:

127.0.0.1:6379> hset user:wasim channel:cricket 1631568216171-0
(integer) 0

To achieve this, we'll need to use the blocking capabilities of the Redis XREAD command. For example, let's block for up to 5 seconds (5000 milliseconds). XREAD returns nil if no new message is added after 5 seconds, or returns earlier with the message data if one is.

For a specific channel (when no new messages appear during the blocking period):

127.0.0.1:6379> xread block 5000 streams channel:cricket $
1) 1) "channel:cricket"
   2) 1) 1) "1631568939209-0"
         2) 1) "type"
            2) "message"
(4.05s)

For several channels at once (when no new messages appear during the blocking period):

127.0.0.1:6379> xread block 5000 streams channel:cricket channel:dance channel:tech $ $ $
(nil)

For several channels at once (when messages appear during the blocking period):

127.0.0.1:6379> xread block 5000 streams channel:cricket channel:tech channel:dance $ $ $
1) 1) "channel:tech"
   2) 1) 1) "1631569235788-0"
         2) 1) "type"
            2) "message"
(3.76s)

Note: Redis blocking commands (such as XREAD with the block option) will block the ioredis client while they are running… no other Redis commands will be executed during this time. You should create and use a second instance of the ioredis client when calling a blocking command.

Here, we'll need to use the RediSearch FT.AGGREGATE command (see docs) to get a de-duplicated list of users that have posted in a given channel. For example, let's find everyone who has posted in tech:

127.0.0.1:6379> ft.aggregate idx:messages "@channel:{tech}" groupby 1 @username
1) (integer) 2
2) 1) "username"
   2) "ada"
3) 1) "username"
   2) "wasim"

Here we're using RediSearch to index and query the data stored in the message hashes.

We've provided a message search schema called idx:messages with the following definition:

Check out the RediSearch query syntax documentation for more details.

Find messages containing the word "match":

127.0.0.1:6379> ft.search idx:messages "match"
1) (integer) 1
2) "message:1631623305083-0"
3) 1) "username"
   2) "wasim"
   3) "channel"
   4) "cricket"
   5) "message"
   6) "Did you see the test match this weekend?"

Find messages in the "cricket" channel posted by user "simon":

127.0.0.1:6379> ft.search idx:messages "@username:{simon} @channel:{cricket}"
1) (integer) 1
2) "message:1631623342744-0"
3) 1) "username"
   2) "simon"
   3) "channel"
   4) "cricket"
   5) "message"
   6) "Fantastic catch to win the game too."

Find messages containing the word "sunny" across all channels:

127.0.0.1:6379> ft.search idx:messages "@message:sunny"
1) (integer) 1
2) "message:1631623332491-0"
3) 1) "username"
   2) "woz"
   3) "message"
   4) "Warm and sunny here in California today, what's it like where you are?"
   5) "channel"
   6) "random"

Find messages posted by "wasim" in the "tech" or "cricket" channels:

​​127.0.0.1:6379> ft.search idx:messages "@channel:{cricket|tech} @username:{wasim}" limit 0 999999
1) (integer) 2
2) "message:1631623324707-0"
3) 1) "username"
   2) "wasim"
   3) "channel"
   4) "tech"
   5) "message"
   6) "Me too,"
4) "message:1631623305083-0"
5) 1) "username"
   2) "wasim"
   3) "channel"
   4) "cricket"
   5) "message"
   6) "Did you see the test match this weekend?"

Perform the above query with ioredis:

const searchResults = await redisClient.call('FT.SEARCH', 'idx:messages', '@channel:{cricket|tech} @username:{wasim}', 'LIMIT', '0', '999999');

searchResults contains:

[
  2,
  'message:1631623324707-0',
  [ 
    'username', 
    'wasim', 
    'channel', 
    'tech', 
    'message', 
    'Me too,' 
  ],
  'message:1631623305083-0',
  [
    'username',
    'wasim',
    'channel',
    'cricket',
    'message',
    'Did you see the test match this weekend?'
  ]
]

Here we're using RediSearch to index and query the data stored in the user profile hashes.

We've provided a user profile search schema called "idx:users" with the following definition:

Let's find users in Lahore (note that this is a case insensitive match):

127.0.0.1:6379> ft.search idx:users "@city:{lahore}"
1) (integer) 1
2) "user:Wasim"
3)  1) "firstname"
    2) "Wasim"
    3) "lastname"
    4) "Akram"
    5) "city"
    6) "Lahore"
    7) "country"
    8) "Pakistan"
    9) "username"
   10) "wasim"

Or users in England:

127.0.0.1:6379> ft.search idx:users "@country:{England}"
1) (integer) 2
2) "user:ada"
3)  1) "firstname"
    2) "Ada"
    3) "lastname"
    4) "Lovelace"
    5) "city"
    6) "Ravenshead"
    7) "country"
    8) "England"
    9) "username"
   10) "ada"
4) "user:simon"
5)  1) "firstname"
    2) "Simon"
    3) "lastname"
    4) "Prickett"
    5) "city"
    6) "Nottingham"
    7) "country"
    8) "England"
    9) "username"
   10) "simon"

Note: By default, RediSearch returns only the first 10 matches. To request more than this, use the LIMIT keyword which takes a start and end offset (see docs):

127.0.0.1:6379> ft.search idx:users "@country:{England}" limit 0 999999

Find all users whose first name is either Simon or Wasim:

127.0.0.1:6379> ft.search idx:users "@firstname:Simon|@firstname:Wasim"
1) (integer) 2
2) "user:simon"
3)  1) "firstname"
    2) "Simon"
    3) "lastname"
    4) "Prickett"
    5) "city"
    6) "Nottingham"
    7) "country"
    8) "England"
    9) "username"
   10) "simon"
4) "user:Wasim"
5)  1) "firstname"
    2) "Wasim"
    3) "lastname"
    4) "Akram"
    5) "city"
    6) "Lahore"
    7) "country"
    8) "Pakistan"
    9) "username"
   10) "wasim"

Check out the RediSearch query syntax documentation for more details.

Here are some additional ideas that you might consider implementing if you have time:

• Add a new type of event in the channel stream for a "join" event when a user first requests messages for a channel. Have these events show up in the channel as messages that say "<username> joined". You'll need to store these as a different event type in the channel's stream, for example:

127.0.0.1:6379> xadd channel:cricket * type join username wasim
"1631573099060-0"

• Add a new type of event in the channel stream for a "leave" event and provide a way for users to leave channels. Have these events show up in the channel as messages that say "<username> left". Example:

127.0.0.1:6379> xadd channel:cricket * type leave username wasim
"1631614788324-0"

• If you choose to implement "join" and "leave" events, consider using Redis Sorted Sets to store each user's last active time for each channel. Using this you could build a "recently active users" feature. Check out the Redis Sorted Set documentation on redis.io for help.

• Extend the user profile with additional fields, for example:

  • Store a user's hobbies. Extend the user profile search to find users by hobby.
  • Store a user's geographic (long/lat) location. Extend the user profile search to find users nearby a given long/lat position.

• Implement an @ mention system for usernames, choosing appropriate Redis data structure(s) to track mentions by username.

• Create a view in your front end that shows messages received in the last 5 minutes for one or more of the channels.

• Use RediSearch to find which users have posted the most messages across all channels.