This consumes the Twitter streaming API, processing and feeding the Redis instance for the rest of Emojitracker.

1. Make sure you have Ruby 2.1.x installed.

2. Get the repository and basic dependencies going:

   git clone mroth/emojitrack-feeder
   cd emojitrack-feeder
   bundle install --without=production
   

3. Copy .env-sample to .env and configure required variables.

4. Make sure you have Redis installed and running. The rules in lib/config.rb currently dictate the order a redis server instance is looked for.

5. Run all processes via foreman start or forego start (depending on which you have installed).

Be sure to note that while the processing power is fairly managable, the feeder component of emojitrack requires on it's own about 1MB/s of downstream bandwith, and ~250KB/s of upstream. You can use the MAX_TERMS environment variable to process less emoji chars if you don't have the bandwidth where you are.

Note, DO NOT run the feeder process with REDIS_URL configured to the production server, ever.

This is but a small part of emojitracker's infrastructure. Major components of the project include:

• emojitrack-web the web frontend and application server
• emojitrack-feeder consumes the Twitter Streaming API and feeds our data pipeline
• emojitrack-streamer handles streaming updates to clients via SSE
  • ruby version (deprecated)
  • nodejs version(deprecated)
  • go version (production)
  • streamer API spec defines the streamer spec & tests servers in staging

Additionally, many of the libraries emojitrack uses have also been carved out into independent emoji-related open-source projects, see the following:

• emoji_data.rb utility library for handling the Emoji vs Unicode nightmare (Ruby)
• emoji-data-js utility library for handling the Emoji vs Unicode nightmare (Nodejs port)
• exmoji utility library for handling the Emoji vs Unicode nightmare (Elixir/Erlang port)
• emojistatic generates static emoji assets for a public CDN

As well as some general purpose libraries:

• cssquirt Embeds images (or directories of images) directly into CSS via the Data URI scheme
• sse-bench benchmarks Server-Sent Events endpoints