Code Monkey home page Code Monkey logo

winds's Introduction

๐Ÿ›‘ Notice: This repository is no longer maintained; No further Issues or Pull Requests will be considered or approved.

Winds - A Beautiful Open Source RSS & Podcast App Powered by GetStream.io

Slack Community Build Status codecov Open Source Maintenance Built With StackShare

Description

Winds is a beautiful open-source RSS and Podcast app created using React & Redux on the frontend and Express.js on the backend. Use the free hosted version or run it on your own server and customize it as you see fit. Contributions in form of pull requests are always appreciated. Activity Feeds & Discovery in Winds are powered by Stream, the app leverages Algolia for search, AWS for hosting, MongoDB Atlas for a hosted database (DBaaS), and SendGrid for email. All of these services have a free tier.

Getting Started

To get started with Winds, please download the latest release

Featured RSS & Podcasts

Have a popular RSS or Podcast and want to be featured? Reach out to [email protected]. We reply to every message.

Features at a Glance

Winds is packed full of awesome features behind a beautiful user interface and user experience. Here's a quick look at what the project has to offer:

Beautiful UI

Winds UI

RSS & Podcast Recommendations

Winds RSS & Podcast Recommendations

Integrated Search

Winds Search

Podcast Player

Winds Podcast Player

RSS Reader

Winds RSS Reader

TOCd

Roadmap

Help us improve Winds and/or vote on the Roadmap for 2.1

  • Search detail screen
  • Playlist support (partially implemented)
  • Team support (share an activity feed with colleagues or friends to discover and collaborate)
  • Mobile application powered by React Native

Powered By

  1. Express
  2. React & Redux
  3. Algolia
  4. MongoDB Atlas
  5. SendGrid
  6. Bull
  7. Mercury
  8. Stream
  9. Sentry
  10. AWS

The full stack can be found on StackShare.io.

Stream

Stream is an API for building activity feeds. For Winds the follow suggestions and the list of articles from the feeds you follow is powered by Stream. Stream accounts are free for up to 3 million feed updates and handle personalization (machine learning) for up to 100 users.

Algolia

Algolia is used for lightning fast and relevant search. We use their JavaScript search client to easily setup the Winds search implementation. Algolia, accounts are free up to 10k records and 100k operations.

MongoDB

MongoDB Atlas provides a Database as a Service, and serves as the backend datastore for Winds.

Tutorials & Blog Posts

The following tutorials will not only help you start contributing to Winds, but also provide inspiration for your next app.

Note: We're actively working on this portion of the README. To stay up to date with the latest information, please signup for the hosted version at https://getstream.io/winds.

  1. Implementing search with Algolia
  2. Stream and Winds
  3. Running PM2 & Node.js in Production Environments
  4. Creating a RESTful API design with Express.js
  5. Takeaways on Building a React Based App with Electron
  6. The Winds Stack
  7. Building Touch Bar Support for macOS in Electron with React
  8. Testing Node.js in 2018
  9. Simple Steps to Optimize Your App Performance with MongoDB, Redis, and Node.js
  10. Getting Started with Winds & Open Source
  11. Deploying the Winds App to Amazon S3 and CloudFront
  12. Deploying the Winds API to AWS ECS with Docker Compose

Download

To download Winds, visit https://getstream.io/winds/.

Contributing to Winds

TL;DR

Commands:

  • brew install redis mongodb
  • brew services start mongodb
  • brew services start redis
  • cd Winds
  • cd api && yarn
  • cd ../app && yarn

Sign up for both Stream and Algolia, and create the following .env file in the app directory, replacing the keys where indicated:

DATABASE_URI=mongodb://localhost/WINDS_DEV
CACHE_URI=redis://localhost:6379
JWT_SECRET=YOUR_JWT_SECRET

API_PORT=8080
REACT_APP_API_ENDPOINT=http://localhost:8080
STREAM_API_BASE_URL=https://windspersonalization.getstream.io/personalization/v1.0

STREAM_APP_ID=YOUR_STREAM_APP_ID
REACT_APP_STREAM_APP_ID=YOUR_STREAM_APP_ID
REACT_APP_STREAM_API_KEY=YOUR_STREAM_API_KEY
REACT_APP_STREAM_ANALYTICS=YOUR_STREAM_ANALYTICS_TOKEN
STREAM_API_KEY=YOUR_STREAM_API_KEY
STREAM_API_SECRET=YOUR_STREAM_API_SECRET

REACT_APP_ALGOLIA_APP_ID=YOUR_ALGOLIA_APP_ID
REACT_APP_ALGOLIA_SEARCH_KEY=YOUR_ALGOLIA_SEARCH_ONLY_API_KEY
ALGOLIA_WRITE_KEY=YOUR_ALGOLIA_ADMIN_API_KEY

Note: If you are running the test suite, you will need to have a test version of the .env file inside of the api/test directory.

Then run:

  • pm2 start process_dev.json
  • cd app && yarn start

Clone the Repo

git clone [email protected]:GetStream/Winds.git

Install dependencies

The following instructions are geared towards Mac users who can use brew (Homebrew) to install most dependencies. Ubuntu users can use apt, and Windows users will have to install directly from the dependency's site. Non-debian-based Linux users will probably be able to figure it out on their own :)

  • cd Winds/app
  • yarn
  • cd ../api
  • yarn

Start MongoDB Locally

Winds uses MongoDB as the main datastore - it contains all users, rss feeds, podcasts, episodes, articles, and shares.

If you're on a Mac, you can install MongoDB through Homebrew by running:

brew install mongodb

(You can also install MongoDB from the official MongoDB site.)

You can also run MongoDB in the background by running:

brew services start mongodb

Start Redis Locally

At Stream, we use Redis as an in-memory storage for the Winds podcast processing and RSS processing workers. It contains a list of podcasts and RSS feeds, which the workers pick up and process using the bull messaging system.

If you're on a Mac, you can install Redis through Homebrew by running:

brew install redis

(You can also install Redis from the official Redis site.)

Then, start Redis by running:

redis-server

...which creates (by default) a dump.rdb file in your current directory and stores a cached version of the database in that location.

You can also run Redis in the background by running:

brew services start redis

Loading Test Data

For testing purposes, you will want to use the test data located here.

Use mongoimport or mongorestore to import the data. There are two username and password combinations for testing:

Username: [email protected]
Password: admin

Username: [email protected]
Password: test

You will need to run the FLUSHALL command in Redis to ensure that the new content is picked up.

Note: This will override any local data that you may have. Please be cautious! Also, this will not create Stream follows โ€“ please follow feeds manually to generate them.

Stream

Sign up and Create a Stream App

To contribute to Winds, sign up for Stream to utilize the activity and timeline feeds.

(Reminder: Stream is free for applications with less than 3,000,000 updates per month.)

Add your Stream App ID, API Key, and API Secret to your .env

Append the Stream App ID, API Key, and API secret to your .env file:

STREAM_APP_ID=YOUR_STREAM_APP_ID
STREAM_API_KEY=YOUR_STREAM_API_KEY
STREAM_API_SECRET=YOUR_STREAM_API_SECRET

Create Your Stream Feed Groups

Once you've signed in, create "feed groups" for your Stream app.

A "feed group" defines a certain type of feed within your application. Use the "Add Feed Group" button to create the following feeds:

Feed Group Name Feed Group Type
podcast flat
rss flat
user flat
timeline flat
folder flat
user_episode flat
user_article flat

It's fine to enable "Realtime Notifications" for each of these feed groups, though we won't be using those yet.

Algolia

Sign up for Algolia and Create an Algolia App and Index

In addition to Stream, you also need to sign up for Algolia, to contribute to Winds, for the search functionality.

(Algolia is free for applications with up to 10k records.)

  • Sign up for Algolia here
  • From the Applications page, click "New Application" and create a new Algolia application. (We recommend something similar to my-winds-app)
    • (Select the datacenter that's closest to you.)
  • From the application page, select "Indices" and click "Add New Index". (We recommend something similar to winds-main-index)

Add Your Algolia Application Id, Search-Only Api Key and Admin Api Key to Your .env File

From your app, click the "API Keys" button on the left to find your app ID and API keys.

Append your Algolia application ID, search-only API Key and Admin API Key to your .env file to look like this:

REACT_APP_ALGOLIA_APP_ID=YOUR_ALGOLIA_APP_ID
REACT_APP_ALGOLIA_SEARCH_KEY=YOUR_ALGOLIA_SEARCH_ONLY_API_KEY
ALGOLIA_WRITE_KEY=YOUR_ALGOLIA_ADMIN_API_KEY

Start Backend Services

From the root directory, run:

pm2 start process_dev.json

To see logging information for all processes, run:

pm2 logs

Start Frontend Electron / Web App Server

cd app && yarn start

Running tests

Winds API server uses:

Tests are located in api/test folder.

File structure is intentionally mirroring files in api/src to simplify matching tests to tested code.

To run tests:

cd api && yarn run test

To run tests with extended stack traces (useful when debugging async issues):

cd api && yarn run test_deep

Adding new tests

Add your code to a file in api/test folder (preferably mirroring existing file from api/src folder).

Refer to Mocha documentation for examples of using BDD-like DSL for writing tests.

Modules are mocked in api/test-entry.js as mocks have to be installed before any modules are loaded.

Fixtures are loaded via loadFixture function from api/test/fixtures folder

Various utility functions are provided in api/test/util.js (See other tests for examples of usage).

Building a Production Version

Build a production version of Winds by running from root directory:

./api/build.sh

This creates production-ready JS files in api/dist.

To run the production JS files:

pm2 start process_prod.json

OR

Prepare the build for Docker:

cd api && cd scripts && ./make-build.sh

Build the Docker container (API & all workers):

cd ../ && docker-compose up

The commands above will prepare and start the API (and all workers). The frontend will still need to be started manually.

Debugging RSS & Podcast Issues

Unfortunately there is no unified standard for RSS. Go to the api directory and run yarn link to make these commands available:

winds rss https://techcrunch.com/feed/

Podcasts:

winds podcast https://www.npr.org/rss/podcast.php\?id\=510289

Open Graph scraping:

winds og http://www.planetary.org/multimedia/planetary-radio/show/2018/0509-amy-mainzer-neowise.html

RSS Discovery:

winds discover mashable.com

Article parsing (w/ Mercury):

winds article https://alexiskold.net/2018/04/12/meet-12-startups-from-techstars-nyc-winter-2018-program/

Pull requests for improved RSS compatibility are much appreciated. Most of the parsing codebase is located in api/src/parsers/.

Support

All support is handled via GitHub Issues. If you're unfamiliar with creating an Issue on GitHub, please follow these instructions.

Maintenance and Contributions

Thank you to all of the maintainers and contributors who've helped Winds become what it is today and help it stay up and running every day. We couldn't do it without you!

Special Shoutouts To:

Primary Maintainers

Contributors

Revive RSS

RSS is an amazing open standard. It is probably the most pleasant way to stay up to date with the sites and podcasts you care about. Our reasons for contributing to Winds are explained in the blogpost Winds 2.0 It's time to Revive RSS. In this section we will list other open source and commercial projects that are having an impact on Reviving RSS:

We are hiring!

We've recently closed a $38 million Series B funding round and we keep actively growing. Our APIs are used by more than a billion end-users, and you'll have a chance to make a huge impact on the product within a team of the strongest engineers all over the world.

Check out our current openings and apply via Stream's website.

winds's People

Contributors

aalvrz avatar astrotars avatar cardoso avatar durairaj avatar dwightgunning avatar ferhatelmas avatar gitter-badger avatar hamidzr avatar imgbotapp avatar jcamilom avatar jeltef avatar jon-dearaujo avatar julianxhokaxhiu avatar k3ith-fowl3r avatar kenhoff avatar kzeni avatar mahboubii avatar mathias avatar meriadec avatar nekuromento avatar sauban avatar sifex avatar tbarbugli avatar tilton avatar tschellenbach avatar zhomart avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

winds's Issues

Bug: Imported feeds don't load

I imported an OPML file from Inoreader and it turns out that the subscriptions simply keep loading forever, never retrieving any items. Subscribing to that same existing feed again manually fixes the issue.

Performance: Scraping isn't keeping up

There are over 2000 feeds to keep in sync now, scraping isn't keeping up. We need to make it a bit more efficient.

Some ideas:

  • Prioritize popular feeds while scraping
  • Run on multiple cores
  • Reduce timeouts
  • Disable getMetaInformation for less popular feeds
  • Use a task queuing system to distribute the work

Bug: Duplicate feeds (HTTP vs HTTPS)

When a URL is provided, it can sometimes cause a duplicate entry if the protocol is http or https. This is because they are essentially different URLs.

Fix: Remove protocol before submission and run through normalize.js

Idea: change PM2 PORT environment from 80 to 3000

This is an outcome of the yesterday outcome of when I was working on the Dockerfile.

As you may see the Dockerfile, we are currently exposing the 3000 port, because the application is running within his own user. Since this will affect also any other user that would like to self-host this application, probably on Linux or Mac environments would not be possible, unless you change the port in the configuration file.

What I suggest, since the pm2 process usually is also being run ( or mostly being run ) by non-root users, is to change the port to something over 1024, in this case I've chosen 3000. Whatever port it will be, please align the change in the EXPOSE rule of the Dockerfile and remove this hacky line.

Thank you in advance,
Julian

Idea: Docker support

What about an official Docker for this awesome project?

Thank you in advance :)

Idea: Different views of articles

The primary feature stopping me from using Winds now is the ability to read the content of the articles.

Currently we can switch between list (just the title) and card (image, title, and brief summary) views. Personally, I prefer to read as much content as the feed provides, i.e., the full description, if available, otherwise the summary. I'll only open the link to the original source if the content included is incomplete and it is interesting enough.

There are several possibilities that are quite common among readers:

  • List : just the title
  • Card : grid layout of title, image, and brief summary
  • Magazine : list layout of title, image, and brief summary
  • Full : title, image, and the full description, or the summary if the description isn't available

Having the full content of an article in the app might disrupt the 'click' engagement personalization metric, but perhaps that could be countered by introducing other metrics, e.g., time spent on an article while active. And tagging.

relies on proprietary APIs

some users aren't using stream and others aren't using sendgrid for sending emails, it would be nice if replacements for those were possible.

Idea: add feeds without protocol

When manually entering the URL for a site it is easy to forget the 'http://'. If the protocol could be added by the feed checker that would be nice.
Right now it says "Please enter a URL" when trying to do so, which is potentially confusing for less advanced users.

Bug: Broken feeds

Here's an overview of some feeds that currently don't parse.

http://tjomlid.com/feed/
https://theintercept.com/feed/?lang=en
https://www.javascript.com/feed/rss
http://blog.squarespace.com/blog?format=rss
http://blog.nodejs.org/feed/
http://blog.tmorris.net/atom.xml
http://www.digitaling.com/rss
http://feeds.feedburner.com/ProbablyOverthinkingIt
http://36kr.com/feed
https://coreos.com/atom.xml
http://mcavage.me/atom.xml
https://www.keithcirkel.co.uk/rss
http://www.qdaily.com/feed.xml

Make RSS feeds exportable to an importable backup file

I think it would be good if it was possible to export RSS feeds exportable to an importable backup file. But not to back up every single article that is posted in the RSS feeds, only to back up the RSS feeds themselves.

Idea: Loglevels

The loglevel should be easily configurable via the command line for scrape_feeds, scrape_favicons and the test flow. Right now you need to edit the file to change the loglevel which isn't great.

Idea: Secondary links

Show a secondary link to the comments for sites like Product Hunt, Reddit, HNews, Lobsters, DesignerNews etc

Roadmap: 0.2

We're currently discussing the roadmap for 0.2.0
So far we're thinking about:

  • OPML import

  • Listen to changes to feeds in realtime

  • Link to comments on HNews, PH, Reddit etc.

  • Improving RSS scraping performance

  • Stats on which feeds are failing when users add them

  • Docker

What else should be part of 0.2.0?

Future releases:

  • Social auth (Facebook/Google)
  • Expand the number of topics
  • Weekly emails
  • Suggestions for feeds to follow

Idea: Article specific comments page with sharing functionality

When clicking on a comments link below the article, you would be taken to a link within the Winds application and be able to comment via Disqus. You would also be able to share the article on your Facebook feed or via Twitter. This would give Winds a viral boost.

Discussion: Explicit upvotes

Right now we only use clicks and impressions as an indicator for interest. User in this thread:

#11

Suggested adding direct up and down votes.

Why all HackerNews?

I hope this is the place to raise this concern.

I initially subscribed to

  1. Programming
  2. Gaming
  3. AI & Machine Learning

then I noticed that all the items in my feed are from HackerNews.

While I will not be surprised if a lot of HN items turn up, I would expect items from other sources too. Gaming items aren't always found on HN, for instance.

Is this by design?

Idea: Single User usage

Actually Winds allows every user who knows the Url where the app is being hosted, to be used by everyone.

Would be cool to have the possibility to configure only one user, in order to make it more personal ( say you want to host this into your own website hosting space ).

Thank you in advance :)

Redirect from http to https

I get CORS errors when I try to log in unless I use https:

Fetch API cannot load https://winds.getstream.io/api/topics?limit=10&sort=createdAt%20DESC. 

Response to preflight request doesn't pass access control check: The 'Access-Control-Allow-Origin' header contains the invalid value ''. 
Origin 'http://winds.getstream.io' is therefore not allowed access. 
Have the server send the header with a valid value, or, if an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Logging in via https is fine, and preferable, so perhaps requests via http should be automatically redirected to https?

Bug: Csrf issue prevents to reset the password if the app crashes ( restarted by PM2 )

I found out that if for some reasons the application crashes during an API processing ( say Password Reset ), the CSRF token prevents the user to use any other call to the Winds app.

Therefore a way to handle this situation should be possible. As of today, the only way to fix this is by clearing the localStorage and/or related Cookies.

Thanks in advance :)

Idea: Rename feeds

For various reasons feeds could have an uninformative name, for example when subscribed to subfeeds of publications, which cover only part of a topic but their name doesn't show that. There are quite a few scenarios.
I propose being able to change the display name of a feed.

Discussion: Shrinkwrapping dependencies

Not sure how prevalent this is in OS, but I just had a linter error and realized I had to stick to a previous version for my eslint-plugin-jsx-a11y devDependency. Have you considered shrinkwrapping dependencies?

Reduce API calls to Stream

Right now every article is repeatedly sent to Stream even if the data didn't change. We can optimize that and improve performance.

Idea: import OPML files

Most people aren't going to switch if they have to enter urls in one by one, not worth the time to try something you might not like.

Discussion: Winds v0.3.0

Here are some items we're considering for Winds 0.3:

  • Improvements to ranking algorithm
  • Sharing via Buffer
  • Suggested feeds: #37
  • Social auth (Facebook)
  • Explicit upvotes: #29

Let us know what changes you'd like to see for Winds!

Bug: is ORM with MongoDB working?

As said yesterday,
this is an outcome of when I was working yesterday on the Docker.

I was trying to setup my MONGO_URI environment variable to something like

MONGO_URI="mongodb://localhost:27017/winds"

but the application complained that something with orm could not be started ( I guess the sails-mongo package was not able to use it ).

So I went to your sourcecode and I found this and this where you are actually declaring MONGO_URL, which is not the same declared in the README.md file.

Therefore I suggest to update the readme with a valid example of connection string that actually works. On my side, the Mongo default installation doesn't have any requirement for user or password, you can just connect and use it.

Thanks in advance,
Julian

Password reset doesn't work and changing password almost never works

The password reset feature doesn't work and changing password almost never works. Currently neither the password I used before or the password I tried to change to works, for some reason. So it looks like I will be locked out of my account when I close my current Winds session.

Idea: setup SMTP email server editing config/emails.js

This is an outcome of the yesterday outcome of when I was working on the Dockerfile.

I was trying to setup my current email server on Winds, when I found that the application is not able to setup any kind of other server, because of the way the source code is being written up.

Actually you can change the configuration on this file although, the reality is that you need ( as an example of the smtp ) is to totally override the provider obvject inside the Mail Service, changing only Backend, Username and Password is not enough.

My proposal is to make the configuration more flexible, in order to pass the full provider object in the config file. Therefore it will be possible to use any other methodology without the need to rewrite the Service.

Thanks in advance,
Julian

Account deactivation

How does one delete a registered account? It should not be overly difficult to find the option to delete your own account.

Bug: Duplicate keys for React

Warning: flattenChildren(...): Encountered two children with the same key, site-581cab545632dccc09c8455f. Child keys must be unique; when two children share a key, only the first child will be used.

Idea: export a list of RSS feeds

After I try the demo, if I want to host my own version, I'd like a way to export the URLs I've entered on winds.getstream.io to then import into my own hosted version.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.