sparkfun / phant Goto Github PK

Currently when you run phant, the output looks like this:

$ phant
phant http server running on port 8080
phant telnet server running on port 8081

Add a bit more guidance on startup, and rework logging to support logging levels (info, debug, err, etc).

Like https://data.sparkfun.com/streams/ and https://data.sparkfun.com/streams/PUBLIC_KEY view, create a view for localhost setup.

1.) Handle /streams/ route.
2.) Create 1 view to show all the streams and
3.) Another page to show specific stream when clicked from #2 or accessed as http://localhost/streams/PUBLIC_KEY

The backend should already support paging, but there's no way for users to easily request the next page.

Create a rate limiting system that can be used in output modules. The default rate limiting system can be memory based, or just always return true. The production module will most likely be memcached based, since multiple servers will need to check feed status.

Wunderground has a really neat system that notifies the owner of the weather station when it has not reported data after 2 hours, then 24 hrs. It would be a 'nice to have' : an email when your feed has not been pushed new data after 2 hours have elapsed. The wunderground feature has been crucial when developing my station and is a great notification to get.

The user should probably enable or opt-in this feature.

Here's the email I envision:

Title: "Data Channel Alert: [name of channel] Disruption Detected"

Hi owner of [name of feed],

We noticed your feed traffic has been disrupted. Click here to see the latest log.

We want to let you know so that you can check your hardware if necessary. Click here to change your notification settings or unsubscribe.

-SparkFun

Here's the wunderground error email for comparison:

Title: Wunderground Alert: PWS ID #KCOBOULD115 Reporting Error Detected

ATTENTION: Weather Station Owner

The Weather Underground PWS network has detected that your personal weather station (ID #KCOBOULD115) has not reported to www.wunderground.com for more than 2 hours.

Please check that your station is operational by visiting your station's settings here: http://www.wunderground.com/personal-weather-station/mypws

For more details and information about troubleshooting visit our PWS wiki page: http://wiki.wunderground.com/index.php/PWS_-_Personal_Weather_Stations

We want to ensure that you are informed on the status of your weather station so you can continue to receive accurate data from Weather Underground.

Thank you,

• The Weather Underground Staff

When I go to https://data.sparkfun.com/streams/Jxyjr7DmxwTD5dG1D1Kv on Mozilla Firefox (v30.0), I do not see any of the data, only empty rows. Please see this screenshot:

Add a hidden blank field to the stream create form. If the field is populated on submit then it's a bot that's filling in all form fields, so fail.

Cheaper than a captcha but just as effective for all passive bots that come through, while not impacting user experience.

Currently stream stats are split into separate calls, and it's incredibly slow when using GlusterFS. Move everything into one call so it only has to loop through the files once.

Add a route to the management modules that allows users to delete stream metadata and logged data.

I try to start phant, and I'm getting these errors. I have no experience with Node.js, so I don't know if that is the issue.

Weird. Creating a channel with name 'Test', description of "lkj" and "temp" for the only field throws the following error:

Error
Cannot read property 'length' of undefined

add an option to the creation step to email users the keys to their newly created stream

Add an optional field or map to allow users to show where their feed is located.

documentation is fairly scarce. fix that.

It would be great to have the ability to store stream data in MongoDB.

GPL makes some folks slightly uncomfortable for a number of reasons. MIT and Apache2 are more common in the Node community.

A coworker just considered forking the project for use in other software. The GPL license could be a blocker for us down the road.

Would you consider a change?

Thanks!

Any input that contains a backslash automatically converts that backslash to a forward slash in the stream.

New streams need a couple more checks when created:

• Field count limit
• restrict to a-z, 0-9 & _

In addition to (or in lieu of) jsonp, I was wondering if support could be added for CORS?

Timothy (via email) requested that we make data.sparkfun.com a channel at ifttt.com. Look into if that's possible.

I got the auto-email from phantworker with the delete key but I don't know how to use it - so I am going to create a new channel then delete them both once I have the link structure. Would be good to have the structure (or a link to a page that has it) in the email.

Can we simplify the stream schema and remove markdown from the stream descriptions? Is there a compelling reason to keep them formatted with markdown?

The advantage of a short text input are: less opportunity for spam and simplified layouts. The advantage of markdown is obviously increased flexibility.

If we keep it as markdown, I think we should only display a snippet in the stream/list and stream/view. And reveal the full description with a "more" link.

Thoughts?

Sort public stream list by most active

Test all http output formats, stats and paging

Add a tag list method to metadata & a tag stream list responder to the http manager.

MQTT (http://mqtt.org) seems to be an appropriate protocol for data transfer from small embedded machines. Client libraries for a wide range of languages are available. The broker of choice currently seems to be Mosquitto, but other MQ brokers also understand the mqtt protocol.
Would be nice, if phant could subscribe to a mqtt topic and collect data from there.

expressjs is overkill for the http input and output modules. rework routing and param parsing so no (or minimal) external dependencies are needed.

Just so I know how where to place this in the playingfield: how is sparkfun related to Stream processing engines like Storm / Samza / Spark Streaming?

From my understanding Sparkfun gices you the tools to easily publish streams, whereas Storm, etc. enable you to subscribe to those streams and process them. Is that correct?

Add a JSON responder to the HTTP Manager so users logged into sparkfun.com can interact with their feeds on data.sparkfun.com.

Make public keys to streams globally unique

phant-keychain-hex & phant-throttler-memcache are the only modules with unit tests. add separate unit tests for all modules in the main phant repo.

Change the route from DELETE /streams/KEY/delete to DELETE /streams/KEY

Titles and keys should not need to be predefined, unless desired. Please allow more free-form data to be sent, similar to http://dweet.io/

Using the telnet interface, creating a stream should not require tags to be defined. According to the documentation they are optional.

The site navigation on phones could use some work.

Add support for HTTPS

Add a hidden flag to metadata so users can hide feeds from the main stream list.

Add a route to http input or output that allows a user to clear their stream.

When creating a feed, if there is a problem please don't clear all the info I've typed in. Please just indicate which box needs attention.

Today when creating a new stream I used a tag with a space in the name and got this code:

Creating stream failed - invalid tag name: event trigger. valid characters are _, a-z, A-Z, and 0-9

And lost all the description, etc I had inputted.

From this bug report: https://forum.sparkfun.com/viewtopic.php?t=38165#p170711

Add test that checks limit, headers, and window reset.

The stream data view should support paging and streams a large amount of columns.

Add notifiers to telnet manager, and add phant-notify-email to the default server config.

Make sure these commands are tested:

• create
• list
• delete

Make the telnet stream creation output generate example URLs, so it's easier for users to get started.

Set a sane limit to the size of a GET request and return HTTP 414 when over the limit.

I haven't perused the code (yet) but it seems to me that if an IoT device has to make a connection to Phant.io via HTTP (i.e. unencrypted) and has to put the private key into the POST command, this can be easily intercepted by a third party between the IoT device and the Phant.io server, and can be abused to post false updates, or do other Bad Stuff(TM).

Private keys (by nature) should remain private and shouldn't be shared over the network. That's what public keys are for :-)

The README in the examples directory makes reference to a stream that does not exist.

https://data.sparkfun.com/streams/v0gxy8OnGVCOGl38oqM4

People should be able to subscribe to streams. Probably use Atom.

• https://www.npmjs.org/package/feed

The default http server only responds to /input and /output, and currently hangs if any other path is requested.