Title and keys must be predefined about phant HOT 8 CLOSED

By keys, do you mean field definitions or public/private/delete keys?

from phant.

field definitions.

from phant.

That's one feature we started with, but here are a few reasons why we now require field definitions:

• Phant storage is modular. Dynamic field definitions work when the storage used is something like MongoDB or JSON, but doesn't work for something like CSV storage. CSV is a good storage system because we can avoid having to include field names for every row, and that keeps the file size small.
• Dynamic fields limit output formats. TSV and CSV output wouldn't really work if you had new field definitions for every row, and those formats are handy for people that want to load the data into spreadsheet software.
• Field definitions allow us to validate sensor input. If a device doesn't send the correct fields, we can alert the user because we know what to expect. This can be very helpful when debugging.

We also wanted to make sure that the streams were described in some way (title, description) so users can browse the public stream list and find interesting data to play with.

Dweet looks very interesting. Thanks for sharing the link!

from phant.

@naikrovek I'm going to close this one, but feel free to reopen this if you have ideas for how to support things like CSV output with dynamic fields in every row.

from phant.

feel free to close again; I wanted to write this here rather than in a private email.

I don't see an issue with CSV output in this specific case. One can assume that if a user has different fields regularly that they know what they're doing, and that they won't output with CSV, but with JSON. In my mind this is simply a limitation of CSV and is probably partially why JSON is so effective for things like this.

If you must support CSV output in this specific scenario, then offer the user an option to map all the Phants' columns to exported spreadsheet columns (hard) or just export one column per unique field name and let the end-user munge the data themselves (easy).

I have been operating on some assumptions which put me on a different plane than SparkFun in terms of expectations, and the expectation mismatch is my fault, not SparkFun's.

My assumptions:

1. The end user will know how to deal with their data, whatever it is.
2. The end user would rather not need to configure a Phant before it is used, or knows the data logged may change based on conditions which might not be known at configuration time.
   • If it's raining, I don't need to log humidity.
   • If a sensor fails, its data might not log at all.
   • If I switch a sensor and the output format or scale or units of measure change, I don't want to create an entire new Phant to use it.
   • If I decide I no longer want a Lux sensor at all, and want an ambient noise sensor instead, I don't want to create a whole new Phant to accommodate that. I also have 12 other sensors that I'd have to lose historical data on when I switch to a new Phant, or I'll have to merge the data later.
   • If there is a bug in my code, and I spell a field name wrong, and I don't notice it right away, I'd rather the entire entry not fail because of that single typo.

I don't know. Maybe i'm way off base. Go ahead and close if you like.

from phant.

@naikrovek I think these suggestions make sense for outputting CSV data, but phant uses CSV for data backend storage by default, not JSON. It's set up this way because JSON files are huge compared to CSV, and if you have a sensor that's logging a bunch of data automatically storage size becomes important. Because of this, I think trying to make CSV storage deal with dynamic fields would make it unstable.

Adding other backend storage solutions like SQL based dbs would also be a problem if we allowed dynamic fields, because we'd have to ugly things like schema changes, or we'd have to serialize the data and shove it into one field. I think you are definitely right if phant wasn't built to be completely modular, but since it is we have to try to come up with solutions that fit all possible backends... like thermal printer output.

from phant.

I am not a JavaScript developer. I think object storage is what Redis does, though. Maybe that would be more robust for storage? I dunno.

For a relational DB, you could use the tuple design pattern. Schema doesn't change based on data, and can be extended all you want. Uses LOTS of rows, though.

I see LOTS of great potential for this software. This (and dweet.io) are like twitter for machines.

I won't comment further. My id@gmail if you want to continue without polluting this bug any more.

from phant.

@naikrovek Yep. Redis would work great, as would any NoSQL solution, but that's not really the point. The difference between phant and other solutions like dweet.io is that we hope to easily support any storage system or live output that you'd like to plug into it, and we wanted to make the default install have no dependencies other than node.js, and still perform well on single board computers like the Ras Pi or BeagleBone Black.

If you already run MySQL, and want to write a phant output module for MySQL so you can easily run queries against your logged data, you can. It should be fairly easy to do that currently, and I think the pattern you described would make it hard to query because you'd have to pivot the logged data in SQL or in code, which kind of goes against why you'd use a SQL based storage system in the first place.

You could use the same key/value scheme for dynamic field support when pushing data to phant. It would work, but it's just not ideal. Believe me, I want to support dynamic fields if users want it. It just would come at the cost of making a whole set of storage solutions that expect field definitions ugly to implement and use.

I don't think this discussion is bad to have, and it's good to have it where others can get involved in the conversation. We made a new forum for larger design discussions like this one.

Check it out: https://forum.sparkfun.com/viewforum.php?f=44

from phant.