Verify that multiple local units can be independently communicated with using the new single proxy interface. This will greatly ease development resources and allow much more simple single user experimentation. Prior proxy interaction required two independently configured proxy's to operate and were hard to manage.

A contributors guide is needed to help contributors understand the standards for pulling back into the master. Although all the code is not currently compliant the high level overview is:

• Python follows PEP8 standards for code formatting
• Ignore binaries (i.e. .pyc files)
• Follows our layering practices with IO at an interface (i.e. Flask) for applications with the UI abstracted to another level.

This is present in most places but we should sanitize user input. I realized this when I could program a default GPS location onto Faraday with an extra number and it caused erroneous operation. Things like this. Everywhere.

It's been noted that the message application in tutorials seems to be dropping bytes in multi-packet messages. Any message longer than the single packet (causing fragmentation) will cause bytes to drop between the concatenated packets. A screenshot has been provided.

LEFT = RX
RIGHT = TX

Note the missing "f" in the sentence for the work "of" and the missing numbers in the counting sequence.

In most cases this is only an issue if you configure Faraday to update over UART with a long interval ( >5 seconds could get annoying). The symptoms are:

• Proxy will respond with an error saying that callsign-nodeid does not exist or the port doesn't exist
• If UART is set to never update then you will never be able to get data from Faraday, even if it sends down Remote RF unit traffic.

Regardless, this is poor design. Proxy should know what's connected and what to expect prior to any data being sent down over USB.

All our configuration INI files are not included in the .gitignore file. This is because we want to track the files and they were under heavy development. However, we are now slowing down on these files changing and it is relevant to remove them so we do not accidentally commit them with our own serial ports and callsigns.

When these files are updated we can simply force Git to add them for that single special commit.
git add --force my/ignore/file.foo

Tasks

• Ensure all current .INI files in Proxy, Device Configuration, Telemetry, and APRS use "REPLACEME" for CALLSIGN, NODEID, and COM
• Update .gitignore to ignore the specific INI files in these documents which change often and should be ignored since they change with every user (callsign, nodeid, etc).

Update Telemetry application as it is moved from SVN to GitHub. Overall functionality:

• Provide worker thread which queries Proxy to obtain telemetry and save into queue
• Save data into an SQLite database
• Provide a RESTful interface to decoded telemetry

Per standards we need to update all old code to PEP8.

For proper Github support update the readme files to .md files for viewing on Github.

https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet

As discovered in FaradayRF/Faraday-Firmware#52 the program sent un-encoded data through the network interface and this cause framing errors (Start Byte) during some configuration settings (default factory reset).

Root Cause

I determined the root cause to be the returning of a raw parsed device configuration dictionary, when the unit was factory reset there was a non ASCII / utf-8 character that caused a JSON or FLASK interface framing error. When a unit was configured the invalid byte(s) were not present.

http://stackoverflow.com/questions/22216076/unicodedecodeerror-utf8-codec-cant-decode-byte-0xa5-in-position-0-invalid-s

http://stackoverflow.com/questions/4342176/how-do-i-serialize-a-python-dictionary-into-a-string-and-then-back-to-a-diction

Solution

The program requires returning the parsed dictionary through the FLASK interface and therefor needed to figure out how to transfer the parsed dictionary through FLASK. BASE64 was used on Proxy for the RAW HEX data packets to safely transfer them through the network interface, the same can be applied to this problem.

• Decode device configuration packet BASE64
• Parse device configuration packet into dictionary
• Serialize dictionary with cPickle
• Encoded cPickled dictionary with BASE64
• Return encoded packet through JSON and the FLASK interface

else:  # If a GET command
        """
            Provides a RESTful interface to device-configuration at URL '/'
            """
        try:
            # Obtain URL parameters
            callsign = request.args.get("callsign", "%")
            nodeid = request.args.get("nodeid", "%")

            callsign = str(callsign).upper()
            nodeid = str(nodeid)

            # Flush all old data from recieve buffer of local unit
            proxy.FlushRxPort(callsign, nodeid, proxy.CMD_UART_PORT)

            proxy.POST(str(callsign), int(nodeid), UART_PORT_APP_COMMAND,
                       faradayCmd.CommandLocalSendReadDeviceConfig())

            # Wait enough time for Faraday to respond to commanded memory read.
            time.sleep(2)

            try:
                # Retrieve the next device configuration read packet to arrive
                data = proxy.GETWait(str(callsign), str(nodeid), proxy.CMD_UART_PORT, 2)

                # Create device configuration module object
                device_config_object = deviceconfig.DeviceConfigClass()

                # Decode BASE64 JSON data packet into
                data = proxy.DecodeRawPacket(data[0]["data"])  # Get first item
                data = device_config_object.extract_config_packet(data)

                # Parse device configuration into dictionary
                parsed_config_dict = device_config_object.parse_config_packet(data)

                # Encoded dictionary data for save network transit
                pickled_parsed_config_dict = cPickle.dumps(parsed_config_dict)
                pickled_parsed_config_dict_b64 = base64.b64encode(pickled_parsed_config_dict)


            except ValueError as e:
                print e
            except IndexError as e:
                print e
            except KeyError as e:
                print e
            except StandardError as e:
                print e

        except ValueError as e:
            logger.error("ValueError: " + str(e))
            return json.dumps({"error": str(e)}), 400
        except IndexError as e:
            logger.error("IndexError: " + str(e))
            return json.dumps({"error": str(e)}), 400
        except KeyError as e:
            logger.error("KeyError: " + str(e))
            return json.dumps({"error": str(e)}), 400

        return json.dumps({"data": pickled_parsed_config_dict_b64}, indent=1), 200, \
               {'Content-Type': 'application/json'}

Using POSTMAN this is shown to provide the encoded dictionary through the FLASK interface.

I wrote a quick receive size test scrip that prints the dictionary on the receive size by simply decoding the the base64 and de-pickling.

Although I'd like to just return a dictionary directly or at most only require a decoding of BASE64 it seems correct to also use cPickle to provide the parsed results directly to the FLASK GET request.

Device Configuration Application / Interface Example

Below is an excerpt of the code in the Device Configuration tutorial after updating to the encoded configuration dictionary data implementation. Note the BASE64 and cPickle operations.

#Display current device configuration prior to configuration flash update (Send UART telemetry update now command)
#Send the command to read the entire Flash Memory Info D allocations

try:
    r = requests.get("http://127.0.0.1:8002", params={'callsign': str(local_device_callsign), 'nodeid': int(local_device_node_id)})
except requests.exceptions.RequestException as e:  # This is the correct syntax
    print e

#Print JSON dictionary device data from unit
print r, r.json()
raw_unit_json = r.json()

# Decode and depickle (serialize) device configuration parsed dictionary data
b64_unit_json = base64.b64decode(raw_unit_json['data'])
unit_configuration_dict = cPickle.loads(b64_unit_json)

Testing Bug Fix Implementation Application Side

It looks like its working!

try:
    r = requests.get("http://127.0.0.1:8002", params={'callsign': str(local_device_callsign), 'nodeid': int(local_device_node_id)})
except requests.exceptions.RequestException as e:  # This is the correct syntax
    print e

#Print JSON dictionary device data from unit
raw_unit_json = r.json()

# Decode and depickle (serialize) device configuration parsed dictionary data
b64_unit_json = base64.b64decode(raw_unit_json['data'])
unit_configuration_dict = cPickle.loads(b64_unit_json)


print "\n************************************"
print "PRIOR TO CONFIGURATION UPDATE"
print "Unit Callsign-ID:\n", str(unit_configuration_dict['local_callsign'])[0:unit_configuration_dict['local_callsign_length']] + '-' + str(unit_configuration_dict['local_callsign_id'])
print "RAW Unit JSON Data:", unit_configuration_dict
print "************************************"

outputted a valid dictionary as shown in the output of the tutorial print statements:

Wahoo! It worked! I successfully programs a factory default REV D1 Faraday unit.

In the effort to avoid maintaining multiple FaradayIO modules in each Tutorial example script a relative import to a common copy should be implemented.

Altitude is checked up to 17999.90 meters, however, we should be able to program static faradayconfig.ini files during Faraday configuration up to 17999.99.

Problem is in update_gps() in Faraday_Proxy_Tools/FaradayIO/deviceconfig.py and includes the check as well as the warning response text when the check fails.

When a unit without a GPS is configured with set coordinates, the GPS Altitude that is returned is invalid unless it has 4 decimal points.

Configuration:
DEFAULT_ALTITUDE = 185
Returns:
"GPSALTITUDE": "185\u0000\u0000\u0000\u0000\u0000",

Configuration:
DEFAULT_ALTITUDE = 185.000
Returns:
"GPSALTITUDE": "185.000\u0000",

Configuration:
DEFAULT_ALTITUDE = 185.0000
Returns:
"GPSALTITUDE": 185.0,

We should help make the error responses more specific, granular, and overall more helpful in pinpointing exactly what the user typed in wrong.

If we plan on integrating this with a GUI we should probably think about actual JSON responses that can automate highlighting of erroneous fields.

Proxy.py: https://github.com/FaradayRF/Faraday-Software/blob/master/proxy/proxy.py

While writing the device configuration program I found that the Proxy program POST expected 164 byte items and this doesn't make much sense. @kb1lqc informed me that this was because of the fixed packet length of the UART layer 4 protocol.

This doesn't make sense because fixed packets are 128 byte's in total prior to framing. The only reasonable explanation is that 164 bytes were a chance length from his testing of telemetry packets and are a result of the specific case of testing and BASE64 encoding size increase. This length would likely change with different data POSTed.

The POST command filtering data for fixed length to avoid random data is pretty useless because:

• It's the UART network stack's job to filter out random data
• Unlikely to pass random unwanted data (noise)
• Just because a layer if fixed length doesn't mean the tranmission medium (FLASK) should expect it. If the byte length is increased the FLASK proxy server will also need to be changed. Decoupling is a good idea.

total = len(data["data"])
            sent = 0
            for item in data['data']:
                if len(item) != 164:
                    logger.warning("Packet not 164 characters long!")
                    logger.warning(str(item))
                else:
                    # Correct length packet, send it by putting on the queue!
                    try:
                        postDicts[station][port].append(item)
                    except:
                        postDicts[station][port] = deque([], maxlen=100)
                        postDicts[station][port].append(item)
                    sent += 1
            return json.dumps(
                {"status": "Posted {0} of {1} Packet(s)"
                    .format(sent, total)}), 200

The process as initial release requires the configuration of identical local unit callsign/id in INI files and to update the tutorial script (lines 18/19). This is confusing and error prone. It should be simplified to a single INI file for local device selection.

We no longer need to use this file. Simply remove it!

Per some comments on #76

• Make a dedicated checking program to check all configuration inputs.
• Update to use logging module

Move proxy code over from SVN into GitHub. In doing so, clean up the code and simplify it according the the current understanding of the software architecture using POST and GET responses to localhost.

• Should cleanly implement POST and GET functions with BASE64 packets
• Should be lightweight
• License properly

When I have two units in the the proxy.ini file, proxy won't start if only one unit is plugged in.

Using the following proxy configuration, when both units are plugged in proxy.py starts properly.

However, with only the unit plugged into COM4 proxy.py shows this error.

Still with only the unit plugged into COM4 and commenting out the first unit in proxy.ini, proxy.py was able to start for the one unit that was connected.

When using the APRS program in Massachussetts it was realized that the APRS server upload was not accepting the station data, packets were received but APRS.fi was showing currupted packets.

Investigation showed good receive data but the APRS conversion prior to upload shows the leading 0 of the longitude getting dropped. I added debug code to print lat/lon and verified.

Until a standard module for Flask IO is created for users a basic IO should be created to maintain the layered interface architecture as described. The UI of the messaging application as it is developed in tutorials should be created to allow any script (and eventually any FLASK interface) to interface and input data and export data. This is not only flexible but allows an easy fork into a pure binary data version.

Software support for bootstrap loading is currently a wrapper program suite that glues:

• TI BSL script autogeneration python script
• FTDI CBUS GPIO bitbang python script to enable Faraday into BSL mode
• TI supplied windows program that performs BSL communications and uploading (.exe)

This currently will not support Linux.

Possible paths:

• Move to TI support cross platform MSP430 tools for BSL support
  • https://pypi.python.org/pypi/python-msp430-tools
  • http://pythonhosted.org/python-msp430-tools/commandline_tools.html#msp430-bsl5
• Run Wine on linux to support the simple .exe
• Roll our own BSL support for cross-platform

Add a tutorial section/page over-viewing how to config, setup, and use a Faraday digital radio with the telemetry application and RF transmissions. This should be really as simple as explaining how to configure the RF beacon enable, power, frequency, and interval timing in the device configuration. It may be wise to also give general tips and practices.

This should probably be written by @kb1lqc to maintain the flow of wording in the /start, telemetry, and aprs tutorials.

Per FaradayRF/Faraday-Firmware#65

Add decoding support to parse and display the firmware revision.

Following our contributions guide the Proxy application should be updated with proper DocString Sphinx documentation notes. The Tutorials section of FaradayIO contains examples of this intended format.

I noticed that APRS will not start with the latest telemetry.py update. The APRS window crashes and programs fails to start.

Bug tracked to the "getStations()" telemetry call that APRS tries to use.

Telemetry.py prompt reports this failure:

I printed the callsign variable at line 357 and it was NoneType

I believe this is failing because the .upper() assumes a string but got None. This was recently merged and this situation (using APRS) probably wasn't vetted properly.

Actions

I'm going to push a simple pull request to remove the .upper() to allow the program to work. This may cause failure if the user doesn't enter their callsign in all UPPERCASE in the configuration(s).

Title says it all. Ensure the Readme is as thorough as possible for first Faraday users to get up and running. This includes:

• Configuration documentation
• Browser API use, especially for debugging
• SQLite Browser installation and use

We need to add/verify the correct GPL license file to all files/folders/repositories applicable.

As shown below, there's an obvious "tick" when plotting ADC data. This could be a poorly implemented parser or it could actually be in the Firmware @kb1lqd. If nothing about the ADC code changed since the Balloon flight then it's probably the software. I could also use the balloon software to cross check this firmware.

Not critical to fix right now (10/17/2016) but we should get onto this when looking at ADC firmware or telemetry.

• Cross-checked balloon flight (8/2016) telemetry and proxy software

It's become clear that when dealing with multiple units there needs to be an easy way to query the units information without programming it first. This issue enhancement is asking to add a file like simpleconfig.py that only reads the currently connected units configuration.

Issue
I've noticed that after starting Proxy the first data sent to the unit always fails. Retrying the transmission of the data is successful as is all future transmissions (to my knowledge).

The program first using proxy displays this error:

I see proxy log:

Proper Operation
Proper operation would not cause error on the first data transmission to the unit. Although I have not debug breakpointed or scope captured on the Faraday device to confirm I have no reason to believe data is being transmitted to the unit on first attempt. It looks as if Proxy can't handle the first lookup of the unit correctly.

I just became aware of aprslib which looks like it takes care of a lot of the APRS functions. Also, it's open source and on Github.

We should see if it implements APRS-IS connection as well as packet forming and parsing better than I did. Trying my best seemed to get me usable functionality but I did not implement much of APRS at all and it would be great to leverage a project that actually focuses on implementing APRS.

This would be a very significant amount of work for the APRS application. However, better now than before we attempt to implement more APRS functionality or squash any lingering bugs that this library inherently solves.

Per APRS definition:

Latitude is expressed as a fixed 8-character field, in degrees and decimal minutes (to two decimal places), followed by the letter N for north or S for south

Longitude is expressed as a fixed 9-character field, in degrees and decimal minutes (to two decimal places), followed by the letter E for east or W for west.

I want to ensure that I'm always sending the correct value from the APRS application

Board temperature is parsed as ADC7 when it should be ADC8

An effort should be made to update the developer tutorials to:

The point of this development series is to introduce Faraday as a tool to program with rather than use. The reader is walked through basic local device commanding, remote (RF) commanding, and at the end can tie these simple skills together to use the "Packet Forwarding" command as create a simple single packet messenger.

The developer tutorial series 2 further take this messenger and build packet/file fragmentation functionality and ARQ (automatic retry-request) ability. The end goal is to example using only the non-optimize "packet forwarding" command a complete file transfer program.

• Verify all still work as intended
• Clean and de-clutter
• Update readme.md tutorials to better fit style of basic tutorials @kb1lqc wrote

Tutorial Scope / Descope

I'd like to move or remove

• Tutorial 0 - Proxy basics
  • Remove, it is completely outdate from the /start tutorial(s)
• Tutorial 3 - Device configuration
  • This dives deeping inter the device configuration but is largely convered by the alreaday introduced topic and newly created simpleconfig.py https://github.com/FaradayRF/Faraday-Software/tree/master/Applications/deviceconfiguration
• Tutorial 6 - Device Factory Reset
  • I think we should just move this to a special tutorial section of odd's and end how-to's...

Packet Definitions

@kb1lqc @hdkmike Do you think this is an appropriate tutorial set to introduce packets/frames used in the command application (layer)? If you agree then I can more heavily focus this series on diving into main high level topics like framing, error detection, fragmentation, encapsulation, and ARQ using Faraday's simple command architecture at the application layer level.

Examples from engineering documents:

Action Plan

• Remove tutorials page for as this was replaced by "Start" tutorials: https://github.com/FaradayRF/Faraday-Software/tree/Developer_Tutorial_Update_2-18-17/Tutorials
  • Move to: https://github.com/FaradayRF/Faraday-Software/tree/Developer_Tutorial_Update_2-18-17/Tutorials/Python_Developer_Tutorials
  • We should probably place some quick info/links in that parent directory...
• Move tutorials to a "misc" or similar tree
  • Device Configuration
  • Device Factory Reset
• Combined tutorial sets 1 and 2 for developers. No need to split. Final tutorial should contain a fully working ARQ based RF file/data transfer program.
• Reformat the main developer overview tutorial page to better match /start tutorials
  • https://github.com/FaradayRF/Faraday-Software/tree/master/Tutorials/start
    Add any missing topics

@kb1lqc and @hdkmike what do you think?

There's a few lingering bugs/enhancements I'd like to make to Telemetry prior to first users. These include:

• Update INI files to uppercase and force uppercase when reading data in
• Fail safely when proxy or INI file data is bad

While updating the device configuration application and respective proxy module I have updated several fields that should be brought back into the main "faradayIO" repository. This is a result of the faraday_config.ini interpretation and upgrades. I also commented out / updated the FaradayIO library deviceconfig.py file.

• Removed specific port 3/4/5 bitmask functions into a single function
• Added GPIO port 5 self variable

Make sure these updates are brought back into the FaradayIO module!

We show example API's and other uses of code, we need to get better at documenting it. Currently we brute force this on markdown pages and show screenshots of output. We should standardize on something.

To facilitate easy updating of units and only requiring the changing of intended fields there should be an update made that allows the application to rewrite the INI file "faraday_config.ini" with the units queried current configuration. This file can then be edited manually and programmed back into the Faraday unit.

The parsing module returns a string for boot counter when it should be an integer. Due to structure parsing no conversions are needed as integers are returned by default.

dictionaryData = {'BootCounter': str(parsed_packet[0]),
                          'ResetCounter': int(parsed_packet[1]),
                          'BrownoutCounter': parsed_packet[2],
                          'Reset_NMICounter': parsed_packet[3],
                          'PMM_LowCounter': parsed_packet[4],
                          'PMM_HighCounter': parsed_packet[5],
                          'PMM_OVP_LowCounter': parsed_packet[6],
                          'PMM_OVP_HighCounter': parsed_packet[7],
                          'WatchdogTimeoutCounter': parsed_packet[8],
                          'FlashKeyViolationCounter': parsed_packet[9],
                          'FLLUnlockCounter': parsed_packet[10],
                          'PeripheralConfigCounter': parsed_packet[11],
                          'AccessViolationCounter': parsed_packet[12],

Should be:

dictionaryData = {'BootCounter': parsed_packet[0],
                          'ResetCounter': parsed_packet[1],
                          'BrownoutCounter': parsed_packet[2],
                          'Reset_NMICounter': parsed_packet[3],
                          'PMM_LowCounter': parsed_packet[4],
                          'PMM_HighCounter': parsed_packet[5],
                          'PMM_OVP_LowCounter': parsed_packet[6],
                          'PMM_OVP_HighCounter': parsed_packet[7],
                          'WatchdogTimeoutCounter': parsed_packet[8],
                          'FlashKeyViolationCounter': parsed_packet[9],
                          'FLLUnlockCounter': parsed_packet[10],
                          'PeripheralConfigCounter': parsed_packet[11],
                          'AccessViolationCounter': parsed_packet[12],

ADC7 in the telemetry application should simply change to the "VCC" column when parsed because it is connected in hardware directly to the VCC pin on Faraday.

In telemetry parsing tutorial the script incorrectly uses a non-extracted payload packet which resulted in the "packet" being decoded being that of a full frame/datagram of the telemetry packet. This resulted in a byteshift of data in the decoding.

Incorrect Output

Correct Output

Root Cause

Fix

Telemetry is currently only able to connect to one Proxy unit. It should be updated to bring in data from as many Faraday radios as are connected to Proxy.

When using proxy and the telemetry application I noticed that when I use proxy with lower case callsign that Telemetry cannot find the unit even though it is connected and proxy running. All uses of callsigns should be forced UPPERCASE but proxy is not:

Configuration 1

Proxy: kb1lqd-1
Telemetry: KB1LQD-1

Results in ERROR

Proxy - Configured "kb1lqd-1"

Telemetry - Configured "KB1LQD-1"

Running Telemetry

Telemetry looks for KB1LQD-1 from proxy but proxy did not force uppercase for callsign and therefor not found.

Configuration 2

Changing Telemetry to lowercase: kb1lqd-1 results in same error as telemetry properly forces the lowercase to uppercase:

Node.JS

• Installed: https://nodejs.org/en/

Dashboard Ideas:

• OpenMCT: https://nasa.github.io/openmct/
• Atlassian Atlasboard: http://atlasboard.bitbucket.org/
• PYXLEY: http://multithreaded.stitchfix.com/blog/2015/07/16/pyxley/

AngularJS Dashboards

• https://angularjs.org/

A simple GUI has been developed in #145. It it by no means a long-term GUI but helps alleviate immediate needs and shows how to interface Python and Javascript!

We are now in Alpha with units in the wild and this means we not only need to be clean with master releases but we need to keep a changelog and versioning.

@kb1lqc and @hdkmike I'd like to get a changelog in play so that everyone can see what the project is up to. I'd like it to not be overly cumbersome and not slow down the flurry of mini commits but be useful as an "unreleased" and a "released" version. I suspect we should chose a point in software and "tag" it. Possibly entering on a specific release cycle is preferred.

I believe we've got to finish up our current push through tutorials but finishing that it might be a good tag.

This (among others) should be compiled into a general guideline checklist for all pull requests to document what is happening in human readable format.

Resources

• http://keepachangelog.com/en/0.3.0/

In an effort to start making issues and work a bit more clear we want to have an issue template to help clearly identify issues/bugs/features. Github supports an issue template which can be visible or hidden.

• I'd like to hide the template, no use in having it visible to most users. This ticket documents it anyways.
• It should clearly identify the issue with a summary followed by a more in-depth explanation
• A simple explanation of what hardware if any was used when the bug was observed
• Understanding what version/branch/commit of the software is also useful

Overview
An unused bit in the GPS bitmask of the FLASH parsing should be updated to communicate the presence of a GPS unit on the Faraday unit. This will be used to easily determine if a GPS should be onboard or not but does not indicate the status of the GPS unit if installed, for that the presence of the telemetry "GPS FIX" is used.

Implementation Details

• Update software to parse "GPS_Present" bit from GPS bitmask
• Update device configuration INI file to include GPS_Present bit configuration
• Update firmware for default unit factory reset value

Operation
This bit is used simple to query the presence of a GPS.

• GPS_Present = HIGH
• GPS is present and Faraday should attempt to parse valid GPS string
• GPS_Present = LOW
• GPS is a not present and Faraday should default to internally saved default location

Per issue #49 the master telemetry now breaks.