Go 1.6 is released.

• Move (some) dependencies to vendor directory

For the first sprint, we will merely deal with proprietary MAC commands which basically do nothing.
The implementation of the LinkCheck, LinkADR, DutyCycle, RXParamSetup, DevStatus, NewChannel and RXTimingSetup will be tackled by the second sprint.

The idea here is to test two commands, one initiated by the device, one initiated by the network in order to ensure the mechanism is working as expected.

We use the paho library as an go MQTT client which works nicely for the moment.

The library defines a ClientInt as an interface for the client, however, they also declares a public type Client which implements that interface and use that type within the latter interface.

The ClientInt is defined with:

type ClientInt interface {
    ...
    Subscribe(string, byte, MessageHandler) Token
    SubscribeMultiple(map[string]byte, MessageHandler) Token
}

where MessageHandler is

type MessageHandler func(*Client, Message)

And this... well.. makes the interface almost useless because we cannot build mock component properly anymore. We're stuck with the Client implementation.
For now, I've made a small workaround but it could be nice to make a PR to the paho lib in a while to fix that and:

• Use the ClientInt in the MessageHandler type (and even use an interface for that type)
• Remove the Client struct from the exported entities
• Rename the ClientInt in Client.

I don't know yet if the above changes are possible. They can make the whole a bit better to use though.

When a server is repeatedly unavailable or returns errors, it doesn't make any sense to keep trying to reach it. Therefore we should introduce some exponential backoff for faulty servers.

Metadata are currently marshaled to binary using an uncompressed json representation which is quite of a huge overhead. Although only the existing fields are present in the json, we still need to use {}":, plus a key to encode a single metadata value.

A more clever BinaryMarshaler for the Metadata object has to be written :)
Will do.

• Not everybody needs it
• You want to firewall this port, instead of only allowing localhost

To be better prepared for OTAA, registration at the Broker should be implemented differently:

1. Registration of the application with the Broker: POST /applications/
   data: appEUI, webhookURL
2. Registration of personalized devices for the application: PUT /applications/appEUI/personalized_devices
   data: devAddr, nwkSKey

Define a simple storage handler which will merely do the following tasks:
Roadmap:

- [x] Uplink handler
     - [x] Hold and gather uplink packets for a while before processing them
     - [x] Lookup info & partition packets list to associate a packet to a tuple (appId, devAddr, ... )
     - [x] For each cluster, deduplicate packets and decrypt associated payload
     - [x] Finalize, Ack / Nack + send to adapter

We need to write a handler which listen on HTTP request, decrypt, deduplicate and store all packets into a mongodb (as Jolie is currently doing).

We will plug several applications on that Handler, and, we should also figure out a way to register applications to that handler (either a shared database or via http requests as well ?).

Automated building and testing

It would be nice to be able to get some live metrics out of a router/broker. This issue is meant to discuss ways to implement this

To get the idea, look at:

• mod_status
• ngx_http_status_module
• StatsD
• Beats

Related to #38.

• In the Semtech Adapter: add the GatewayID to the RXPK
• In the Router: add fields from the latest Gateway stat
  • lati,long,alti
• In the Router: Duty Cycle information (see #50)
• In the Broker: add DevEUI and AppEUI
• In the Handler: mark the "best" gateway

We have to agree on the log format:

• Which pieces of information are important to know ?
• How can we format the output such that it is easy to parse ?
• How many different levels do we need ?

etc..

Because this is not very nice to say:

$ ./router
panic: strconv.ParseUint: parsing "": invalid syntax

goroutine 1 [running]:
main.parseOptions(0x0, 0x0, 0x0, 0x0, 0x32a080)
    /.../github.com/TheThingsNetwork/ttn/integration/router/main.go:105 +0x1e7
main.main()
    /.../github.com/TheThingsNetwork/ttn/integration/router/main.go:24 +0x27

What we call a packet at the current moment is fixed over time and for every component.
That approach is too both too restrictive and wrong. Packet has to evolve over time and each component is contributing to those changes. At the end (after a handler publish it to an application), a packet should be way more than it was when received by a router.

We will consider at first only three different types of message:

Confirmed data
Unconfirmed data
Join

Each message has intrinsically a direction (up or down).
The distinction and the mechanism to handle them should be flexible enough to allow implementation of new types easily. We're likely to consider a Command pattern or something similar here.

To avoid losing time with TCP handshakes, re-use previous TCP connections between Router and Broker (and maybe also Broker-Handler).

The Client's Transport typically has internal state (cached TCP connections), so Clients should be reused instead of created as needed. Clients are safe for concurrent use by multiple goroutines.
See: https://golang.org/pkg/net/http/#Client

Last step of the the first Sprint, the implementation of the Over-The-Air-Activation process. This will require to strengthen the responsibility of the Broker and its Network Controller.
Nothing is settled yet about the way a handler is supposed to confirm a registration to an application.

Remove cmd; context will make clear what command is being run

Rename tag to something that is more specific, for example role, if this field is necessary in the first place

Considering Class A devices, downlink transmissions are merely response to an uplink.
We'll go for something simple at first:

Applications can publish packets to a queue with a given expiration date / TTL.
When a downlink response frame is available, a packet is pulled from the queue until a valid one is grabbed.
The packet is wrapped with the response, and a similar mechanism is set up for the Network Controller and MAC commands.

Then, the router has to determine the best gateway and window to which forward the answer. For the first sprint, a gateway will be picked at random and we'll consider making a cleverer choice in the next sprint.

see #3 on lorawan repo

The following test cases are currently skipped until we find a solution:

• https://github.com/TheThingsNetwork/ttn/blob/develop/core/adapters/http/adapter_test.go#L135-L141
• https://github.com/TheThingsNetwork/ttn/blob/develop/core/packet_test.go#L75-L103

Follow Go conventions and use entry.URL instead of entry.Url

It's not necessary to have a router wait for a response from a broker for more than RECEIVE_DELAY_2, so we should define a reasonable timeout for these connections.

This is not good enough:

In most cases a Gateway is connected to one Router. This means that the Router should calculate duty cycle information (based on Datr and Size from RXPK)

Define handler adapters

- [ ] Registration
     - [ ] Pull registrations from MQTT bloker
     - [ ] For each registration, send a new request to brokers

- [ ] App Adapter
     - [ ] Connect to an MQTT broker using root credentials
     - [ ] On `Send` publish the formatted packet to a dedicated topic
     - [ ] Watch to MQTT registration and forward them to `NextRegistration`

Each Docker container needs to be created, using the existing library and for now, with a basic logger as an error handling system.

This issue keeps track of the progress on the Basic Broker

• Core
  • Lookup for associated device from Network Controller
  • Send acknowledgement or rejection for given packet
  • If necessary, forward packet to right handler
  • Associate handlers to static device addresses
  • Switch from local in-memory storage to Reddis - See #11
• Router adapter
  • Listen to http request and forward valid request to core
  • Respond with 200 OK (ack) or 404 Not Found (nack)
• Handler adapter
  • Send packet to handler
  • Accept registrations of static devAddr from handlers

It should be possible to configure the path to the {router,broker,handler}_storage.db files.

The HTTP and UDP adapters handles don't return any feedback during their processing. Because they don't have access to the logging context neither, there's currently no more feedback from them.

This can be fixed by introducing an error as a return param (as it has been done in the MQTT adapter handlers) to then, log this error from the adapter which has access to the context.

Not a huge thing yet still has to be done.

In the basic http adapter, the context logger Ctx is declared as an exported field. It shouldn't. We'll have concurrent access on that attribute and we need to prevent the outside world to interact with this context.

It might be possible to just declare it as an anonymous field and then aggregate the component behavior to the adapter and other sub adapters.

possible formatting directive in Log call
Refactoring Log to Logf should fix it for now. Migrating to logrus like I suggested in #5 should also fix this.

Related to #7.
Needed for #8.

Semtech package tests suffer from an early clumsy development :|
Tests are working, but they aren't neither readable nor maintainable. Using metadata tests as an inspiration might be a good idea.

This issue keeps track of the progress on the Gateway Simulator

• Types, packages and data structures in use
• Emit udp packets towards a server
• Handle behavior described by the semtech protocol
• Serialize json rxpk/stat object(s)
• Generate json rxpl/stat object(s)
• Update gateway statistics accordingly
• Generate valid fake end-device packet
• Simulate fake end-devices traffic - See #6
• Complete tests set for stats functions
• Implements UDP ReadWriteCloser

The HandleUp and Register methods have to be tested for both component.

Situation

• When HTTP adapter sends to a number of recipients
• And one of those always returns 502 (for example)
• adapter.go:67 adds an error and adapter.go:93 returns an error, even when another server returned a normal response

Refs

1: adapter.go:67
2: adapter.go:93

We have to figure out one (or several) mechanism(s) to properly handle errors. So far, errors are just bunch of strings message with an arguable value. The library-like code itself shouldn't take care that much of these errors but should generate detailed enough one so that the integration part is able to recover from them whenever this is possible.

We could get some inspiration of well-known go repositories.

The stat field from PUSH_DATA contains valuable information from the gateway, so the router should process it and do "something" with it.

The handler maintains an in-memory map of last processed node (It associates a frame counter to a devEUI + appEUI).
This is required to make sure it doesn't handle twice an already processed packet (The handler buffers for a while packets, but then, process all received packets associated to a same FCnt and should just ignore any new one).

Even though it only keeps in memory the last frame counter registered, there is likely to be a memory-leak here: the map grows indefinitely as long as the handler is running. We should define a clever data structure that could flush entries automatically after a while.

This issue keeps track of the progress on the Basic Router

• Core
  • Lookup for device address (only local)
  • Invalidate broker periodically (only local)
  • Acknowledge packet from gateway
  • Forward packet to brokers
  • Create Mock router
  • Create Mock DownAdapter
  • Create Mock UpAdatper
  • Test'em all
  • Switch from local in-memory storage to Reddis - See #11
• UpAdapter
  • Listen and forward incoming packets to Core router
  • Keep track of existing UDP connections
  • Send ack through existing UDP connection
• DownAdapter
  • Listen and forward incoming packet to Core router
  • Broadcast a packet to several brokers
  • Send packet to given brokers (same as above ?)