st-one-io / node-red-contrib-safe-queue Goto Github PK

Remove:

• node_modules
• temp

Also improve .gitignore file, here an example:

# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov

# Coverage directory used by tools like istanbul
coverage

# nyc test coverage
.nyc_output

# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
.grunt

# Bower dependency directory (https://bower.io/)
bower_components

# node-waf configuration
.lock-wscript

# Compiled binary addons (http://nodejs.org/api/addons.html)
build/Release

# Dependency directories
node_modules/
jspm_packages/

# Typescript v1 declaration files
typings/

# Optional npm cache directory
.npm

# Optional eslint cache
.eslintcache

# Optional REPL history
.node_repl_history

# Output of 'npm pack'
*.tgz

# Yarn Integrity file
.yarn-integrity

# dotenv environment variables file
.env

# Visual Studio Code config folder
.vscode

• Add a days parameter, so it deletes only when the messages that are older than days days. (Check https://momentjs.com/ for implementation)
• Use only async methods
• Reuse code shared among the functions

• change getListFiles() to `getMessageList()'
• call the callback in all cases of errors, except when moving an incompatible file to error

Estudar uma solução para arquivos de mesmo nome, nos métodos saveMessage(), doneMessage() e errorMessage().

Change the node colors to the same color (maybe #FFAAAA)?

Test integration lib on Linux

When calling the queue control node to get the size of the queue, done or error folders, it throws an error if the directories have been deleted

Steps to reproduce

1. Create a queue
2. Call the queue size. It will return the correct size.
3. Delete the queue folder
4. Call the queue size.

Expected behavior
The node should return size zero, as there's no item in the queue (the folder doesn't even exists)

Current behavior
The following errors are thrown:

16 Feb 13:22:59 - [error] [queue control:bb0975b7.473118] Error: ENOENT: no such file or directory, scandir '/home/guilherme/test/red/json_events/queue'
16 Feb 13:22:59 - [error] [queue control:eca174d7.e87f18] Error: ENOENT: no such file or directory, scandir '/home/guilherme/test/red/json_events/done'
16 Feb 13:22:59 - [error] [queue control:a5c1b956.8c2e08] Error: ENOENT: no such file or directory, scandir '/home/guilherme/test/red/json_events/error'

• Implement a days parameter, to resend only messages that are newer that days days
• Use only async calls
• Verify if directories are indeed directories before calling readdir() on them

Check on createWatch() if there's already a watcher before creating a new one, and close it if it exists. Also prevent calling this.watcher.close() on close() if watcher has already been closed.

Application tests implementation:

• Test of delete files.
• Test of directory size.
• Test if directory exists.
• Test create and move files.

• Queue config
  • Choose better icons
  • Use a spinner for the timeout
  • Add time unit for timeout (ms)
  • Add a description for the configuration options
• Queue control
  • Set a default operation

getMessage() can throw if JSON.parse throws an exception. This should be handled internally

Remove incompatibleFile(), treat them as a normal error an move to the error folder

Try if it works: Try writing the file, and if it fails because the directory does not exits, try creating the directory and try to write the file again.

Also check for using 'rs+' flag when writing the file.

Add a README.md with basic infos, usage, install, etc. Check other projects for examples.

Function should check for config node before calling function. Code suggestion:

function () {

    if (this.name) {
        return this.name;
    }

    var name = this._("safe-queue.in.define.name")
    var selectConfig = RED.nodes.node(this.config);

    if (selectConfig) {
        name += ": " + selectConfig.label();
    }

    return name;
}

After hitting "deploy" while safe queue was processing a message this error message comes

The config node could check the path of other config nodes for conflict, and display a warning if that happens

We'll use Apache License,Version 2.0, January 2004

• Add LICENSE file to project's root folder
• Check all file's header
• Check package.json

Determinar valores padrão e campos obrigatórios para o nó SafeQueue Config.

Campos Obrigatórios:

• Storage
• Path
• Timeout ACK

Valores padrão:

• Storage : 'fs' FileSystem
• Path : \Users\userName
• Timeout ACK : 1000 microssegundos.

A value of 0 for the timeout should disable it. Also describe the functionality on the info pane

Getting the below error from a "queue out" node uppon receiving a msg from an HTTP GET end-point with following payload. If injecting the JSON directly no error occur, so it may be the msg.req/msg.res objects.

Payload:
{"device_type":"PMS-100-Thermo-Pulse","device":"43ED78","time":"1538630900","data":"1d24162cb4e28900","seq_number":"1135"}

Error:
{"message":"TypeError: Converting circular structure to JSON","source":{"id":"86047baa.8778a8","type":"queue in","count":1},"stack":"TypeError: Converting circular structure to JSON\n at JSON.stringify (<anonymous>)\n at FileSystem.saveMessage (C:\\Users\\Administrator\\.node-red\\node_modules\\node-red-contrib-safe-queue\\src\\FileSystem.js:378:36)\n at SafeQueueConfig.receiveMessage (C:\\Users\\Administrator\\.node-red\\node_modules\\node-red-contrib-safe-queue\\red\\safeQueue.js:219:26)\n at SafeQueueIn.<anonymous> (C:\\Users\\Administrator\\.node-red\\node_modules\\node-red-contrib-safe-queue\\red\\safeQueue.js:515:25)\n at emitOne (events.js:116:13)\n at SafeQueueIn.emit (events.js:211:7)\n at SafeQueueIn.Node.receive (C:\\Users\\Administrator\\AppData\\Roaming\\npm\\node_modules\\node-red\\red\\runtime\\nodes\\Node.js:215:14)\n at FunctionNode.Node.send (C:\\Users\\Administrator\\AppData\\Roaming\\npm\\node_modules\\node-red\\red\\runtime\\nodes\\Node.js:202:14)\n at sendResults (C:\\Users\\Administrator\\AppData\\Roaming\\npm\\node_modules\\node-red\\nodes\\core\\core\\80-function.js:52:18)\n at Object.send (C:\\Users\\Administrator\\AppData\\Roaming\\npm\\node_modules\\node-red\\nodes\\core\\core\\80-function.js:108:21)"}

msg.req:
{"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":null,"tail":null,"length":0},"length":0,"pipes":null,"pipesCount":0,"flowing":null,"ended":false,"endEmitted":false,"reading":false,"sync":true,"needReadable":false,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"destroyed":false,"defaultEncoding":"utf8","awaitDrain":0,"readingMore":true,"decoder":null,"encoding":null},"readable":true,"domain":null,"_events":{},"_eventsCount":0,"socket":"[internal]","connection":"[internal]","httpVersionMajor":1,"httpVersionMinor":1,"httpVersion":"1.1","complete":false,"headers":{"host":"sigfox.ivigilant.com.br:80","accept-language":"fr","accept-encoding":"gzip,deflate","user-agent":"SIGFOX","accept-charset":"UTF-8;q=0.9,*;q=0.7"},"rawHeaders":["host","sigfox.ivigilant.com.br:80","accept-language","fr","accept-encoding","gzip,deflate","user-agent","SIGFOX","accept-charset","UTF-8;q=0.9,*;q=0.7"],"trailers":{},"rawTrailers":[],"upgrade":false,"url":"/sigfox?device_type=PMS-100-Thermo-Pulse&device=43ED78&time=1538630900&data=1d24162cb4e28900&seq_number=1135","method":"GET","statusCode":null,"statusMessage":null,"client":"[internal]","_consuming":false,"_dumped":false,"baseUrl":"","originalUrl":"/sigfox?device_type=PMS-100-Thermo-Pulse&device=43ED78&time=1538630900&data=1d24162cb4e28900&seq_number=1135","_parsedUrl":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?device_type=PMS-100-Thermo-Pulse&device=43ED78&time=1538630900&data=1d24162cb4e28900&seq_number=1135","query":"device_type=PMS-100-Thermo-Pulse&device=43ED78&time=1538630900&data=1d24162cb4e28900&seq_number=1135","pathname":"/sigfox","path":"/sigfox?device_type=PMS-100-Thermo-Pulse&device=43ED78&time=1538630900&data=1d24162cb4e28900&seq_number=1135","href":"/sigfox?device_type=PMS-100-Thermo-Pulse&device=43ED78&time=1538630900&data=1d24162cb4e28900&seq_number=1135","_raw":"/sigfox?device_type=PMS-100-Thermo-Pulse&device=43ED78&time=1538630900&data=1d24162cb4e28900&seq_number=1135"},"params":{},"query":{"device_type":"PMS-100-Thermo-Pulse","device":"43ED78","time":"1538630900","data":"1d24162cb4e28900","seq_number":"1135"},"res":"[internal]","route":{"path":"/sigfox","stack":[{"name":"cookieParser","keys":[],"regexp":{"fast_star":false,"fast_slash":false},"method":"get"},{"name":"httpMiddleware","keys":[],"regexp":{"fast_star":false,"fast_slash":false},"method":"get"},{"name":"corsHandler","keys":[],"regexp":{"fast_star":false,"fast_slash":false},"method":"get"},{"name":"metricsHandler","keys":[],"regexp":{"fast_star":false,"fast_slash":false},"method":"get"},{"name":"<anonymous>","keys":[],"regexp":{"fast_star":false,"fast_slash":false},"method":"get"},{"name":"<anonymous>","keys":[],"regexp":{"fast_star":false,"fast_slash":false},"method":"get"}],"methods":{"get":true}},"cookies":{},"signedCookies":{}}

1. Create a method that count files and checks the file extensions
2. Generalize a method for counting files in a given folder, with option to check subfolders
3. Change getQueueSize, getDoneSize and getErrorSize to use them

• Default directory should be the current running directory (a.k.a. `process.cwd()')
• The path of the queue should not consider the queue's name
• Handle case where FileSystem option's object is null or has a null path

Functions implementation:

• deleteQueue();
• deleteDone();
• deleteError();
• resendErrors();

• Criar flag no SafeQueue Config start job at start?

• queue ack: icon at right side
• queue out: remove input

Try using "mkdirp" on init() of FileSystem, so we don't need to check all the folders individually.

Function implementation - SafeQueue Control - Stop outputs

• Create function to stop outputs.

When the "queue" folder is deleted an uncaught exception is thrown and Node-red crashes

Reuse code that is the same for both. Also extracts a function for getting the time stamp string

Emit an error if the storage mode is not one of the known modes and return from creating the node

Remover implementação de path default do safeQueue.js e fazer a implementação no FileSystem.js.

Use the format <queue name>:<action>, or only <action> when there's no queue configured for all nodes.

Implement negative acknowledges. Some ideas:

• Define a parameter to be checked for positive or negative acknowledges
• Add a checkbox to invert the evaluation of the acknowledge (e.g. having a msg.payload may be good, or having msg.error is bad)

Implementar teste para novos métodos.

• resendErrors().
• deleteDone().
• deleteError().

It should be "ack" instead of "out"

I have a queue with "Start job at start?" unchecked, so that I have to start the queue manually. But after hitting the deploy button on Node-RED, I try to call "start-process", and then I get a "Node out in process" message in the debug tab, and therefore the queue processing doesn't start.

It starts only if I hit stop, and then start again.

Here is my flow

[{"id":"de6f7019.8185f","type":"tab","label":"Flow 1"},{"id":"480714f1.31370c","type":"file in","z":"de6f7019.8185f","name":"","filename":"json-events_122_2018-02-06.log","format":"lines","chunk":false,"sendError":false,"x":440,"y":140,"wires":[["45e20a29.e3caa4"]]},{"id":"620ec8a9.b00228","type":"inject","z":"de6f7019.8185f","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"x":170,"y":140,"wires":[["480714f1.31370c"]]},{"id":"45e20a29.e3caa4","type":"queue in","z":"de6f7019.8185f","name":"","config":"3a96eacd.189176","sendError":true,"x":760,"y":140,"wires":[[]]},{"id":"3db24970.17e1c6","type":"queue control","z":"de6f7019.8185f","name":"","config":"3a96eacd.189176","operation":"stop-process","days":0,"x":350,"y":440,"wires":[[]]},{"id":"297008ec.a19f48","type":"inject","z":"de6f7019.8185f","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":true,"x":130,"y":440,"wires":[["3db24970.17e1c6"]]},{"id":"85d8cbb7.460548","type":"queue control","z":"de6f7019.8185f","name":"","config":"3a96eacd.189176","operation":"start-process","days":0,"x":350,"y":500,"wires":[[]]},{"id":"bbdf9a5f.6ebfd8","type":"inject","z":"de6f7019.8185f","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"x":130,"y":500,"wires":[["85d8cbb7.460548"]]},{"id":"bb0975b7.473118","type":"queue control","z":"de6f7019.8185f","name":"","config":"3a96eacd.189176","operation":"queue-size","days":0,"x":350,"y":560,"wires":[[]]},{"id":"2166bd44.055892","type":"inject","z":"de6f7019.8185f","name":"","topic":"","payload":"","payloadType":"str","repeat":"2","crontab":"","once":false,"x":130,"y":560,"wires":[["bb0975b7.473118","eca174d7.e87f18","a5c1b956.8c2e08"]]},{"id":"eca174d7.e87f18","type":"queue control","z":"de6f7019.8185f","name":"","config":"3a96eacd.189176","operation":"done-size","days":0,"x":340,"y":620,"wires":[[]]},{"id":"a5c1b956.8c2e08","type":"queue control","z":"de6f7019.8185f","name":"","config":"3a96eacd.189176","operation":"error-size","days":0,"x":340,"y":680,"wires":[[]]},{"id":"49baf4cf.8a735c","type":"json","z":"de6f7019.8185f","name":"","pretty":false,"x":310,"y":240,"wires":[["bfe745e9.0b6be8"]]},{"id":"26665d1f.1603c2","type":"switch","z":"de6f7019.8185f","name":"","property":"payload.topic","propertyType":"msg","rules":[{"t":"regex","v":"^AG.*$","vt":"str","case":true},{"t":"else"}],"checkall":"true","outputs":2,"x":630,"y":240,"wires":[["de50126f.61b88"],["5c18df99.e2cf6"]]},{"id":"de50126f.61b88","type":"change","z":"de6f7019.8185f","name":"format influx","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.(\t    $topic := $split(topic, \"_\");\t    $loc := $topic[0];\t    $type := $topic[3];\t    $sa := $topic[4];\t    $pos := $topic[5];\t    \t    $tags := \"events,locale=\" & $loc & \",type=\" & $type & \",sa=\" & $sa;\t    $fields := \"position=\" & $pos & \",value=\" & payload;\t    \t    $tags & \" \" & $fields & \" \" & ts\t)","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":810,"y":220,"wires":[["f7fea7be.bf9088","e6f7a7f.67db958"]]},{"id":"bfe745e9.0b6be8","type":"function","z":"de6f7019.8185f","name":"timeToMillis","func":"if(msg && msg.payload && msg.payload.ts){\n    msg.payload.ts = new Date(msg.payload.ts).getTime()\n}\nreturn msg;","outputs":1,"noerr":0,"x":470,"y":240,"wires":[["26665d1f.1603c2"]]},{"id":"f7fea7be.bf9088","type":"http request","z":"de6f7019.8185f","name":"","method":"POST","ret":"txt","url":"http://127.0.0.1:8086/write?db=andon_events&precision=ms","tls":"","x":990,"y":240,"wires":[["a8fd00c2.c1f4d","7e7b580e.105198"]]},{"id":"1d2a9d9a.41c632","type":"queue out","z":"de6f7019.8185f","name":"","config":"3a96eacd.189176","x":120,"y":240,"wires":[["49baf4cf.8a735c","ce96dfa5.0a83"]]},{"id":"a8fd00c2.c1f4d","type":"queue ack","z":"de6f7019.8185f","name":"","config":"3a96eacd.189176","x":1210,"y":240,"wires":[]},{"id":"e6f7a7f.67db958","type":"debug","z":"de6f7019.8185f","name":"","active":false,"console":"false","complete":"true","x":970,"y":180,"wires":[]},{"id":"d746b374.66403","type":"queue control","z":"de6f7019.8185f","name":"","config":"3a96eacd.189176","operation":"resend-errors","days":0,"x":360,"y":760,"wires":[[]]},{"id":"e91b0764.b334c8","type":"inject","z":"de6f7019.8185f","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"x":130,"y":760,"wires":[["d746b374.66403"]]},{"id":"ce96dfa5.0a83","type":"debug","z":"de6f7019.8185f","name":"","active":false,"console":"false","complete":"true","x":310,"y":280,"wires":[]},{"id":"7e7b580e.105198","type":"debug","z":"de6f7019.8185f","name":"","active":false,"console":"false","complete":"true","x":1150,"y":300,"wires":[]},{"id":"5c18df99.e2cf6","type":"change","z":"de6f7019.8185f","name":"format influx","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.(\t    \"stats,type=\" & topic & \" value=\" & payload & \" \" & ts\t)","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":810,"y":260,"wires":[["f7fea7be.bf9088","4464647f.671c8c"]]},{"id":"4464647f.671c8c","type":"debug","z":"de6f7019.8185f","name":"","active":false,"console":"false","complete":"true","x":970,"y":300,"wires":[]},{"id":"3a96eacd.189176","type":"queue config","z":"de6f7019.8185f","name":"","storage":"fs","path":"json_events","timeoutAck":"10000","startJob":false,"typeTimeout":"move-error","typeError":"move-error","retryTimeout":"","retryError":"","maxInMemory":"10000"}]

Can you check this out?

chai should be on devDependencies section of package.json

Saving files to disk is pretty straightforward, but in the case there are many small messages, storage usage becomes very inneficient. In this case, it's wasting space by an order of magnitute:

An option would be to use a serverless (embedded) database to store the messages, like SQLite or NeDB. The latter is an interesting option, as it's more nodejs friendly (doesn't require compiling anything) and the data is still readable with a simple text editor.

Some aspects need some deeper analysis though, mainly regarding consistency, data corruption possibility/probability, and performance

Remove all unnedded oneditprepare functions. Currently only needed on queue config. Also remove the code for a "default option" there

Fail on integration with Linux. [ Issues #4 ]

• The library need correction on control of files.

Chage category from SafeQueue to safe queue, in alignment with the other existing categories

Create labels for nodes and position descriptions.

Nodes: SafeQueue Config, SafeQueue Control, SafeQueue In, SafeQueue Out and SafeQueue ACK.

Function implementation - SafeQueue Control - Clean outputs

• Create function to clean outputs

When unchecked option Start job at start?.

• The system stop outputs. ok.

But when arrive new message on input, the system start outputs. This is not ok.

Rename the newFile event of the storage layer to newMessage, so we have a more concise name when working with databases in the futue

Use italics for the label style when a user-defined name is configured. Check https://nodered.org/docs/creating-nodes/appearance for an example