Code Monkey home page Code Monkey logo

recsync's Introduction

RecSync

The record synchronizer project includes two parts. A client (RecCaster) which runing as part of an EPICS IOC, and a server (RecCeiver) which is a stand alone daemon. Together they work to ensure the the server(s) have a complete list of all records currently provided by the client IOCs.

Information Uploaded

The RecCaster client currently sends the following information about its Process Database.

  • The EPICS Base Version
  • A white listed set of environment variables
  • The name, type, and description of all records
  • Any info() tags associated with these records
  • Any additional environment variables specified in addReccasterEnvVars iocsh calls

RecCaster Usage

The RecCaster source in the client/ sub-directory is build as an EPICS support module. It provides reccaster.dbd and the reccaster library. The client demo IOC in client/demoApp/ provides an example of how to build RecCaster into an IOC.

RecCaster requires only EPICS Base 3.15 and later 3.14 releases.

RecCeiver Usage

The RecCeiver server in the server/ directory is a Python script using the Twisted networking library. It requires Python 2.7 and Twisted >= 12.0.

The server uses the Twisted plugin interface to make client information available to one or more plugins. See server/demo.conf for an example configuration.

Currently two plugins are provided: show which print client information to screen/log, and db which writes into a SQL database (currently only sqlite3 supported). The SQL table schema used is defined in server/recceiver.sqlite3.

Theory of Operation

RecCaster

The RecCaster client is an epics support module which automatically starts a single thread and begins passively waiting for a RecCeiver server to announce its presence.

Upon receiving an announcement the client will connect and upload its list of records and related information to the server. It then remains connected indefinately and responds to periodic alive tests from the server.

If this connection is lost the client will resume waiting for another announcement.

RecCeiver

The RecCeiver server periodially (every 15 seconds) sends a broadcast announcement of it existance.

When a client connects it will wait for a server handshake message before beginning to upload records. The server will delay sending this message if too many clients are currently uploading.

UDP Protocol

Server announcement are made to UDP port 5049.

When sent to a IPv4 address, such messages will have a length >= 16 bytes. Clients will process only the first 16 bytes and ignore any remaining bytes.

0 1 2 3 4 5 6 7 8 9 A B C D E F
ID 0 X SERV ADDR PORT X X SERV KEY
  • All multibyte fields have big-endian ordering.
  • Fields marked 0 must be zero and clients will (silently) reject messages where these fields are not zero.
  • Fields marked X should be ignored by clients.
  • The ID field must have the value 0x5243 (ascii 'RC').
  • The SERV ADDR and PORT fields is the IPv4 address and TCP port to which clients should connect.
  • SERV KEY is an arbitrary number which the client will later on the TCP connection.

TCP Protocol

The TCP data stream will consist of a sequence of messages having an 8 byte header followed by a variable length body. The header will have the form

0 1 2 3 4 5 6 7
ID MSGID LEN
  • All multibyte fields have big-endian ordering.
  • The ID field must have the value 0x5243 (ascii 'RC').
  • The MSGID field identifies the type of message to follow.
  • The LEN field gives the length in bytes or by message body to follow the header.
  • Messages from server to client will have MSGID >= 0x8000.
  • Messages from client to server will have MSGID < 0x8000.
  • Except where explicitly stated, endpoints must silently ignore messages with unknown MSGID
  • Endpoints must be prepared to receive and ignore extra body bytes.

Messages

MSGID Min. LEN Name 0 1 2 3 4 5 6 7
0x8001 1 Server Greet 0
0x8002 4 Ping NONCE
0x0001 8 Client Greet 0 0 X X SERV KEY
0x0002 4 Pong NONCE
0x0003 9 Add Record RECID ATYPE RTLEN RNLEN RTYPE RNAME
0x0004 4 Del Record RECID
0x0005 4 Upload Done 0 0 0 0
0x0006 9 Add Info RECID KEYLEN X VALEN KEY VALUE
  • NONCE is an number choosen by the server for a Ping message. It must be echoed back by the client in a Pong message.
  • SERV KEY must be the number the client received in the announcement from which it found this server.
  • RECID is an identifier choosen by the client to identify a single record instance. Except in Add Info, must be >0.
  • ATYPE is 0 to add a record entry or 1 to add a record alias. When adding an alias the record type is omitted (RTLEN==0) and the RECID must have been added in a previous message with ATYPE==0.
  • RTLEN, RNLEN, KEYLEN, and VALEN are lengths in bytes.
  • RTLEN and VALEN are allowed to be zero.
  • RNLEN and KEYLEN must be greater than zero.
  • RTYPE, RNAME, KEY, and VALUE are variable length fields whos length is given by the corresponding *LEN field.
  • RTYPE, RNAME, KEY, and VALUE should not include a 0 (ascii null) byte.
  • The RECID field of the Add Info may be zero. When zero the key/value pair is considered to be associated with the client as a whole, and not any individual record.

When a TCP connection is established the client must send the Client Greeting message immediately, then wait to receive the Server Greeting message before sending any further messages.

Once the client has recevieved the Server Greeting it may send begin sending Add Record, Add Info, and Del Record messages. Once the initial upload has been completed, the client must send Upload Done.

After Upload Done is received the server will periodically send Ping messages. The client must respond to each Ping with a Pong. If a client does not respond promptly the server may close the connection.

recsync's People

Contributors

berryma4 avatar dchabot avatar dxmaxwell avatar freddieakeroyd avatar jacomago avatar jeonghanlee avatar kingspride avatar mark0n avatar mdavidsaver avatar mskinner5278 avatar nicdevel avatar shengb avatar shroffk avatar simon-ess avatar tanviash avatar tynanford avatar waynelewis avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

recsync's Issues

supporting more properties via cfstore

Currently cfstore supports
hostName
iocName
time
pvStatus

upgrade 1.
The python channlefinder library already consists of a python script which populates properties by parsing pv names using regular expressions defined in a configuration file.

upgrade 2.
the cfstore plugin should parse the info associated the record itself and attach the appropriate tags/properties and properties. Thus ioc engineer can control the population of information into channelfinder from the ioc configuration files.

self._poller.unregister IOError: [Errno 2] No such file or directory

So I finally realized what is happening here. The issue is that I am trying to be robust by preventing recvfrom() on the broadcasting socket. I do this by calling pauseProducing() for the Transport associated with a UDP socket. Apparently this doesn't work work with the epoll backend.

Traceback (most recent call last):
  File "/usr/bin/twistd", line 11, in <module>
    load_entry_point('Twisted==18.9.0', 'console_scripts', 'twistd')()
  File "/usr/lib/python2.7/dist-packages/twisted/scripts/twistd.py", line 31, in run
    app.run(runApp, ServerOptions)
  File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 674, in run
    runApp(config)
  File "/usr/lib/python2.7/dist-packages/twisted/scripts/twistd.py", line 25, in runApp
    runner.run()
  File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 385, in run
    self.postApplication()
  File "/usr/lib/python2.7/dist-packages/twisted/scripts/_twistd_unix.py", line 254, in postApplication
    self.startApplication(self.application)
  File "/usr/lib/python2.7/dist-packages/twisted/scripts/_twistd_unix.py", line 442, in startApplication
    service.IService(application).privilegedStartService()
  File "/usr/lib/python2.7/dist-packages/twisted/application/service.py", line 283, in privilegedStartService
    service.privilegedStartService()
  File "/home/mdavidsaver/work/epics/recsync/server/recceiver/application.py", line 86, in privilegedStartService
    self.udp.startListening()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/udp.py", line 179, in startListening
    self._connectToProtocol()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/udp.py", line 217, in _connectToProtocol
    self.protocol.makeConnection(self)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/protocol.py", line 771, in makeConnection
    self.doStart()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/protocol.py", line 727, in doStart
    self.startProtocol()
  File "/home/mdavidsaver/work/epics/recsync/server/recceiver/announce.py", line 40, in startProtocol
    self.transport.pauseProducing()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/abstract.py", line 470, in pauseProducing
    self.stopReading()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/abstract.py", line 422, in stopReading
    self.reactor.removeReader(self)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/epollreactor.py", line 172, in removeReader
    EPOLLIN, EPOLLOUT)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/epollreactor.py", line 160, in _remove
    self._poller.unregister(fd)
IOError: [Errno 2] No such file or directory

building on windows

Hello,
when building on windows with MSVC and under EPICS 7, I end up with a bunch of errors ...

is this a known phenomenon or is reccaster client incompatible with windows somehow?

[...]
cl               -nologo -FC -D__STDC__=0 -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE    -Ox -GL -Oy-   -W3        -MD -DEPICS_BUILD_DLL -DEPICS_CALL_DLL    -I. -I../O.Common -I. -I. -I.. -I../../../include/compiler/msvc -I../../../include/os/WIN32 -I../../../include      -IC:/EPICS/epics-support/ipac/include   -IC:/EPICS/epics-support/sscan/include   -IC:/EPICS/epics-support/seq-2.2.8/include   -IC:/EPICS/epics-support/calc/include  -IC:/EPICS/epics-support/autosave/include/os/WIN32 -IC:/EPICS/epics-support/autosave/include   -IC:/EPICS/epics-support/busy/include      -IC:/EPICS/epics-support/asyn/include   -IC:/EPICS/epics-support/motor/include   -IC:/EPICS/epics-support/modbus/include   -IC:/EPICS/epics-support/devlib2/include   -IC:/EPICS/epics-support/StreamDevice/include       -IC:/EPICS/epics-base/include/compiler/msvc -IC:/EPICS/epics-base/include/os/WIN32 -IC:/EPICS/epics-base/include        -c ../sockhelpers.c
sockhelpers.c
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.h(63): error C2061: syntax error: identifier 'shRecvExact'C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.h(63): error C2059: syntax error: ';'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.h(63): error C2059: syntax error: '<parameter-list>'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.h(67): error C2061: syntax error: identifier 'shRecvIgnore'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.h(67): error C2059: syntax error: ';'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.h(67): error C2059: syntax error: '<parameter-list>'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.h(70): error C2061: syntax error: identifier 'shRecvFrom'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.h(70): error C2059: syntax error: ';'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.h(70): error C2059: syntax error: '<parameter-list>'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(100): warning C4244: 'function': conversion from 'SOCKET' to 'int', possible loss of data
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(173): warning C4244: '=': conversion from 'SOCKET' to 'int', possible loss of data
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(161): warning C4244: 'initializing': conversion from 'SOCKET' to 'int', possible loss of data
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(225): error C2061: syntax error: identifier 'shRecvExact'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(225): error C2059: syntax error: ';'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(225): error C2059: syntax error: '<parameter-list>'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(246): error C2061: syntax error: identifier 'shRecvIgnore'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(246): error C2059: syntax error: ';'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(246): error C2059: syntax error: '<parameter-list>'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(267): error C2061: syntax error: identifier 'shRecvFrom'C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(267): error C2059: syntax error: ';'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(267): error C2059: syntax error: '<parameter-list>'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(297): error C2065: 'ssize_t': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(297): error C2146: syntax error: missing ';' before identifier 'ret'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(297): error C2065: 'ret': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(303): error C2065: 'ret': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(303): warning C4267: 'function': conversion from 'size_t' to 'int', possible loss of data
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(304): error C2065: 'ret': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(306): error C2065: 'ret': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(311): error C2065: 'ssize_t': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(311): error C2146: syntax error: missing ';' before identifier 'ret'
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(311): error C2065: 'ret': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(320): error C2065: 'ret': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(320): warning C4267: 'function': conversion from 'size_t' to 'int', possible loss of data
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(321): error C2065: 'ret': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(323): error C2065: 'ret': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(324): error C2065: 'ret': undeclared identifier
C:\EPICS\epics-support\recsync\client\castApp\src\sockhelpers.c(326): error C2065: 'ret': undeclared identifier
make[3]: *** [C:/EPICS/epics-base/configure/RULES_BUILD:259: sockhelpers.obj] Error 2
make[3]: Leaving directory 'C:/EPICS/epics-support/recsync/client/castApp/src/O.windows-x64'
make[2]: *** [C:/EPICS/epics-base/configure/RULES_ARCHS:58: install.windows-x64] Error 2
make[2]: Leaving directory 'C:/EPICS/epics-support/recsync/client/castApp/src'
make[1]: *** [C:/EPICS/epics-base/configure/RULES_DIRS:85: src.install] Error 2
make[1]: Leaving directory 'C:/EPICS/epics-support/recsync/client/castApp'
make: *** [C:/EPICS/epics-base/configure/RULES_DIRS:85: castApp.install] Error 2

C:\EPICS\epics-support\recsync\client>

thanks,
W.

permissions issues

When using RecCeiver which was installed as a Debian (7) package, launching sudo /etc/init.d/recceiver start results in .pid file being created in /var/run/. Then, sudo /etc/init.d/recceiver stop causes Warning: No permission to delete pid file warning. Modifying the init.d script to create the .pid file in a directory writable by recceiver fixes the issue and the file is deleted properly when the service is stopped.

A similar problem appears when trying to make RecCeiver use an sqlite database file. Even if the database file is owned by recceiver, an sqlite.OperationalError: unable to open database file is thrown by the code once a client connects. Placing the database file into a folder writable by recceiver fixes the issue, presumably because "SQLite needs to be able to create additional files in it in order to handle things like the commit log".

No transmission if info tag string is empty

Trying to send an infotag of the form, info(archive, ""), results in the IOC-side dropping transmission of the infotag and its null contents.

See here

Mistake? Or is there some reasoning behind this?
(Please don't say optimization ;-) )

attn: @Dylan171

cfstore plugin requirements

We have been working on developing a plugin for recsync which pipes the data to channelfinder.

Currently the features supported are

  1. which ioc startup, all the records are added to channelfinder along with properties iocName, hostName, time, and ovStatus
  2. upon ioc reboot, the "time"

Missing
When an IOC shuts down the information available in the transaction object consists of only the host ip (as part of the TR.src). Since there might be more than one ioc running on a particular host, the IOCNAME information is necessary to update channelfinder correctly.

How to start RecCeiver on a specified port

[ioc@localhost server]$ PYTHONPATH=$PWD twistd -r poll -n recceiver -f demo.conf
2019-04-14T20:14:05+0800 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 18.9.0 (/irfel/Phoebus/anaconda3/envs/recsync/bin/python 3.6.8) starting up.
2019-04-14T20:14:05+0800 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.pollreactor.PollReactor.
2019-04-14T20:14:05+0800 [stdout#info] Starting
2019-04-14T20:14:05+0800 [-] CastFactory starting on 38000
2019-04-14T20:14:05+0800 [recceiver.recast.CastFactory#info] Starting factory <recceiver.recast.CastFactory object at 0x7f8903c93ac8>
2019-04-14T20:14:05+0800 [-] CastFactory starting on 40943
2019-04-14T20:14:05+0800 [stdout#info] listening on IPv4Address(type='TCP', host='0.0.0.0', port=40943)
2019-04-14T20:14:05+0800 [-] Announcer starting on 55223
2019-04-14T20:14:05+0800 [-] Starting protocol <recceiver.announce.Announcer object at 0x7f8903ca1240>

I am using iptables for firewall configuration. According to the above information, my firewall configuration is as follows.

-A INPUT -m state --state NEW -m udp -p udp --dport 5049 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --sport 5049 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5049 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --sport 5049 -j ACCEPT

-A INPUT -m state --state NEW -m udp -p udp --dport 40943 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --sport 40943 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 40943 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --sport 40943 -j ACCEPT

-A INPUT -m state --state NEW -m udp -p udp --dport 55223 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --sport 55223 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 55223 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --sport 55223 -j ACCEPT

Now RecSync can work normally. But after I restarted RecCeiver, RecSync will not work normally because the port has changed. I want to know how to start RecCeiver on a specified port.

ENH: confine reccaster broadcast reception interface

Use environment variables RECCASTER_BCAST_ADDR and EPICS_CA_ADDR_LIST to confine the interface that reccaster (client) listens for recceiver (server) UDP broadcasts on.

Prefer RECCASTER_BCAST_ADDR over EPICS_CA_ADDR_LIST when both are present.

RecCeiver Service Unhandled Error

The RecCeiver service crashes after a few days of continuous operation with the following exception:

2016-12-30 03:51:07-0500 [-] Unhandled Error
Traceback (most recent call last):
 File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 392, in startReactor
   self.config, oldstdout, oldstderr, self.profiler, reactor)
 File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 313, in runReactorWithLogging
   reactor.run()
 File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
   self.mainLoop()
 File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
   self.runUntilCurrent()
--- <exception caught here> ---
 File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 797, in runUntilCurrent
   f(*a, **kw)
 File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback
   self._startRunCallbacks(result)
 File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 483, in _startRunCallbacks
   raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError: 

@dchabot @berryma4

Channels "time" never update / polling never finishes

Hello,

I have a question regarding the recceivers' behaviour.

see this excerpt from loglevel debug:

2024-04-23T01:15:22+0200 [-] DEBUG:recceiver.cfstore Polling begins...
2024-04-23T01:15:22+0200 [-] DEBUG:recceiver.cfstore Find existing channels by IOCID: 10.0.0.15:34980
2024-04-23T01:15:22+0200 [-] DEBUG:urllib3.connectionpool http://localhost:8080 "GET /ChannelFinder/resources/channels?iocid=10.0.0.15%3A34980 HTTP/1.1" 200 None
2024-04-23T01:15:22+0200 [-] DEBUG:recceiver.cfstore Find existing channels by name: FHIFEL:GagePort:Port1_value|FHIFEL:GagePort:Port2_value|FHIFEL:GagePort:Port1_count|FHIFEL:GagePort:Port2_count
2024-04-23T01:15:22+0200 [-] DEBUG:urllib3.connectionpool http://localhost:8080 "GET /ChannelFinder/resources/channels?~name=FHIFEL%3AGagePort%3APort1_value%7CFHIFEL%3AGagePort%3APort2_value%7CFHIFEL%3AGagePort%3APort1_count%7CFHIFEL%3AGagePort%3APort2_count HTTP/1.1" 200 None
2024-04-23T01:15:22+0200 [-] DEBUG:recceiver.cfstore Add existing channel with different IOC: {'name': 'FHIFEL:GagePort:Port1_value', 'owner': 'admin', 'properties': [{'name': 'hostName', 'owner': 'cfstore', 'value': 'fel03'}, {'name': 'iocName', 'owner': 'cfstore', 'value': 'iocGagePort'}, {'name': 'iocid', 'owner': 'cfstore', 'value': '10.0.0.15:34980'}, {'name': 'pvStatus', 'owner': 'cfstore', 'value': 'Active'}, {'name': 'time', 'owner': 'cfstore', 'value': '2024-04-23 01:15:22.941623'}, {'name': 'recceiverID', 'owner': 'cfstore', 'value': 'fel-recceiver'}, {'name': 'recordType', 'owner': 'cfstore', 'value': 'ai'}], 'tags': []}
2024-04-23T01:15:22+0200 [-] DEBUG:recceiver.cfstore Add existing channel with different IOC: {'name': 'FHIFEL:GagePort:Port2_value', 'owner': 'admin', 'properties': [{'name': 'hostName', 'owner': 'cfstore', 'value': 'fel03'}, {'name': 'iocName', 'owner': 'cfstore', 'value': 'iocGagePort'}, {'name': 'iocid', 'owner': 'cfstore', 'value': '10.0.0.15:34980'}, {'name': 'pvStatus', 'owner': 'cfstore', 'value': 'Active'}, {'name': 'time', 'owner': 'cfstore', 'value': '2024-04-23 01:15:22.941623'}, {'name': 'recceiverID', 'owner': 'cfstore', 'value': 'fel-recceiver'}, {'name': 'recordType', 'owner': 'cfstore', 'value': 'ai'}], 'tags': []}
2024-04-23T01:15:22+0200 [-] DEBUG:recceiver.cfstore Add existing channel with different IOC: {'name': 'FHIFEL:GagePort:Port1_count', 'owner': 'admin', 'properties': [{'name': 'hostName', 'owner': 'cfstore', 'value': 'fel03'}, {'name': 'iocName', 'owner': 'cfstore', 'value': 'iocGagePort'}, {'name': 'iocid', 'owner': 'cfstore', 'value': '10.0.0.15:34980'}, {'name': 'pvStatus', 'owner': 'cfstore', 'value': 'Active'}, {'name': 'time', 'owner': 'cfstore', 'value': '2024-04-23 01:15:22.941623'}, {'name': 'recceiverID', 'owner': 'cfstore', 'value': 'fel-recceiver'}, {'name': 'recordType', 'owner': 'cfstore', 'value': 'ai'}], 'tags': []}
2024-04-23T01:15:22+0200 [-] DEBUG:recceiver.cfstore Add existing channel with different IOC: {'name': 'FHIFEL:GagePort:Port2_count', 'owner': 'admin', 'properties': [{'name': 'hostName', 'owner': 'cfstore', 'value': 'fel03'}, {'name': 'iocName', 'owner': 'cfstore', 'value': 'iocGagePort'}, {'name': 'iocid', 'owner': 'cfstore', 'value': '10.0.0.15:34980'}, {'name': 'pvStatus', 'owner': 'cfstore', 'value': 'Active'}, {'name': 'time', 'owner': 'cfstore', 'value': '2024-04-23 01:15:22.941623'}, {'name': 'recceiverID', 'owner': 'cfstore', 'value': 'fel-recceiver'}, {'name': 'recordType', 'owner': 'cfstore', 'value': 'ai'}], 'tags': []}
2024-04-23T01:15:22+0200 [-] INFO:recceiver.cfstore Total channels to update: 4

what I understand that happens is:

  • recceiver server announces presence
  • reccaster client connects
  • client uploads records
  • server checks for existing matching channels in channelfinder, by "iocid".
  • does not find any since the client port changed since last time (iocid contains IP:Port)
  • updates channel completely "update channel with different IOC"
  • client and server keep connection up, and play pingpong
  • nothing else happens.

my question now is:
is it intended that the time, and pvStatus properties never update? why?
I noticed that the debug log states "Polling begins", but never "polling complete", or "update channel with same IOC".
shouldnt the time and pvStatus be regularly updated as some kind of heartbeat, and put a channel "inactive", when a client disconnects / disappears?

For example, I have a couple of channels in my channelfinder marked as active, but they are offline right now. You cant tell from the channelfinder and would assume they are usable.

I need some clarification :D

thanks in advance,
William

error: KeyError: 'show ; just print'

I built recsync on EPICS 7.0.2, Anaconda3-5.3.1 and CentOS7.
The client woks well. However, I found some errors when I started the server.

[yifans@ server] $ ls
cf.conf  demo.conf  MANIFEST.in  README  recceiver  recceiver.sqlite3  setup.py  twisted
[yifans@ server] $ sqlite3 test.db -init recceiver.sqlite3 .exit
-- Loading resources from recceiver.sqlite3
[yifans@ server] $ export PYTHONPATH=.
[yifans@ server] $ twistd -r poll -n recceiver -f demo.conf
Traceback (most recent call last):
  File "/home/yifans/anaconda3/bin/twistd", line 11, in <module>
    sys.exit(run())
  File "/home/yifans/anaconda3/lib/python3.7/site-packages/twisted/scripts/twistd.py", line 31, in run
    app.run(runApp, ServerOptions)
  File "/home/yifans/anaconda3/lib/python3.7/site-packages/twisted/application/app.py", line 674, in run
    runApp(config)
  File "/home/yifans/anaconda3/lib/python3.7/site-packages/twisted/scripts/twistd.py", line 25, in runApp
    runner.run()
  File "/home/yifans/anaconda3/lib/python3.7/site-packages/twisted/application/app.py", line 381, in run
    self.application = self.createOrGetApplication()
  File "/home/yifans/anaconda3/lib/python3.7/site-packages/twisted/application/app.py", line 448, in createOrGetApplication
    ser = plg.makeService(self.config.subOptions)
  File "/home/yifans/hals/epics/software/recsync/server/recceiver/application.py", line 112, in makeService
    ctrl = ProcessorController(cfile=opts['config'])
  File "/home/yifans/hals/epics/software/recsync/server/recceiver/processors.py", line 89, in __init__
    plug = plugs[plugname]
KeyError: 'show ; just print'
[yifans@ server] $ twistd --version
twistd (the Twisted daemon) 18.7.0
Copyright (c) 2001-2018 Twisted Matrix Laboratories.
See LICENSE for details.

If `infotags` is blank then the recceiver crashes

If you have a configuration that has the following:

[cf]
infotags =

then the recceiver crashes on loading:

Traceback (most recent call last):
  File "/home/recceiver/venv/bin/twistd", line 8, in <module>
    sys.exit(run())
  File "/home/recceiver/venv/lib/python3.10/site-packages/twisted/scripts/twistd.py", line 35, in run
    app.run(runApp, ServerOptions)
  File "/home/recceiver/venv/lib/python3.10/site-packages/twisted/application/app.py", line 674, in run
    runApp(config)
  File "/home/recceiver/venv/lib/python3.10/site-packages/twisted/scripts/twistd.py", line 29, in runApp
    runner.run()
  File "/home/recceiver/venv/lib/python3.10/site-packages/twisted/application/app.py", line 374, in run
    self.postApplication()
  File "/home/recceiver/venv/lib/python3.10/site-packages/twisted/scripts/_twistd_unix.py", line 254, in postApplication
    self.startApplication(self.application)
  File "/home/recceiver/venv/lib/python3.10/site-packages/twisted/scripts/_twistd_unix.py", line 455, in startApplication
    app.startApplication(application, not self.config["no_save"])
  File "/home/recceiver/venv/lib/python3.10/site-packages/twisted/application/app.py", line 689, in startApplication
    service.IService(application).startService()
  File "/home/recceiver/venv/lib/python3.10/site-packages/twisted/application/service.py", line 276, in startService
    service.startService()
  File "/home/recceiver/venv/lib/python3.10/site-packages/twisted/application/service.py", line 276, in startService
    service.startService()
  File "/home/recceiver/venv/lib/python3.10/site-packages/twisted/application/service.py", line 276, in startService
    service.startService()
  File "/home/recceiver/venv/lib/python3.10/site-packages/recceiver/cfstore.py", line 57, in startService
    self._startServiceWithLock()
  File "/home/recceiver/venv/lib/python3.10/site-packages/recceiver/cfstore.py", line 95, in _startServiceWithLock
    whitelist.append('recordDesc')
AttributeError: 'str' object has no attribute 'append'

This is due to

wl = self.conf.get('infotags', list())
whitelist = [s.strip(', ') for s in wl.split()] \
    if wl else wl

if the infotags is empty (but present) then wl is an empty string, so we choose the second branch of the conditional and return whitelist as an empty string.

recceiver does not work with latest versions of twisted

recceiver does not work with twisted 16.5.0, 17.9.0, 18.7.0:

(channelfinder27) [waynelewis@vip-knotapple server]$ twistd -r poll -n recceiver -f demo.conf
Usage: twistd [options]
Options:
  -b, --debug          Run the application in the Python Debugger (implies
                       nodaemon),         sending SIGUSR2 will drop into
                       debugger
      --chroot=        Chroot to a supplied directory before running
  -d, --rundir=        Change to a supplied directory before running [default:
                       .]
  -e, --encrypted      The specified tap/aos file is encrypted.
      --euid           Set only effective user-id rather than real user-id.
                       (This option has no effect unless the server is running
                       as root, in which case it means not to shed all
                       privileges after binding ports, retaining the option to
                       regain privileges in cases such as spawning processes.
                       Use with caution.)
  -f, --file=          read the given .tap file [default: twistd.tap]
  -g, --gid=           The gid to run as.  If not specified, the default gid
                       associated with the specified --uid is used.
      --help           Display this help and exit.
      --help-reactors  Display a list of possibly available reactor names.
  -l, --logfile=       log to a specified file, - for stdout
      --logger=        A fully-qualified name to a log observer factory to use
                       for the initial log observer.  Takes precedence over
                       --logfile and --syslog (when available).
  -n, --nodaemon       don't daemonize, don't use default umask of 0077
  -o, --no_save        do not save state on shutdown
      --originalname   Don't try to change the process name
  -p, --profile=       Run in profile mode, dumping results to specified file.
      --pidfile=       Name of the pidfile [default: twistd.pid]
      --prefix=        use the given prefix when syslogging [default: twisted]
      --profiler=      Name of the profiler to use (profile, cprofile).
                       [default: cprofile]
  -r, --reactor=       Which reactor to use (see --help-reactors for a list of
                       possibilities)
  -s, --source=        Read an application from a .tas file (AOT format).
      --savestats      save the Stats object rather than the text output of the
                       profiler.
      --spew           Print an insanely verbose log of everything that happens.
                       Useful when debugging freezes or locks in complex code.
      --syslog         Log to syslog, not to file
  -u, --uid=           The uid to run as.
      --umask=         The (octal) file creation mask to apply.
      --version        Print version information and exit.
  -y, --python=        read an application from within a Python file (implies
                       -o)

twistd reads a twisted.application.service.Application out of a file and runs
it.
Commands:
    conch            A Conch SSH service.
    dns              A domain name server.
    ftp              An FTP server.
    inetd            An inetd(8) replacement.
    mail             An email service
    manhole          An interactive remote debugger service accessible via
                     telnet and ssh and providing syntax coloring and basic line
                     editing functionality.
    news             A news server.
    portforward      A simple port-forwarder.
    procmon          A process watchdog / supervisor
    socks            A SOCKSv4 proxy service.
    web              A general-purpose web server which can serve from a
                     filesystem or application resource.
    words            A modern words server
    xmpp-router      An XMPP Router server

/home/waynelewis/.conda/envs/channelfinder27/bin/twistd: Unknown command: recceiver
(channelfinder27) [waynelewis@vip-knotapple server]$

Full Python package list:

(channelfinder27) [waynelewis@vip-knotapple server]$ conda list
# packages in environment at /home/waynelewis/.conda/envs/channelfinder27:
#
# Name                    Version                   Build  Channel
appdirs                   1.4.3            py27h28b3542_0
asn1crypto                0.24.0                   py27_0
attrs                     18.1.0                   py27_0
automat                   0.7.0                    py27_0
ca-certificates           2018.03.07                    0
certifi                   2018.8.24                py27_1
cffi                      1.11.5           py27h9745a5d_0
chardet                   3.0.4                    py27_1
constantly                15.1.0           py27h28b3542_0
cryptography              2.3              py27hb7f436b_0
cryptography-vectors      2.3                      py27_0
enum34                    1.1.6                    py27_1
httplib2                  0.11.3                   py27_0    conda-forge
hyperlink                 18.0.0                   py27_0
idna                      2.7                      py27_0
incremental               17.5.0                   py27_0
ipaddress                 1.0.22                   py27_0
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libstdcxx-ng              8.2.0                hdf63c60_1
ncurses                   6.1                  hf484d3e_0
nose                      1.3.7                    py27_2
openssl                   1.0.2p               h14c3975_0
pip                       10.0.1                   py27_0
pyasn1                    0.4.4                    py27_0
pyasn1-modules            0.2.2                    py27_0
pycparser                 2.18                     py27_1
pyopenssl                 18.0.0                   py27_0
pysocks                   1.6.8                    py27_0
python                    2.7.15               h1571d57_0
readline                  7.0                  ha6073c6_4
requests                  2.19.1                   py27_0
service_identity          17.0.0           py27h28b3542_0
setuptools                40.0.0                   py27_0
six                       1.11.0                   py27_1
sqlite                    3.24.0               h84994c4_0
tk                        8.6.7                hc745277_3
twisted                   18.7.0           py27h14c3975_1
urllib3                   1.23                     py27_0
wheel                     0.31.1                   py27_0
zlib                      1.2.11               ha838bed_2
zope.interface            4.1.1                    py27_0
(channelfinder27) [waynelewis@vip-knotapple server]$

recceiver works with twisted 14.0.2 and 15.5.0:

(channelfinder27) [waynelewis@vip-knotapple server]$ conda list twisted
# packages in environment at /home/waynelewis/.conda/envs/channelfinder27:
#
# Name                    Version                   Build  Channel
twisted                   14.0.2                   py27_0
(channelfinder27) [waynelewis@vip-knotapple server]$
(channelfinder27) [waynelewis@vip-knotapple server]$ twistd -r poll -n recceiver -f demo.conf
2018-08-27 20:11:25+0200 [-] Log opened.
2018-08-27 20:11:25+0200 [-] twistd 14.0.2 (/home/waynelewis/.conda/envs/channelfinder27/bin/python 2.7.15) starting up.
2018-08-27 20:11:25+0200 [-] reactor class: twisted.internet.pollreactor.PollReactor.
2018-08-27 20:11:25+0200 [-] Starting
2018-08-27 20:11:25+0200 [-] CastFactory starting on 32888
2018-08-27 20:11:25+0200 [-] Starting factory <recceiver.recast.CastFactory instance at 0x7f9d45b1add0>
2018-08-27 20:11:25+0200 [-] CastFactory starting on 43660
2018-08-27 20:11:25+0200 [-] ('listening on', IPv4Address(TCP, '0.0.0.0', 43660))
2018-08-27 20:11:25+0200 [-] Announcer starting on 54219
2018-08-27 20:11:25+0200 [-] Starting protocol <recceiver.announce.Announcer instance at 0x7f9d4723b638>

Full Python package list in working version:

(channelfinder27) [waynelewis@vip-knotapple server]$ conda list
# packages in environment at /home/waynelewis/.conda/envs/channelfinder27:
#
# Name                    Version                   Build  Channel
appdirs                   1.4.3            py27h28b3542_0
asn1crypto                0.24.0                   py27_0
attrs                     18.1.0                   py27_0
automat                   0.7.0                    py27_0
ca-certificates           2018.03.07                    0
certifi                   2018.8.24                py27_1
cffi                      1.11.5           py27h9745a5d_0
chardet                   3.0.4                    py27_1
constantly                15.1.0           py27h28b3542_0
cryptography              2.3              py27hb7f436b_0
cryptography-vectors      2.3                      py27_0
enum34                    1.1.6                    py27_1
httplib2                  0.11.3                   py27_0    conda-forge
hyperlink                 18.0.0                   py27_0
idna                      2.7                      py27_0
incremental               17.5.0                   py27_0
ipaddress                 1.0.22                   py27_0
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libstdcxx-ng              8.2.0                hdf63c60_1
ncurses                   6.1                  hf484d3e_0
nose                      1.3.7                    py27_2
openssl                   1.0.2p               h14c3975_0
pip                       10.0.1                   py27_0
pyasn1                    0.4.4                    py27_0
pyasn1-modules            0.2.2                    py27_0
pycparser                 2.18                     py27_1
pyopenssl                 18.0.0                   py27_0
pysocks                   1.6.8                    py27_0
python                    2.7.15               h1571d57_0
readline                  7.0                  ha6073c6_4
requests                  2.19.1                   py27_0
service_identity          17.0.0           py27h28b3542_0
setuptools                40.0.0                   py27_0
six                       1.11.0                   py27_1
sqlite                    3.24.0               h84994c4_0
tk                        8.6.7                hc745277_3
twisted                   15.5.0                   py27_0
urllib3                   1.23                     py27_0
wheel                     0.31.1                   py27_0
zlib                      1.2.11               ha838bed_2
zope.interface            4.1.1                    py27_0
(channelfinder27) [waynelewis@vip-knotapple server]$

INFO tag data not arriving if alias comes before info tag in record def

When a record contains an alias and an info tag in a record definition, the order appears to matter.

We are running ChannelFinder and recsync on many IOCs and have been populating CF with basic information on the records as well as a custom INFO tag we call "archive".

I have been getting this archive information from CF for a few weeks and fixing small bugs in my code as more IOC's get the
archive tag. Recently, many records with both an alias and the archive tag, have not shown up in CF with data from the tag.

I tried changing a few records in the db file, to have the archive tag come first, then the alias.
I've added _log.info lines to recceiver code in recast.py to expose when an alias is found, it's recname and rid.
I also log the info tag data found. Then I can use grep on the logs to match up the rid with the recname, alias and any info tag data.

The aliases are not consistently showing up with info tag data.

Looking at the code in recsync client, dbcd.c, pushRecord, line 70. I wonder if the return upon detecting an alias happens too soon?

Push CA and PVA port numbers to channel finder

Based on code-a-thon discussion we would like to push RSRV_SERVER_PORT and PVAS_SERVER_PORT to channel finder by default in cfstore.py . These variables were added to the default reccaster env list here 58870d2

The proposed property names are:

  • iocIP
  • caPort
  • pvaPort
  • recsyncPort

There already exists a property iocid which contains the IP address and TCP port used for reccaster communication. Since we need the IP address for the CA & PVA ports it would be better to make the IP address it's own property. For backwards compatibility with clients who might use iocid, this property will be made optional

Scalability issues

FRIB (@berryma4) reported that in situations where many IOC's each with 10k's of pvs are started at the same time the receiver crashes.
It is due to memory overflows with the receiver bufferring the record lists from all the IOC's.

RecSync documentation missing information

I'm currently developing a Python implementation of RecCaster to be used alongside p4p, Unfortunately, I've found the documentation a bit unclear or incomplete. Could you please help clarify or provide additional information?

In the existing implementation, the UDP announcement message includes a server address "SERV ADDR". However, it uses the broadcast address and not the actual IP used for TCP connections. RecCaster current implementation checks if "255.255.255.255" is received as "SERV ADDR" then uses the UDP source IP address as "SERV ADDR" instead.
Would it possible to update the documentation to provide a clearer explanation Of this behavior?

Would it be possible to expand the documentation to include more detailed information on the "Add Record" and "Add Info" TCP messages?

I am currently unsure about the specific expectations and requirements that RecCeiver has for these messages. While I have able to experiment with various values by reading RecCeiver source code.

These are the following question or additional documentation needed for "Add Record" and "Add Info"

Add Record

  • What are RTYPE types, are they just EPICS types?
  • What RecCeiver is expecting for RNAME? Is it just a full record name? For example, "PREFIX:RECORD:VALUE"

Add Info

  • What is the format for the key/value?
  • What specific information should be included in the "Add info" key/value pair?

BUG: Recsync server crashes when CF server is down

I am running the ChannelFinder with Glassfish. I start both glassfish and recsync-server with systemd, on the same machine.
The recsync-server service is configured to start after the glassfish service. However at this point the ChannelFinder service does not seem to be up, causing a crash in the recsync-server. It would be nice if this was handled and that the recsync-server tries to connect again a while later.

It works fine after restarting recsync-server service .

recsync-server-log.txt

Issue deploying mulitple recceiver instances

When deploying multiple recceivers pushing to one channel finder instance, there is an issue with the clean_service method where it will set all pvStatus properties on all PVs in CF to "Inactive" on recceiver startup. This works fine if you have one recceiver but once there are more, as you start each recceiver then it will mark all the PVs belonging to the other recceiver instances "Inactive"

Setting the cleanOnStart/cleanOnStop preference setting stops the clean_service method from getting called but I think you then lose some functionality. Updating the CF query in clean_service with a preference setting to allow you to limit the scope of the clean could be useful but not sure how to limit it. Query on iocid since usually it's a per subnet basis for each instance?

https://github.com/ChannelFinder/recsync/blob/master/server/recceiver/cfstore.py#L284

occasional recceiver error - string indices must be integers, not unicode

Documenting here an error we occasionally (~once every 6 months) get in recceiver. Usually someone notices that after adding a new IOC or updating reccaster not all the expected info made it to channel finder.

This is the error in recceiver. A restart of recceiver fixes things but I haven't had time to look into what is actually causing the initial problem.

2022-09-19 16:10:01-0700 [-] ERROR:recceiver.cfstore CF_COMMIT FAILURE: [Failure instance: Traceback: <type 'exceptions.TypeError'>: string indices must be integers, not unicode
        /usr/lib64/python2.7/threading.py:785:__bootstrap
        /usr/lib64/python2.7/threading.py:812:__bootstrap_inner
        /usr/lib64/python2.7/threading.py:765:run
        --- <exception caught here> ---
        /usr/lib64/python2.7/site-packages/twisted/python/threadpool.py:167:_worker
        /usr/lib64/python2.7/site-packages/twisted/python/context.py:118:callWithContext
        /usr/lib64/python2.7/site-packages/twisted/python/context.py:81:callWithContext
        /opt/recsync/recceiver/cfstore.py:269:_commitWithThread
        /opt/recsync/recceiver/cfstore.py:552:poll
        /opt/recsync/recceiver/cfstore.py:430:__updateCF__
        ]

cfstore properties

The current cfstore stores all the pertinent information (host ip, connection port) however this is not very end user friendly
next set of properties.

property name property description
hostname host name provided by epics ioc env variables (TR.infos.get('HOSTNAME'))
iocname ioc name provided by epics ioc env variables (TR.infos.get('IOCNAME'))
owner ioc owner provided by epics ioc env variables (TR.infos.get('USER'))
status Active or Inactive depending on if a valid IOC is know to be hosting this channel
time last time the status of this channel was updated
iocid 'TR.src.host':'TR.src.port' the connection information used to uniquely assign channels to a particular IOC

while the hostname and iocname properties populated form the env variables are user friendly, they are not enough (#1). The additional property connection stores the information which would allow for the correct processing of the shutdown/close TR.
The cfstore client would serach for past channels based on the connection property value and not the hostname/iocname.

@Dylan171 @mdavidsaver @mskinner5278

Recceiver does not shutdown cleanly

Recceiver process does not stop cleanly, kill -9 is sometimes required. This seems to correlated with the retry algorithm invoked when an error occurs when connecting to CF server.

Wrong output location for RecCeiver

When twistd --logfile is used, client information output generated by Transaction is shown on stdout instead of going into a log file specified for twistd. Consequently, if twistd is run as a daemon, all valuable information is muffled and does not appear in the log.

recsync fails to correctly identify different iocs

As reported by ESS in certain situations the TR object src information seems to be the same for different ioc. The problem was noticed when cfstore properties were not correctly updated with ioc boots.

Recceiver fails to start

@shengb

I am seeing an issue where receiver fails to start at commit 7ebed83. I'm using Debian Jessie, Twisted 14.0.2, Python 2.7.9. It seems to be working before this commit. Has anything changed in the way recceiver is to be run (like python version)? @yifans reported a similar issue on his setup #34 (comment)

devuser@develop-vmphy0:~/recsync/server$ twistd -r poll -n recceiver -f demo.conf
Traceback (most recent call last):
File "/usr/bin/twistd", line 14, in
run()
File "/usr/lib/python2.7/dist-packages/twisted/scripts/twistd.py", line 27, in run
app.run(runApp, ServerOptions)
File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 642, in run
runApp(config)
File "/usr/lib/python2.7/dist-packages/twisted/scripts/twistd.py", line 23, in runApp
_SomeApplicationRunner(config).run()
File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 376, in run
self.application = self.createOrGetApplication()
File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 436, in createOrGetApplication
ser = plg.makeService(self.config.subOptions)
File "/home/devuser/recsync/server/recceiver/application.py", line 112, in makeService
ctrl = ProcessorController(cfile=opts['config'])
File "/home/devuser/recsync/server/recceiver/processors.py", line 89, in init
plug = plugs[plugname]
KeyError: 'show'

The steps I did were following. I did not have recceiver package installed on my system.
git clone https://github.com/ChannelFinder/recsync.git
cd recsync/server
git reset --hard 7ebed83
twistd -r poll -n recceiver -f demo.conf

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.