connect-platform / connect-platform Goto Github PK

View Code? Open in Web Editor NEW

98.0 9.0 17.0 5.77 MB

Visual Backend Programming Platform

Home Page: https://connect-platform.com

License: MIT License

JavaScript 35.93% HTML 29.39% CSS 8.38% TypeScript 26.30%

nodejs nodejs-framework backend microservice visual-programming-language low-code low-code-development-platform

connect-platform's Introduction

HIRE CONNECT DEVELOPERS

CONNECT platform is a graphical backend development platform aiming to speed up development of async logical microservices (like drastically). The core idea is that async logic is inherently a "Graph" and a representation via a "Chain" (i.e. text-based code) is extremely sub-optimal (so yes the "visuality" is not for non-programmers to be able to play around).

Since not all parts of a microservice logic are necessarily async, CONNECT facilitates injecting sync pieces of code on the fly. However, since sync code is more computationally focused, it is probably less specific to the business logic of the microservice and more a general algorithm, hence probably the best place for it is at the lowest layers in the architecture. So generally it is advisable to attach sync code in form of packages.

🚀 Platform to use: https://connect-platform.com/connect/

📗 Documentation: https://medium.com/connect-platform/guides/home

🎬 Video (Change the world and stuff): https://www.youtube.com/watch?v=Orjtawd7_EU

🐦 Twitter (@CONNECT_pltfrm): https://twitter.com/connect_pltfrm

💬 Discord Community: https://discord.gg/z22kZh

How to setup 📋

▶️easy way with docker:

create a folder, cd to it,
run this:

docker run -dit \
          --name connect-platform \
          -p 4000:4000 \
          -v $(pwd)/panel-generated:/app/panel-generated \
          -v $(pwd)/secure:/app/secure \
          --env CONNECT_INSTALL_EXTERNAL_PACKAGES=true \
          --env CONNECT_REMOTE_SHELL_ENABLED=true \
          loreanvictor/connect-platform

▶️easy way without docker: https://github.com/CONNECT-platform/connect-platform-boilerplate

▶️easy way for developing a package: https://github.com/CONNECT-platform/connect-platform-package-boilerplate

▶️more comprehensive setup: https://medium.com/connect-platform/how-to-setup-connect-platform-d82d49e029ee

How to configure

You can set config values directly from environment variables using a simple (and probably familiar) templating language. You can just specify the value in curly braces as folllows "{{ ENV_VAR || default_value }}".

It is also possible to specify a request size limit through the request_limit config property.

Socket support is enabled through the enable_sockets property. Socket.io support can be further configured through the socket_config config object. socket_config.use_redis activates support for redis for which socket_config.redis or socket_config.redis_pub and socket_config.redis_sub have to be specified to configure the redis connection for the pub/sub support (can be empty objects). Further information on how to use the socket.io-redis package can be found here.

An example configuration is shown below for reference.

{
  "config_from_env": "{{ CONNECT_CONFIG_FROM_ENV || default }} ",
  "request_limit": "100mb",
  "enable_sockets": true,
  "socket_config": {
    "use_redis": true,
    "redis": {},
    "redis_pub": {},
    "redis_sub": {}
  }
}

Environment variables that are already loaded by the provided launch script are shown in the following table.

Variable Name	What it does?
CONNECT_PRODUCTION_MODE	Run CONNECT platform in production mode
CONNECT_ENABLE_SOCKET	Enable socket.io support

How to contribute

Take a look here.

connect-platform's People

Contributors

Stargazers

Watchers

Forkers

alidadsetan nikhildowhile farnasirim bhagaban must afrog33k zhouxigui007 bwaris94 schultek visionarylab marcushawley datacraft-gaas ahmadfikrimasyhur zino974 vikingdadmedic jhoneme snafi99

connect-platform's Issues

Optimization Suggestion: Eliminate render-blocking

Eliminate render-blocking JavaScript and CSS in above-the-fold content.

In inedx.html, line 7:

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>

Note:

If the "src" attribute is present, the <script> element must be empty.
If async is not present and defer="defer": The script is executed when the page has finished parsing

eg:
If the HTML document looks like this

<head>
    <script type="text/javascript" src="small.js"></script>
</head>

Then you can inline the script as follows:

<script type="text/javascript">
      /* contents of a small JavaScript file */
</script>

Source: PageSpeed Insights
Reference:
https://developers.google.com/speed/docs/insights/BlockingJS
https://www.w3schools.com/tags/tag_script.asp

API versioning

API versioning is an important aspect of any API. How can the connect platform handle this aspect especially with changing API interfaces, this can be of detrimental impact on users of you service and needs to be addressed by the platform as a way of enforcing these API contracts depending on the version of usage.

This is particularly important in the case of a third-party packages market place.

Articles worth taking a look at:
Introduction to API Versioning Best Practices
Versioning a REST API
Best practices for API versioning?

Deselecting boxes on click

Boxes should be deselected when the user click out side of the box. This is of course a quality of life change.

Add a quick-start repository or module that can be easily cloned / installed on your machine

[this was originally mentioned by @tkw1536, adding an issue to migrate the discussion here]

Instead of having to copy/paste code from the tutorial one can just run “git clone …”, or just run “npx connect-platform-quickstart”
The former is really quick to do, and would improve things a lot
The latter could a la https://github.com/facebook/create-react-app

Do you have a non-trivial example?

[this was first brought up by @tkw1536, adding an issue to migrate the discussion here]

I have seen the simple examples, like adding and connecting a handful of nodes, but so far I could have done all that with code just as well.
Here is the example from your website, but in modern JavaScript: https://gist.github.com/tkw1536/4fe9560f045abc6f5b2ff96f6bcf54e6

Ability to record multiple invocations of a node instead of just the first one in "Watch"

As the title says, it would be nice to have multiple calls in a row watched and listed with the developer being able to access each and play it back.

Multiple box selection for re-arrangement

When handling complex graphes it will be more practical to have the ability to select multiple boxes (Shift select for example) to be able to rearrange, duplicate or delete them.

Persist timeout save functionality

Add save functionality for timeout (saved for each node). This should be the default in my opinion.

the panel does not sync the registry properly with the backend

yeah sometimes this happens. should investigate why, and should fix it.

Could a sandbox mode be interesting?

A sort of debug / sandbox mode would be interesting to test out things without having an impact on the actual logic. This is a random thought that needs to be explored more in depth as we dive into more concrete, real-world applications.

Ability to do async calls

The ability to do Async calls will be important for well designed APIs. Hence, it should be carefully thought out how to implement this in the calls from the visual aspect of the platform. Also, in this context, callbacks would be necessary for some workflows / logic.

500 Error when the watch timer runs out

It would be better to add a warning for watch timeout running out (both in the actual response and the connect interface)

Better error handling for loaded packages

When developing packages, it can get really annoying to not know about syntax errors (or other errors) that prevent packages from getting loaded. The only warning that you would get is that the module couldn't be loaded. An expressive error would provide a much better hint on what to do to solve the error efficiently.

This is probably only relevant for people who will develop packages on top of the platform but could potentially affect users which could theoretically run into problems when setting up the platform and using packages that might have errors due to the environment of the setup.

Interconnected service doesn't handle trailing slash changes

When you change an end point name / URI to remove a trailing slash then it is not recognized by nodes that already use it with the trailing slash name.

Maximum Stack Calls on Multi-Watch

when watching multiple nodes, console.log will cause a maximum stack call as multiple console watchers attempt to override the native functionality.

Exporting Interface

[first mentioned by @tkw1536, adding issue to migrate the discussion here]

A button “Copy as CURL request” or “Copy as JavaScript Client Function” would be really great. Then just copy-pasting that into another project would really add to usability for me.

Return home button

This is a really superficial comment, but the button for getting back home is not intuitive as it is placed in the menu on the bottom left under the save button. A more efficient and intuitive way of getting back to the home panel would be better in my personal opinion.

Pre-made end-points

A sort of templates for package usage could provide an intuitive way of not only documenting a package but also for easy setup when it comes to new users.

So the idea is basically block constructs that are a pre-arranged ensemble of nodes and their connections which are specific to a package and come with it.

Refactor into node

The idea consists of being able to automatically refactor selected blocks into a new node that is used in the existing node code.

Optimizing Images

Properly formatting and compressing images can save many bytes of data.
Reduction of 84.9KiB (50% reduction)

Source: PageSpeed Insights
Below are compressed images:

shot1.png ( 59.9KB --> 27.3KB )

shot2.png ( 107KB --> 54.3KB)

Pack/Unpack nodes

packing/unpacking is a recurrent thing and maybe adding some native nodes helps lazy developers on this one. @CowboyBebug first mentioned this during doing Telegram stuff, and I have some idea of how to pull that off:

pack: should get a number of inputs and pack them into one object,
unpack: should get an input and extract one property out of it.

logging errors

Is a logging mechanism for errors provided by the platform? @loreanvictor

Updated with detailed description:
So my question is more related to run-time errors as a way to provide a mechanism for logging errors that are not to be expected under normal circumstances and hence provide a way for the visual developers to debug their visual code. This is especially important for production as unforeseen complexities (Bugs) will eventually arise (Murphy's law).

Also logging can provide other levels of information (Debug, Info etc..)

Adding cmd+s (ctrl+s) as shortcut for saving

It would be nice to get the ability to save progress just by using the shortcut instead of having to click the save button every time.

panel-generated files corruption

it happens sometimes that some files generated by the panel become corrupted, causing bogus registry info to be passed to the client. observed cases: panel-generated/index.js (was empty), panel-generated/config.json (was empty). this probably occurs due to immature interruption of saving/deleting processes, and should be checked with a high priority.

Missing package "event-stream" in dependencies.

When installing on a local machine using npm i:
404 Not Found: [email protected]

That version is not available on npm, possibly due to the hack on that library.

Box size and automatically adapting to it's contents

This also references issue #14. I noticed that making calls to longer URLs can get a little bit messy with the call box / search menu sizes.

Type hinting / contract hinting

As an analogy to interfaces in Object Oriented Programming languages, it would provide a basis for best practice coding when there's a kind of type check on each call made out to nodes.

Errors due to bad configuration can lead to un-launchable instance

I tried to launch a package which used a configuration to connect to a redis server. When I used a wrong port for testing (by setting it through the config panel) it won't let me start the connect instance anymore (which means I could only remove the config from the config file and not through the platform to remedy this problem.

Potential problems with cache control

API caching will be a really important aspect of the platform as it could lead to better performance when done right and to outright wrong results if done wrong.

Some articles that might help:
Implementing API cache control
How-to: HTTP Caching for RESTful & Hypermedia APIs
RESTful Web Services - Caching

Add production configuration support

Add support for production configuration that only loads on production and is not committed to the repo.

Click to dive deeper

It would be a nice feature to be able to directly jump into node's code from the call itself. So when I call a node on the visual interface, it would prove to be of good value to be able to just double click or click a button to directly go into that node.

Ability to rearrange input / output order

This could be interesting when the graph gets more complicated and you have intersecting arrows which can't simply be solved by rearranging the blocks. Therefore, an automatic rearrangement of the inputs / outputs order could provide a potential solution to this problem.

Command palette

It will be useful to add a command panel (like with sublime text CMD+SHIFT+P) with useful commands like Add call block, add expression block, add input, add config, add output

race condition error

sometimes when we've got racing data links, the node might get stuck for some reason. this might be some silenced error or some mishandling of race conditions. anyways since race conditions aren't supposed to work fine, this is of low priority for now. but should nevertheless get fixed.

panel allows you to create incompatible pin connections

it should not

"Install custom code" hides on input click bug

When you open the Install custom code modal / page and then click on one of the inputs or tab to the other one, it automatically closes. This only happens once (first click) for each of the two inputs.

cc @loreanvictor

Mock testing and making sure End-points work

It will become of crucial importance in real world cases that APIs are tested out before actually going into production.

What happens if: you need to mock test for example production includes list of 1M email to transact and you need to make sure 100% your code works before you send the 1M emails? a way to dev/prod switch?

Streaming Nodes

what about nodes streaming data instead of putting out once? i.e. the node does not stop execution after first output and actually multiple outputs can be outputted? the node obviously would then need to explicitly finish the stream. I don't know, it might be a useful concept specifically as it enables asynchronous iteration (something we currently cannot support). however, it also makes understanding and specifically debugging the code much harder.

@must @CowboyBebug ideas?

Ability to duplicate nodes / blocks

It would be really nice to have the ability to duplicate nodes and blocks (With the shortcut of copy paste and also from a right click menu as an option).

Blocks reference calls / component logic that is used in a node.
Nodes are the macro-logic end-points.

This is interesting because it simplifies creating the visual logic in some cases where reusing a graph is relevant either in package development or in actual development on top of the platform.

Generating Documentation

[first mentioned by @tkw1536, adding an issue to migrate the discussion here]

Generating human-readable documentation, maybe even with descriptions for all fields, would be huge.

Hanging errors / Timing out?

When running API end-points, it can get really frustrating that a call would hang for no apparent reason only to find out that the problem is an output that hasn't been passed to an input. A sort of time out mechanism should be provided to remedy this problem.

Input & Output Treatment

[this topic was first brought up by @tkw1536, adding an issue to migrate the discussion]

The type of each input was unclear. Maybe indicate if inputs are e.g. a string or a number. You could even validate them at runtime and throw an error if they are of the wrong type.

Ability to export/import nodes

An accessible way of exporting / importing end-point logic would be interesting from multiple perspectives:

Save logic for sharing with other people (like snippets)
Shareable test cases
Easily accessible logic blocks for documentation purposes / Tutorials

I'm pretty sure we can come up with many more use cases for a functionality like this. And I am aware that the logic is already translated into a file format somewhere in the platform. This functionality is about providing an easy way of exporting / importing such a logic.

Bundle paths by base path

As the number of paths grow bigger, a structured Tree (through URIs) would be really helpful in keeping everything in order. Also, getting the ability to reorder nodes on the main interface could also help maintaining things organized.

Copy paste blocks

We definitely need the ability to copy paste blocks as that would make the dev workflow faster.

Treatment of Node Data

[this issue was mentioned originally by @tkw1536, adding this issue to migrate the discussion over here].

When I add one in the panel, it was at first unclear to me how this node was stored. Does it remain after I restart the server? Is it stored on disk?

I had a slightly closer look and noticed that they were stored in some JSON format on disk.
IMO this is fine for development usage, but if I actually want to use this in production that could be bad.

Parsing / Interpreting some JSON can be slow, especially if some values have to be propagated through a graph.

If I see this right, atm this is done for every request.
Does it not make sense, to “compile” nodes into JS codes for production usage / on every change?This would avoid having to propagate the graph manually every time.
If done right, this could lead to automatically generating an entire codebase (e.g. an express app), that could then be run in stand-alone mode efficiently. Makes connect useful as a rapid prototyping tool.

loops cause platform to crash

running a node with a loop within it causes the platform to become unresponsive, although the node itself works correctly on first execution. probably some sub-nodes keep working indefinitely even after the master-node has responded.

Layers for links

It would be really helpful to be able to filter the different layers of logic into 3:

Data layer with all the data links
Control layer with all the precedence control logic
Error layer with all the error logic or a way to handle error logic in a different way

Delete block with DEL / backspace

A quality of life change would be to be able to delete blocks just by using the DEL or backspace key.

Bigger Expression Box

[this was first mentioned by @tkw1536, adding an issue to migrate the discussion here]

Sometimes “Expressions” grow big. “a + b” fits in a text box, but how about having a button “Create node in external editor”, that opens up a normal editor where you can type code (either on the browser like Cloud9, or via the operating system)