stackvana / microcule Goto Github PK

We should have an integration test that iterates through all supported programming languages and attempts to assert the responses of all hello world / echo services located in https://github.com/Stackvana/microservice-examples, per language.

These test will ensure the target language is working and supported on the system. ( Currently we only test with bash and node )

/usr/local/lib/node_modules/microcule/node_modules/run-service/index.js:16
    throw new Error('service is undefined');
    ^
service is undefined

I was just trying to run the express-simple.js example

$ microcule express-simple.js
javascript microcule started at: http://0.0.0.0:3000
running service /

Currently PHP services use $Hook['params'] scope for merged get and post variables.

It would be somewhat less surprising for PHP users if we also populated PHP's $_GET and $_POST scopes with incoming service parameters.

Our current configuration of babel for microservices is a bit hard-coded. We should respect the users .babelrc configuration / make babel configurable in the stack tool.

see: https://github.com/Stackvana/stack/blob/master/lib/compileServiceCode/babel/index.js#L7

I'm not personally using babel for development, so it any babel users can assist here, it would be much appreciated.

cc @ljharb

In stack/bin/stack file the require in the following line fails:
https://github.com/Stackvana/stack/blob/master/bin/stack#L39

I think it should be

var viewPresenter =  #require('../lib/viewPresenter');

I have run in to a problem, trying to use microcule with python 3.5.3 in virtualenv.

I am currently on Debian 9 Stretch (stable).
My version of npm was installed from https://deb.nodesource.com/setup_9.x And not from the official apt repository, as it wasn't properly maintained, and therefore isn't present in the official apt repo.

When trying to use use Microcule with python3, i was presented with the following error.

Traceback (most recent call last):
  File "/usr/lib/node_modules/microcule/bin/binaries/micro-python3", line 16, in <module>
    __prepare()
  File "/usr/lib/node_modules/microcule/bin/binaries/micro-python3", line 12, in __prepare
    import microcule
  File "/usr/lib/node_modules/microcule/bin/binaries/lib/python/microcule/__init__.py", line 10, in <module>
    fd3 = os.fdopen(3, 'w+')
  File "/home/morten/.virtualenvs/DSEnv/lib/python3.5/os.py", line 1072, in fdopen
    return io.open(fd, *args, **kwargs)
io.UnsupportedOperation: File or stream is not seekable.

Looking in to it a little, i found that, there seems to be a discrepancy in the content of the file microcule/bin/binaries/lib/python/microcule/__init__.py
on github, and the file one gets when microcule is installed according to the recommended "npm install -g microcule"

On Github there is an if else statement, distinguishing between python versions < 3 and others.

# open incoming connection from fd3
if sys.version_info[0] < 3:
  fd3 = os.fdopen(3, 'w+')
else:
  fd3 = os.fdopen(3, 'wb+', buffering=0)

This if else statement is not present in the microcule installed by npm, the first lines of my local file looks like this instead.

# open incoming connection from fd3
fd3 = os.fdopen(3, 'w+')

I'm trying to run microcule on an Ubuntu Server, I had it working but now I only get:
Which looks like Node doesn't have some read or write permission maybe.
I've emptied the hook to just contain:
`module['exports'] = function fakeData (hook) {

}`

Error: spawn EACCES at exports._errnoException (util.js:1050:11) at ChildProcess.spawn (internal/child_process.js:319:11) at Object.exports.spawn (child_process.js:378:9) at spawn (/var/www/db/node_modules/cross-spawn/index.js:17:18) at _spawnService (/var/www/db/node_modules/microcule/lib/plugins/spawn/index.js:376:12) at spawnServiceMiddleware (/var/www/db/node_modules/microcule/lib/plugins/spawn/index.js:157:7) at spawnService (/var/www/db/node_modules/microcule/bin/microcule:280:13) at watchSpawn (/var/www/db/node_modules/microcule/bin/microcule:238:9) at Layer.handle [as handle_request] (/var/www/db/node_modules/express/lib/router/layer.js:95:5) at trim_prefix (/var/www/db/node_modules/express/lib/router/index.js:317:13)

Just saw an instance where an environment or http params variable name containing - was causing the lua codegen to crash.

We need to ensure that the lua code won't crash on variable names with - or any other values that need to be escaped. Pretty sure we can do something like: [[lua string with escape codes]]

It seems it's possible to execute arbitrary bash commands through HTTP parameters when using bash services ( and other possible languages too ).

This isn't intended behavior and should be fixed.

The actual security implications would be determined by the configuration of the server environment microcule is running on. For hook.io, it's not really a problem ( since workers running code are secure on the operating system level ), but it could cause issues for on non-secure systems running untrusted source code.

Now that we have a good way to compile source code into binaries, we should update the coffee-script and babel code paths to use the new compiled language plugin / API.

The main reason for this change will be unifying the code paths for compile / transpile steps and being able to use the new provider API and shasum checksums.

Medium priority as current performance seeks OK.

Currently, all Node.js services are spawned in a new process using the micro-node binary, which is called via microcule.spawn. This ensures process level isolation of the untrusted source code.

Inside micro-node we are using the run-service module, which provides Node.js in-processvm based isolation of the untrusted source code.

Technically ( and previously ) run-service could be executed in-process where microcule.spawn is called ( instead of spawning a new micro-node process ). This could be dangerous as a memory leak or CPU leak in the untrusted source code could affect other running services. It could also be advantageous because it would significantly reduce the amount of resources needed to run Node services, as well as reduce response times by 150+ milliseconds.

We should consider merging the in-process-node branch as a configurable option. Depending on the server environment and intended use-case, being able to spawn in-process Node scripts may be better than enforcing process isolation per Node service.

Currently there is no easy way to send logging messages from ruby.

It should be easy, just create a log helper method.

Here is some sample lua code to give an idea:

function log(...)
  io.stderr:write('{ "type": "log", "payload": { "entry": ' .. json.stringify(arg, false) .. '}}')
end

log method would probably go here: https://github.com/Stackvana/microcule/blob/master/bin/binaries/micro-ruby#L4

The current code generation for lua services puts all Hook.params and Hook.env in a flat variable structure such as Hook_params_foo or Hook_env_secretA.

This isn't ideal, and the data should be put into a Lua table ( or other equivalent object ).

I consistently run into errors with the npm package version, however installing from GitHub works. I suspect the npm package needs updating.

Since all logic for spawning services are essentially stream transformations, we should remove the optional schema and view-presenter and move them into optional plugins.

In order to do this, we'll need a simple .use or .pipe pattern for adding stream transformation plugins.

When running a hook with deprecated api usage you can see:

(node:13604) DeprecationWarning: Using Buffer without `new` will soon stop working. Use `new Buffer()`, or preferably `Buffer.from()`,
 `Buffer.allocUnsafe()` or `Buffer.alloc()` instead.

if you try to use a package like trumpet things get even weirder.

Possible solutions would be to somehow set the process the same way as passing --no-deprecation in the cli, perhaps nodejs/node@c6656db can shed some light on that.

Right now we are sending all HTTP params for golang and rust as individual argv arguments.

It would be better if we sent the HTTP params as JSON and updated the examples to dump the JSON, similar to how JavaScript / Ruby / Python / Lua / etc all currently work.

Currently, the signature for Node.js functions is: function (hook).

It would be more clear to users and node.js friendly if the signature was: function(req, res).

The new function signature would actually be backwards compatible, so this wouldn't break any existing services. Update will be required in run-service npm package.

We could also consider function(req, res, next), but implementing next correctly might be non-trivial.

In some cases, the source code for a service may be stored in a remote location and is not immediately available.

Examples include source code as gist, git repo, vfs, or any remote location.

We should have a plugin available that will parse the incoming service object for potential remote sources and fetch them asynchronously. There should also be a built in cache provider ( redis and in-memory ) that can optionally cache the remote source codes.

I was wondering how one would go about detecting when a user closes / clicks stop in a loading tab? I found a stack overflow thread on how to do it in vanilla express:

req.on("close", function() {
  // request closed unexpectedly
});

req.on("end", function() {
  // request ended normally
}):

Granted I never tested this myself but I did test it inside a hook but to no avail.

How easy would be to add support for WebSockets connections to the created endpoints?

We've gotten a bit of feedback regarding the project name.

All things aside, we have a conflict with Haskell's stack binary, so we must change this project's core binary at the least. Will probably take this opportunity to rename the project to something more unique / easier to find.

Right now, bash services are still injecting all request and env parameters into separate unique bash variables ( like how Lua use to work prior to #18 )

It would be better if we used bash 4 feature of associative arrays for creating two objects, Hook.params and Hook.env, which are populated with many items ( instead of creating many unique variable names )

While trying to implement a microcule powered scraper I've run in to what looks like a broken stream problem, but only if the whole thing is hosted in a docker container (I know, there's a few deps here that could be the root cause but I figured asking for help can't hurt).

To reproduce clone the gist from above:

git clone https://gist.github.com/d10ee3863a6f0773c5386b77abea77fa.git

Build the docker image:

docker build -t microcule-docker

Run the docker image in interactive terminal:

docker run -it -p 3000:3000 microcule-docker

After this hit the following URL's:

http://localhost:3000/request/hook
http://localhost:3000/scrape/hook
http://localhost:3000/scrape/middleware

The second URL breaks in the middle of the JSON. However if you just try a simple npm start and repeat it all runs fine.

One thing to note here is the /request/hook URL works fine in both native/docker so I guess I'm wondering what x-ray does to the stream internally to just close in the middle.

Part of the problem is that debugging this with docker + microcule is a bit tricky and since I suspect that @Marak know a thing or two about hosting hooks inside docker I decided it could be worth filing an issue.

Build badge in the README file is showing build | failing, and as travis-ci log indicates, the test suite has been failing for a long time. 8 of the tests in test/rate-limit-test.js does not pass.

resource-http is too full featured to be included in microcule. The only place resource-http is being used is in the microcule binary itself, and should be able to be replaced with or express or stack ( maybe even just core node http module )

The primary reason for this switch will be to reduce the amount of dependencies in microcule.

We shouldn't lose any features here. Developers could still use resource-http separately and use microcule as a middleware.

Ideally, we wouldn't want to be requesting the service's source from Github on every request.

It would be better if the plugins for Github remote sources were able to accept a cache provider ( similar to how the rate limiter works ) in order to provide the option of caching the source code ( for fast retrieval ).

We'd also want to accept an option to indicate that the cache should be invalidated ( on request ).

This project is legit and deserves more traction!

I think a first step is to think about how the content above the fold in the README could be simplified so that noobs like me would understand why this is amazing within 10 seconds.

I would then use this simplified copy and create a gh-page. This page should preferably have a live demo, where you can play around with a microservice, similar to hook.io's web editor.

To further drive interested, I think it would be dope if the CLI supported creations of microservices.

I was trying to use microcule to run a google assistant example that was designed to run on GCF. However, the API.AI webhooks require an https connection. Looks like microcule is hard coded to start only http servers. Could it be enhanced to have the option of using the https module for starting the server?

This link documents the API.AI webhook requirements.

I am not able to find the log statement added with console.log() in microcule js example index,js. also,In microcule any Js example index.js file am adding console.log() ,but that log statement is not displaying in my command prompt.If i use res.write('') that logs displaying in browser,any possibilities to support logging statement using console.log that will display in console not browser?

The expressjs examples from https://github.com/Stackvana/microcule/tree/master/examples, don't close the connection. Only the javascript code example works as there is a res.end() in it.

When trying to require('./lib/file.js') I get a "module not found" error. Is there a special way to require a module when using microcule?

The default behavior for Ruby is to populate the ENV scope.

We are currently populating Hook.env scope for Ruby.

I believe the path of least surprise for Ruby users will be to use the ENV scope ( as it's already expected ). We can keep both APIs for backwards compatibility.

The core spawning logic at https://github.com/Stackvana/microcule/blob/master/lib/plugins/spawn/index.js currently has two concerns coupled together: preparing the service options for spawning, and managing the spawned service.

microcule's spawning logic should only be concerned with preparing the service options. The actual spawning and managing of STDIO communications should be handled in a separate module.

To start, we can separate these concerns into separate files within this project. If it makes sense to then move the spawn logic itself to a separate package, we will.