Code Monkey home page Code Monkey logo

erllambda's Introduction

erllambda

Enable AWS Lambda functions to be written in Erlang

logo

Overview

The erllambda library provides all functionality needed to build and deploy fully functional AWS Lambda functions, written entirely in Erlang.

Erlang Lambda functions implement a simple two function behavior:

-module(hello_lambda).

-behavior(erllambda).
-export([handle/2]).

-spec handle( Event :: map(), Context :: map() ) -> {ok, Body} | {error, ErrorBody}.
handle( Event, Context ) ->
    erllambda:succeed( "Hello Event ~p", [Event] ).

There are really two ways to get started. First you can checkout and review the erllambda_example project. This is a complete working AWS Lambda written in Erlang, with the goal of demonstrating the capabilities available as part of the framework, and how to write Lambda functions for different purposes in AWS.

The second path is to use the rebar3_erllambda plugin for rebar3. This will produce a fully working erllambda project that can be used as a starting point for any new development. This plugin also implements additional rebar3 erllambda zip building and AWS Lambda function packaging that simplify development.

More detailed information about developing Lambda functions using erllambda can be found in:

Ownership

The erllambda application and its supporting libraries are primarily owned by motobob and velimir

Dependencies

The erllambda application is built using rebar3, and all other dependencies are automatically pulled in when erllambda is used in other projects rebar.config.

The following are used as part of the Erlang/Elixir package for AWS Lambda:

Initial setup, compilation and testing

TLDR; as long as your basic Erlang environment is setup, getting started developing erllambda should be as easy as forking the repo, and then:

git clone [email protected]:${USER}/erllambda.git
cd erllambda
git remote add upstream [email protected]:alertlogic/erllambda.git
rebar3 compile 
rebar3 ct
rebar3 erllambda zip

Packaging and Deployment

There are two key points about running Erlang AWS Lambda functions:

OpenSSL version

Important notice: at this moment AWS Lambda native runtime has openssl 1.0.1k while latest linux systems have 1.0.2 or even 1.1.1.. See AWS Lambda

To be able to run Erlang Lambda functions in AWS Lambda it is vital to package your SW with Erlang built against openssl 1.0.1. You will want to setup DockerMachine and utilize the erllambda_docker repo for this purpose. This will allow you to perform release builds for erllambda based components directly from the command line.

Users can have their own Erlang dockers and have those with statically linked preferred openssl versions. However statically linking for base libraries is discouraged.

Lambda Memory Size

Current testing has shown that it does not make sense to run Erlang on AWS Lambda functions with less then 256MB RAM. Having 512-1024MB is optimal for most of the use cases.

Configuration

Code loading mode

Depending on a use case, erlang code loading mode can significantly affect execution performance.

To switch between interactive or embedded modes set CODE_LOADING_MODE environment variable on AWS Lambda function creation step with a desired value:

aws --profile default --region <region> \
 lambda create-function \
 --function-name <your_function> \
 --memory-size 1024 \
 --handler <your_function_module_name> \
 --zip-file fileb://_build/prod/<your_function>-0.0.0.zip \
 --environment "Variables={CODE_LOADING_MODE=interactive}"
 --runtime provided \
 --role <role-arn>

Basic Deployment

See Erllambda Example for the step-by-step procedure to deploy your Lambda. It boils down to following steps:

  • compile and build a prod profile release of your application.
rebar3 erllambda zip
  • create your AWS Lambda function
aws --profile default --region <region> \ 
 lambda create-function \
 --function-name <your_function> \
 --memory-size 1024 \
 --handler <your_function_module_name> \
 --zip-file fileb://_build/prod/<your_function>-0.0.0.zip \ 
 --runtime provided \
 --role <role-arn>
  • or update your previously deployed AWS Lambda function
aws --profile default --region <region> \ 
 lambda update-function-code /
 --function-name <your_function> \
 --zip-file fileb://_build/prod/<your_function>-0.0.0.zip
  • and invoke it
aws --profile default --region <region> \
 lambda invoke  --function-name <your_function> \
  --log-type Tail \
  --payload '{"msg": "hello"}' \
  outputfile.txt

It is however recommended to use CloudFormation based approach described in rebar3_erllambda

How to contribute

Contributions to this repo are always welcome. If you have an idea for improving the this or related components, please submit a github issue, or simply submit a PR directly that implements your improvement.

For complex changes, or the introduction of a major feature, it is beneficial to discuss ideas before implementing them, so that your efforts can focus on pull requests that will be accepted more easily.

As you prepare your pull request, make sure that you follow the coding conventions that exist in the files, and always make sure that all unit and common tests run. Please ensure that your contribution always adds to the coverage percentage, and does not decrease it.

How to report defects

If you encounter an problem, or simply have a question about using this repo, please submit a github issue.

erllambda's People

Contributors

al-pavel-baturko avatar al-roman avatar antonzaets avatar cgibbons-al avatar fishliver avatar getong avatar hmartel-alertlogic avatar key-master avatar lehoff avatar mikebenza avatar motobob avatar rmpalomino avatar sanmiguel avatar velimir avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

erllambda's Issues

pthread error during thaw phase of Erlang VM

As mentioned in #68, there is a persistent pthread error that causes a crash of the Erlang VM in long running AWS Lambda functions. This was noticed in a Lambda function that processes a steady AWS Kinesis stream.

I have opened this issue after #68 was resolved, because this underlying issue is not fixed; #69 only prevents the resulting Linux core dumps from filling disk space, it does not prevent the Erlang VM from crashing during the AWS Lambda thaw phase and causing a core dump.

The pthread error shown in #68 is:

pthread/ethr_event.c:164: Fatal error in wait__(): Operation not permitted (1)

Also see the error and additional discussion in leo-project/leofs#843 (comment):

Sep 20 18:55:53 bodies-master leo_manager[406]: pthread/ethr_event.c:164: Fatal error in wait__(): Invalid argument (22)

move ERTS to AWS Lambda Layer and support that deployment model

currently we package both ERTS and application code into a single zip.
it makes long term sense to separate out Erlang it self into AWS Layer. while making the application code be part of the zip and just referring the AWS Layer with a desired runtime version.

remove references to internal tools

%% <li>Environment Variable <code>REGION</code>
%% <p>The <code>makeincl<code> development tools set the
%% <code>REGION</code> environment variable when executing eunit or
%% common_tests, to allow local workstation execution to use AWS
%% resources during testing.</p>
%% </li>

%% makeincl sets ENVIRON and IWS containers set IWS_BASESTACKNAME
case os:getenv( "ENVIRON", os:getenv( "IWS_BASESTACKNAME" ) ) of

case file:read_file( "/var/alertlogic/data/base-stack-name" ) of

@motobob please let me know when all code is migrated and this is no longer needed.

a call to `erllambda:message/2` with empty `Values` list brakes logging

the following code:

handle(_Event, _Context) ->
    erllambda:message("Starting logs test"),
    erllambda:message("This line brakes logging", []),
    erllambda:message("This and any further logs are not printed"),
    {ok, "done"}.

gives the following log at the first invoke:

START RequestId: c51bef51-f4cf-11e8-a390-f701c3391a2f Version: $LATEST
[context@36312 aid="c51bef51-f4cf-11e8-a390-f701c3391a2f"] 127.0.0.1 - - [30/Nov/2018:18:43:05 -0000] Invoke Next
Next returns, in invoke 1543603385146
Starting logs test
END RequestId: c51bef51-f4cf-11e8-a390-f701c3391a2f
REPORT RequestId: c51bef51-f4cf-11e8-a390-f701c3391a2f  Init Duration: 696.26 ms    Duration: 1.97 ms   Billed Duration: 700 ms     Memory Size: 2048 MB    Max Memory Used: 75 MB

and the following log at further invokes:

START RequestId: e5ea113c-f4cf-11e8-8c3c-75c4192a8db6 Version: $LATEST
END RequestId: e5ea113c-f4cf-11e8-8c3c-75c4192a8db6
REPORT RequestId: e5ea113c-f4cf-11e8-8c3c-75c4192a8db6  Duration: 9.09 ms   Billed Duration: 100 ms     Memory Size: 2048 MB    Max Memory Used: 75 MB 

Are there any metrics about the cold start time

Actually the Java and .Net runtimes for lambda have pretty slow cold start times, I imagine this will be a thing also with Erlang but will be nice to measure that and compare with other runtimes. I probably can start something and write a blog post, great job folks, I'm excited about this!

Remove references to internal tools from README

The README currently has a lot of references to internal things like dsh the Data Access team and so forth. If this is going to be published publicly we should remove that stuff because it will just confuse people.

Don't finish the lambda until all logs will appear in the output

Problem statement
Due to asynchronous nature of the output we have a situation when logs for the particular invocation doesn't appear between START <invocation id> and END <invocation id>:

  1. Appear in the next invocation;
  2. Don't appear at all if lambda's container is killed due to some reasons.

In the past (in js version) we waited for special sequence in the output of Erlang VM and later directly called console.log from js. I don't exactly know why it worked better (maybe it was overloaded, maybe it prints synchronously).

Example of the logs that was truncated:


START RequestId: e4031f8b-b8de-4dec-bde4-8804ceb606d4 Version: $LATEST
17:11:12 <normal logs without any errors started here>
17:11:12 <last normal log without any errors>
17:11:16 END RequestId: e4031f8b-b8de-4dec-bde4-8804ceb606d4
17:11:16 REPORT RequestId: e4031f8b-b8de-4dec-bde4-8804ceb606d4  Duration: 4551.62 ms    Billed Duration: 4600 ms Memory Size: 2048 MB   Max Memory Used: 354 MB
17:11:24 START RequestId: a7ce76e0-bbda-44d9-87c8-13bd393f24ef Version: $LATEST
17:11:29 END RequestId: a7ce76e0-bbda-44d9-87c8-13bd393f24ef
17:11:29 REPORT RequestId: a7ce76e0-bbda-44d9-87c8-13bd393f24ef  Duration: 4623.58 ms    Billed Duration: 4700 ms Memory Size: 2048 Max Memory Used: 2048 

e4031f8b-b8de-4dec-bde4-8804ceb606d4 really finished with an error {"Unhandled",<<"{\"errorMessage\":\"{{invalid_cast,{read,{error,{http_error,timeout}}}},\\n {gen_server,call,[<0.521.0>,next,30000]}}\",\"errorType\":\"HandlerFailure\"}">>} - we see this in the place where lambda is called. However, there aren't any logs in the output.

I see that the next invocation is failed with OOM (Memory Size: 2048 Max Memory Used: 2048 ) and perhaps this the cause why logs from previous execution didn't appear in the logs for a new invocation.

Proposal

I propose to add some option to erllambda config like wait_for_logs or sync_logs with boolean value and that option will trigger mechanism for logs synchronisation. The mechanism should be found in experiments.

@motobob @velimir what do you think, guys?

@lehoff FYI

need better exception handling

one of issues reported is when erllambds itself swalloed an exception and went into perpetual mode of

{{case_clause,{ok,{{403,"Forbidden"},[{"content-length","127"},{"date","Fri, 16 Nov 2018 07:01:05 GMT"},{"content-type","application/json"}],<<"{\"errorMessage\":\"Transition from STATE_INVOKE_NEXT to STATE_INVOKE_NEXT is not allowed.\",\"errorType\":\"InvalidStateTransition\"}\n">>}}},[{erllambda_poller,invoke_next,1,[{file,"/home/jenkins/workspace/search-filter/_build/default/lib/erllambda/src/erllambda_poller.erl"},{line,160}]},

need better protection form such of errors on our side and make it fail the whole VM (erlang:halt() type) and report the issue to AWS Runtime so that the container is killed.

Bad call to erllamba:message/2 breaks subsequent logging

Similar to #45.

Calling erllambda:message/2 breaks logging by causing a crash in io_lib:scan_format/2, in turn causing the error_logger handler to be unregistered permanently.

If I'm reading the OTP gen_event documentation right and testing right, the erllamba_error_handler gen_event behavior doesn't actually crash and get re-initialized. It just gets its terminate/2 method called and unregistered from error_logger. Since the behavior is added as a handler during its initialization, but never re-initialized, it is never re-added as a handler.

Edit: Short reproduction steps in rebar3 shell:

1> erllambda:message("Good ~s", [<<"format">>]).
ok
Good format
2> erllambda:message("Bad format", [any]).
ok
3> erllambda_error_handler terminated with reason {error,{'EXIT',{badarg,[{io_lib,scan_format,[[66,97,100,32,102,111,114,109,97,116],[any]],[{file,[105,111,95,108,105,98,46,101,114,108]},{line,215}]},{erllambda,line_format,2,[{file,[47,85,115,101,114,115,47,114,112,97,108,111,109,105,110,111,47,116,101,109,112,47,101,114,108,108,97,109,98,100,97,47,95,98,117,105,108,100,47,100,101,102,97,117,108,116,47,108,105,98,47,101,114,108,108,97,109,98,100,97,47,115,114,99,47,101,114,108,108,97,109,98,100,97,46,101,114,108]},{line,330}]},{erllambda_error_handler,output,2,[{file,[47,85,115,101,114,115,47,114,112,97,108,111,109,105,110,111,47,116,101,109,112,47,101,114,108,108,97,109,98,100,97,47,95,98,117,105,108,100,47,100,101,102,97,117,108,116,47,108,105,98,47,101,114,108,108,97,109,98,100,97,47,115,114,99,47,101,114,108,108,97,109,98,100,97,95,101,114,114,111,114,95,104,97,110,100,108,101,114,46,101,114,108]},{line,88}]},{erllambda_error_handler,handle_event,2,[{file,[47,85,115,101,114,115,47,114,112,97,108,111,109,105,110,111,47,116,101,109,112,47,101,114,108,108,97,109,98,100,97,47,95,98,117,105,108,100,47,100,101,102,97,117,108,116,47,108,105,98,47,101,114,108,108,97,109,98,100,97,47,115,114,99,47,101,114,108,108,97,109,98,100,97,95,101,114,114,111,114,95,104,97,110,100,108,101,114,46,101,114,108]},{line,57}]},{gen_event,server_update,4,[{file,[103,101,110,95,101,118,101,110,116,46,101,114,108]},{line,577}]},{gen_event,server_notify,4,[{file,[103,101,110,95,101,118,101,110,116,46,101,114,108]},{line,559}]},{gen_event,handle_msg,6,[{file,[103,101,110,95,101,118,101,110,116,46,101,114,108]},{line,300}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,249}]}]}}} and state #{}
3> erllambda:message("Good ~s", [<<"format">>]).
ok

when handler crashes stack trace posted as a string rather than a list

{
  "stackTrace": "[{erllambda_crash_handler,handle,2,\n                          [{file,\"/Users/gstarinkin/src/erllambda/_build/test/lib/erllambda/test/erllambda_crash_handler.erl\"},\n                           {line,11}]},\n {erllambda,invoke_exec,3,\n            [{file,\"/Users/gstarinkin/src/erllambda/_build/test/lib/erllambda/src/erllambda.erl\"},\n             {line,307}]},\n {erllambda,'-invoke/3-fun-0-',4,\n            [{file,\"/Users/gstarinkin/src/erllambda/_build/test/lib/erllambda/src/erllambda.erl\"},\n             {line,288}]}]",
  "errorType": "HandlerFailure",
  "errorMessage": "terminated with exception {error, {badmatch,<<\"foobaz\">>}}"
}

replace jiffy with jsone

It's advised to build a package using erllambda contianer since the library depends on jiffy, that can't be loaded in AWS Lambda container if it was built on a different platform.

To make packetization easier how about we change json library to something that is erlang native like jsone?

invoke_exec should be done via spawn_link

previously when in cowboy times, actually user handlers were done in cowboy handlers...now they are part of poller process . not cool.

need to spawn_link ;) to simplify GC between warm invokes.

found to cause delays/memory issues as per @key-master

@velimir0xff FYI

need to flush out std out before invoke next

example of spill logs between invokes

MONITORING|1542361428|31|histogram|search.sort.batch.time|#operation:map_write
MONITORING|1542361428|32|histogram|search.sort.batch.time|#operation:create_new_map
MONITORING|1542361428|1292|histogram|search.sort.invoke.time|#invoke_type:merge_batches
END RequestId: 1d6b832c-e984-11e8-a0c1-51c269721f59
REPORT RequestId: 1d6b832c-e984-11e8-a0c1-51c269721f59	Duration: 1297.37 ms	Billed Duration: 1300 ms Memory Size: 2048 MB	Max Memory Used: 746 MB	
START RequestId: 1e375de9-e984-11e8-b495-17f47950124b Version: $LATEST
MONITORING|1542361428|5|count|search.sort.lhttpc|#conn,end
Invoke Success path 1542361428640 http://127.0.0.1:9001/2018-06-01/runtime/invocation/1d6b832c-e984-11e8-a0c1-51c269721f59/response
Invoke Next path 1542361428641 http://127.0.0.1:9001/2018-06-01/runtime/invocation/next

notice success log of previous invoke printed after new one actually started.

@velimir0xff

Runtime issues: pthread and awk error writing to stdout

I've noticed pthread errors off and on in for a long time but it hasn't been a big problem. In the past week I've started to see a new failure scenario where it seems like, according to the error logged, that the Lambda temporary disk space might be filled up.

I'm not sure what to make of it, but it is causing ~2 hours of failed processing every time that it happens. In the last occurrence, logs of the error were limited to a single log group of a single Lambda function that has two Kinesis event sources. I assume that the ~2 hours is the lifecycle of that failed container since logs eventually completely stop for that log group.

My first thought was that maybe the pthread error is causing a crashdump to be stored despite the bootstrap script trying to disable them, and then too many of them cause the disk space to run out. I thought this because the execution before the awk error in these occurrences has consistently been the pthread error.

I tried to bump the configured memory for the Lambda function, despite the error mentioning disk space instead of memory, and that didn't appear to help at all.

Erlang runtime details:
OTP version: 21.3.8.4
OpenSSL version: 1.0.2k-fips
erllambda version: 2.1.3

Example of the error is below. The sequence of executions can be summarized as:

  1. Execution success
  2. pthread error
  3. awk error
    ... awk error repeats for all executions until log stream ends...
Invoke Success path 1571007997408 http://127.0.0.1:9001/2018-06-01/runtime/invocation/ffcddaa8-63e1-4a12-b6e7-5b9cbd3dac39/response
Invoke Next path 1571007997410 http://127.0.0.1:9001/2018-06-01/runtime/invocation/next
END RequestId: ffcddaa8-63e1-4a12-b6e7-5b9cbd3dac39
REPORT RequestId: ffcddaa8-63e1-4a12-b6e7-5b9cbd3dac39	Duration: 134.89 ms	Billed Duration: 200 ms	Memory Size: 768 MB	Max Memory Used: 386 MB	

START RequestId: cec77f66-f012-4052-8c31-afebbcc079ae Version: $LATEST
pthread/ethr_event.c:164: Fatal error in wait__(): Operation not permitted (1)
END RequestId: cec77f66-f012-4052-8c31-afebbcc079ae
REPORT RequestId: cec77f66-f012-4052-8c31-afebbcc079ae	Duration: 664.46 ms	Billed Duration: 700 ms	Memory Size: 768 MB	Max Memory Used: 386 MB	
RequestId: cec77f66-f012-4052-8c31-afebbcc079ae Error: Runtime exited with error: signal: aborted (core dumped)
Runtime.ExitError
creating necessary erllambda run dirs
OpenSSL is OpenSSL 1.0.2k-fips 26 Jan 2017
starting ErlangVM
awk: cmd. line:5: (FILENAME=- FNR=7) warning: error writing standard output (No space left on device)

START RequestId: cec77f66-f012-4052-8c31-afebbcc079ae Version: $LATEST
creating necessary erllambda run dirs
OpenSSL is OpenSSL 1.0.2k-fips 26 Jan 2017
starting ErlangVM
awk: cmd. line:5: (FILENAME=- FNR=7) warning: error writing standard output (No space left on device)
END RequestId: cec77f66-f012-4052-8c31-afebbcc079ae
REPORT RequestId: cec77f66-f012-4052-8c31-afebbcc079ae	Duration: 86.19 ms	Billed Duration: 100 ms	Memory Size: 768 MB	Max Memory Used: 12 MB	
RequestId: cec77f66-f012-4052-8c31-afebbcc079ae Error: Runtime exited with error: exit status 1
Runtime.ExitError

START RequestId: b8c66eb1-e0ea-4967-a1a0-5530e002f343 Version: $LATEST
creating necessary erllambda run dirs
OpenSSL is OpenSSL 1.0.2k-fips 26 Jan 2017
starting ErlangVM
awk: cmd. line:5: (FILENAME=- FNR=7) warning: error writing standard output (No space left on device)
END RequestId: b8c66eb1-e0ea-4967-a1a0-5530e002f343
REPORT RequestId: b8c66eb1-e0ea-4967-a1a0-5530e002f343	Duration: 64.65 ms	Billed Duration: 100 ms	Memory Size: 768 MB	Max Memory Used: 15 MB	
RequestId: b8c66eb1-e0ea-4967-a1a0-5530e002f343 Error: Runtime exited with error: exit status 1
Runtime.ExitError

A GitHub search reveals a different pthread related crash: leo-project/leofs#843 (comment)

Sep 20 18:55:53 bodies-master leo_manager[406]: pthread/ethr_event.c:164: Fatal error in wait__(): Invalid argument (22)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.