excamera / mu Goto Github PK
View Code? Open in Web Editor NEWFramework to Run General-Purpose Parallel Computations on AWS Lambda
Framework to Run General-Purpose Parallel Computations on AWS Lambda
Example:
If a __server.py binary runs with a bucket that does not exist (-b option). Errors are thrown _after* all the lambdas are configured.
SERVER HANDLING (1) FAIL(retrieving norun:sintel-1k-png16/00000007.png->/tmp/lambda_Sc4R9q/00000007.png from s3:
Traceback (most recent call last):
Might be better to catch bad configurations early on before we start invoking any lambdas.
It looks like the lambdaize.sh helper utility fails if the executable name has a 'dot' in it.
`$ MEM_SIZE=1536 TIMEOUT=180 ./src/lambdaize/lambdaize.sh gpb
Function name: a.out_Qm2Ce8JN
A client error (ValidationException) occurred when calling the CreateFunction operation: 1 validation error detected: Value 'a.out_Qm2Ce8JN' at 'functionName' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:aws:lambda:)?([a-z]{2}-[a-z]+-\d{1}:)?(\d{12}:)?(function:)?([a-zA-Z0-9-]+)(:($LATEST|[a-zA-Z0-9-]+))?`
To keep track of where time is spent in each lambda it would be nice to have an additional mode that invokes the context object i.e. context.get_remaining_time_in_millis() around specific functions/lines.
This way we can keep track of overheads vs. s3 access time vs. actual user function code time and have finer grain trace information.
While running the example in mu/README_xc-enc.md, the xcenc_server.py script errors out with the following trace:
Traceback (most recent call last):
File "/tmp/mu_example/mu/src/lambdaize/xcenc_server.py", line 280, in <module>
main()
File "/tmp/mu_example/mu/src/lambdaize/xcenc_server.py", line 277, in main
run()
File "/tmp/mu_example/mu/src/lambdaize/xcenc_server.py", line 255, in run
server.server_main_loop(ServerInfo.states, XCEncSettingsState, ServerInfo)
File "/tmp/mu_example/mu/src/lambdaize/libmu/server.py", line 265, in server_main_loop
rnext = r.do_read()
File "/tmp/mu_example/mu/src/lambdaize/libmu/machine_state.py", line 108, in do_read
return self.do_handle()
File "/tmp/mu_example/mu/src/lambdaize/libmu/machine_state.py", line 85, in do_handle
state = state.transition(msg)
File "/tmp/mu_example/mu/src/lambdaize/libmu/machine_state.py", line 165, in transition
return self.post_transition()
File "/tmp/mu_example/mu/src/lambdaize/libmu/machine_state.py", line 279, in post_transition
return self.loopState(self)
File "/tmp/mu_example/mu/src/lambdaize/xcenc_server.py", line 202, in __init__
dist_from_end = (kfDist if kfDist is not None else ServerInfo.num_passes) - effActNum
TypeError: unsupported operand type(s) for -: 'tuple' and 'int'
Looking into the culprit line, ServerInfo.num_passes is a tuple, while effActNum is an integer.
Output s3 buckets are hardcoded based on the name of the input parameters:
outName = "%s-y4m_%02d" % (ServerInfo.video_name, ServerInfo.num_frames)
ServerInfo should have another option to specify an output s3 bucket, else use the default parameters.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.