nficano / python-lambda Goto Github PK
View Code? Open in Web Editor NEWA toolkit for developing and deploying serverless Python code in AWS Lambda.
License: ISC License
A toolkit for developing and deploying serverless Python code in AWS Lambda.
License: ISC License
I am new to lambda and Bash script. When I modify the service.py under the root folder, e.g. change e + pi to e - pi, it did not return the result after the changes. Anyone could help? Or do I need to recompiled the service.pyc file everytime I make the changes?
Thanks for the help.
hi folks,
having one more issue here :
when i test the function on the lambda console - it works fine - but when I try to call the API , the following error occurs and the cloudwatch log suggests no parameters were sent.
Traceback (most recent call last):
File "/var/task/service.py", line 11, in handler
"body": e + pi
TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType'
called like so:
curl https://<endpoint>/prod/my_lambda_function --data '{"pi" : 3.14, "e" : 2.718}' -H 'Content-Type: application/json' -v
* Trying 54.192.136.115...
* Connected to l3rj7tq81d.execute-api.us-west-2.amazonaws.com (54.192.136.115) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: //anaconda/ssl/cacert.pem
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.execute-api.us-west-2.amazonaws.com
* start date: May 30 00:00:00 2017 GMT
* expire date: Feb 28 23:59:59 2018 GMT
* subjectAltName: host "l3rj7tq81d.execute-api.us-west-2.amazonaws.com" matched cert's "*.execute-api.us-west-2.amazonaws.com"
* issuer: C=US; O=Symantec Corporation; OU=Symantec Trust Network; CN=Symantec Class 3 Secure Server CA - G4
* SSL certificate verify ok.
> POST /prod/my_lambda_function HTTP/1.1
> Host: l3rj7tq81d.execute-api.us-west-2.amazonaws.com
> User-Agent: curl/7.49.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 26
>
* upload completely sent off: 26 out of 26 bytes
< HTTP/1.1 502 Bad Gateway
< Content-Type: application/json
< Content-Length: 36
< Connection: keep-alive
< Date: Mon, 10 Jul 2017 15:58:50 GMT
< x-amzn-RequestId: aa719f09-6588-11e7-a0b0-657c544d8223
< X-Cache: Error from cloudfront
< Via: 1.1 941cbd1049a1cc3d6d633dce8d55cf36.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: KbnVOMc8Fwkm3cLO_pU2X8R5Sr54XIeUdpq9qtgq77QZF7zww65_zw==
<
* Connection #0 to host l3rj7tq81d.execute-api.us-west-2.amazonaws.com left intact
{"message": "Internal server error"}
When building or deploying the lambda script doesn't follow the --use-requirements
flag. The click group describes it's use. What am I missing?
When following the steps in Getting Started, I see this error when I run pip install python-lambda
:
Collecting python-lambda
Using cached python-lambda-0.5.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/br/bx8tyrg91fx1pgqwvnwc315w0000gn/T/pip-build-l7DEkV/python-lambda/setup.py", line 16, in <module>
pip_requirements = [str(r.req) for r in requirements]
File "/Users/tmh/virtualenvs/pylambda/lib/python2.7/site-packages/pip/req/req_file.py", line 84, in parse_requirements
filename, comes_from=comes_from, session=session
File "/Users/tmh/virtualenvs/pylambda/lib/python2.7/site-packages/pip/download.py", line 425, in get_file_content
'Could not open requirements file: %s' % str(exc)
pip.exceptions.InstallationError: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/br/bx8tyrg91fx1pgqwvnwc315w0000gn/T/pip-build-l7DEkV/python-lambda/
Possibly using https://coverage.readthedocs.io/en/coverage-4.4.1/ or something else you know of. This will help immensely for those who prefer TDD.
I'm not sure if there's a way to do this already with python-lambda, but the node-lambda library handles it this way: https://github.com/motdotla/node-lambda#custom-environment-variables
I am deploying psycopg2
with my lambda build. Because psycopg2
relies on the statically linked libpq.so
library, pip install
doesn't work (so neither does --local-package
).
What I've done in the past is to pre-compile the psycopg2 package and copy the folder directly to lambda dist (no pip install). Seems like a useful feature.
Currently filenames are coded only with a timestamp... ideally there would be provisions for creating the filename based on a version string. The same input could then in turn be used to set the version as per the AWS Lambda API:
https://docs.aws.amazon.com/lambda/latest/dg/versioning-intro.html
This could be implemented by adding a "version: x.x.x" line to the config.yaml file, if present then the version provided will be used, otherwise the code will exhibit the current behavior for backward compatibility. Another option would be to add a '--version=x.x.x' command line parameter to 'lambda build' and the pending S3 change 'lambda upload'.
It would break 100% compatibility with current outputs, but it would also likely be more intuitive to have the filenames generated be in the format -<version|timestamp>.zip rather than the current -.zip.
subdirectories in the the root folder are being skipped in the build function.
When calling invoke, the environment variables from config.yaml are not loaded into the environment. Any script that relies on the variables will fail with a KeyError.
I am developing on macOS and want to package Pillow. I have installed and downloaded pillow from an ec2 instance and want it to be packaged instead of the one in site-packages. Right now only way to do is to include PIL in source_directories and uninstall it via pip before packaging.
Is there a better way to do it?
When I try to deploy my lambda function with lambda deploy_s3
I receive the following error:
TypeError: deploy_s3() got an unexpected keyword argument 'use_requirements'
I've installed the dependencies of my project trough a requirements.txt
file with the command pip install -r requirements.txt
. The error doesn't arise if I use lambda deploy
.
Hello!
Thought you might be interested in incorporating lambda-packages that we use with Zappa as part of your library's build process. You can see how we do it automatically swap out lambda-compatible dependencies.
What do you think about adding an .env file into every lambda folder? Sending environment variables can be accomplished after uploading the zip file.
# http://docs.aws.amazon.com/lambda/latest/dg/env_variables.html
aws lambda create-function \
--region us-east-1
--function-name myTestFunction
--zip-file fileb://path/package.zip
--role role-arn
--environment Variables={LD_LIBRARY_PATH=/usr/bin/test/lib64}
--handler index.handler
--runtime nodejs4.3
--profile default
Hi, When running invoke locally it appears it does not configure logging..
Should it ?
No handlers could be found for logger "root"
The current implementation of function_exists requires the calling account to have list_functions permissions. It also iterates through the functions. A better implementation would be to use the get_function function.
The VPC config settings are used during update_function
but not in create_function
Hi there.
We are getting the follow errors using python3:
Traceback (most recent call last):
File "/Users/cristian.oliveira/.virtualenv/test/bin/lambda", line 53, in <module>
cli()
File "/Users/cristian.oliveira/.virtualenv/test/lib/python3.6/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/Users/cristian.oliveira/.virtualenv/test/lib/python3.6/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/Users/cristian.oliveira/.virtualenv/test/lib/python3.6/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/cristian.oliveira/.virtualenv/test/lib/python3.6/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/cristian.oliveira/.virtualenv/test/lib/python3.6/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/Users/cristian.oliveira/.virtualenv/test/bin/lambda", line 20, in init
aws_lambda.init(CURRENT_DIR)
File "/Users/cristian.oliveira/.virtualenv/test/lib/python3.6/site-packages/aws_lambda/aws_lambda.py", line 149, in init
copy(dest_path, src)
File "/Users/cristian.oliveira/.virtualenv/test/lib/python3.6/shutil.py", line 243, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/Users/cristian.oliveira/.virtualenv/test/lib/python3.6/shutil.py", line 122, in copyfile
with open(src, 'rb') as fsrc:
IsADirectoryError: [Errno 21] Is a directory: '/Users/cristian.oliveira/.virtualenv/test/lib/python3.6/site-packages/aws_lambda/project_templates/__pycache__'
It is trying to copy a folder, in this case __pycache__
, which I think is a wrong behavior. I have already fixed that here. I going to open a PR with the fix.
when i follow the instructions in the README and
lambda deploy , I get the following stack trace:
(pylam) $ lambda deploy
Gathering pip packages
Installing boto3==1.4.1
Installing botocore==1.4.61
Installing click==6.6
Installing docutils==0.12
Installing futures==3.0.5
Installing jmespath==0.9.0
Installing pip==9.0.1
Installing pyaml==15.8.2
Installing python-dateutil==2.5.3
Installing PyYAML==3.11
Installing s3transfer==0.1.10
Installing setuptools==36.0.1
Installing six==1.10.0
Installing wheel==0.30.0a0
Installing wsgiref==0.1.2
Bundling: 'event.json'
Bundling: 'service.py'
Bundling: 'service.pyc'
Starting new HTTPS connection (1): lambda.us-west-2.amazonaws.com
Creating your new Lambda function
Starting new HTTPS connection (1): sts.amazonaws.com
Creating lambda function with name: my_lambda_function
Traceback (most recent call last):
File "/pylam/bin/lambda", line 62, in <module>
cli()
File "/pylam/lib/python2.7/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/pylam/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/pylam/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/pylam/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/pylam/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/pylam/bin/lambda", line 47, in deploy
aws_lambda.deploy(CURRENT_DIR, use_requirements, local_package)
File "/pylam/lib/python2.7/site-packages/aws_lambda/aws_lambda.py", line 89, in deploy
create_function(cfg, path_to_zip_file)
File "/pylam/lib/python2.7/site-packages/aws_lambda/aws_lambda.py", line 371, in create_function
Publish=True
File "/pylam/lib/python2.7/site-packages/botocore/client.py", line 159, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/pylam/lib/python2.7/site-packages/botocore/client.py", line 470, in _make_api_call
api_params, operation_model, context=request_context)
File "/pylam/lib/python2.7/site-packages/botocore/client.py", line 523, in _convert_to_request_dict
api_params, operation_model)
File "/pylam/lib/python2.7/site-packages/botocore/validate.py", line 270, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in input: "Environment", must be one of: FunctionName, Runtime, Role, Handler, Code, Description, Timeout, MemorySize, Publish, VpcConfig
when environment_variables are commented out, lambda_invoke failed with the following exception:
Traceback (most recent call last):
File "/home/ubuntu/Envs/env1/bin/lambda", line 67, in <module>
cli()
File "/home/ubuntu/Envs/env1/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/home/ubuntu/Envs/env1/local/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/home/ubuntu/Envs/env1/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ubuntu/Envs/env1/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ubuntu/Envs/env1/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/home/ubuntu/Envs/env1/bin/lambda", line 40, in invoke
aws_lambda.invoke(CURRENT_DIR, event_file, verbose)
File "/home/ubuntu/Envs/env1/local/lib/python2.7/site-packages/aws_lambda/aws_lambda.py", line 147, in invoke
for key, value in cfg.get('environment_variables').items():
AttributeError: 'NoneType' object has no attribute 'items'
This happened on a project where I didn't need env vars and can't deploy them due to KMS issue when encrypting env vars.
When calling invoke, if you try to get a env var (os.environ['host']
) set in the config file like so:
host: $HOST
it will print out $HOST
instead of the actual value of the envar $HOST
Hi,
My deployment completes successfully the first time, but errors out on subsequent deployments with this error:
botocore.exceptions.ClientError: An error occurred (ResourceConflictException) when calling the CreateFunction operation: Function already exist: es_get_channel_http_microservice
I'm attempting to deploy the basic demo app to our account, just to get familiar with the workflow. However, when attempting to initialize the deploy, I receive this error, which is traced to line 371 of the aws_lambda.py code (the section dealing with "Environment" variables):
File "/Users/x/Envs/pylambda_slack/bin/lambda", line 62, in <module>
cli()
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/Users/x/Envs/pylambda_slack/bin/lambda", line 47, in deploy
aws_lambda.deploy(CURRENT_DIR, use_requirements, local_package)
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/aws_lambda/aws_lambda.py", line 89, in deploy
create_function(cfg, path_to_zip_file)
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/aws_lambda/aws_lambda.py", line 371, in create_function
Publish=True
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/botocore/client.py", line 159, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/botocore/client.py", line 470, in _make_api_call
api_params, operation_model, context=request_context)
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/botocore/client.py", line 523, in _convert_to_request_dict
api_params, operation_model)
File "/Users/x/Envs/pylambda_slack/lib/python2.7/site-packages/botocore/validate.py", line 270, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in input: "Environment", must be one of: FunctionName, Runtime, Role, Handler, Code, Description, Timeout, MemorySize, Publish, VpcConfig
This appears to be the function being called - any thoughts? I'm not looking to push any environment variable through in this deploy, just get a working app up.
client.create_function(
FunctionName=func_name,
Runtime=cfg.get('runtime', 'python2.7'),
Role=role,
Handler=cfg.get('handler'),
Code={'ZipFile': byte_stream},
Description=cfg.get('description'),
Timeout=cfg.get('timeout', 15),
MemorySize=cfg.get('memory_size', 512),
Environment={
'Variables': {
key.strip('LAMBDA_'): value
for key, value in os.environ.items()
if key.startswith('LAMBDA_')
}
},
Publish=True
)
When I tried to deploy through S3 using lambda deploy_s3
I encountered three issues.
role: basic_s3_upload
in config.yaml
did not work, since it then tried to give that role to the lambda function.I solved this issue by removing the role
definition in config.yaml
, and then everything worked.
update/create_function
functionFrom what I can see the argument list for update/create_function
(https://github.com/nficano/python-lambda/blob/master/aws_lambda/aws_lambda.py#L462) wants use_s3
to be a list of arguments and s3_file
to be a dictionary of keyword arguments. These two values should be a boolean and a string, and when I changed the code to use them as such everything is working fine for me.
event
parameter in the handler function did not workI had to change the code to something like this:
url = json.loads(event.get('body')).get('url')
Are the changes I had to do due to something strange on my system, or are these actually bugs?
I'm using Python 3.6, and have not tested this on any other Python versions.
I extended my python code to include a main lambda handler file and a module in a subdirectory:
main lambda handler: service.py
newly created module: utils/slotutils.py where it contains the get_slot() and elicit_slot() functions.
In service.py, get_slot() and elicit_slot() were imported and referenced
I was able to run the code locally with the command "lambda invoke -v"
However, when the package was uploaded to AWS Lambda, executing the same python code resulted the following error message:
"Unable to import module 'service': No module named utils.slotutils"
Is there an extra setting needed to tell AWS to search for my newly created sub-directory "utils"?
Note: when I moved slotutils.py one directory to co-locate with service.py, my AWS Lambda worked fine. Thank you so much for sharing this great tool.
Great project @nficano, thanks for making it.
I noticed that lambda deploy
zips up and deploys all the packages in the virtualenv, including ones that are only needed to run python-lambda itself locally (e.g. boto, pip, pyaml, etc.) and therefore unneeded on AWS. It would be nice to be able to restrict this, maybe an additional config variable where you can indicate package names needed for your code?
Hi,
After building a simple function and running lambda deploy
I receive this error:
botocore.exceptions.ClientError: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda.
Has anyone else had this issue? Is there a config file somewhere specifying the role that I can edit?
Lots of packages cannot be installed when using lambda deploy
for example,
Installing screen-resolution-extra==0.0.0
Could not find a version that satisfies the requirement screen-resolution-extra==0.0.0 (from versions: )
No matching distribution found for screen-resolution-extra==0.0.0
Thanks for this project and I followed the README very easily, but I think README needs some supplement for new users of AWS (like me).
While I was following that README, Before running of lambda deploy
, I should create IAM users for getting aws_access_key_id
and aws_secret_access_key
. Nextly, I should create ~/.aws/credentials
files for saving both key or just input to config.yaml
(don't need to share). Finally, I should attach two policies to my IAM users, AWSLambdaFullAccess
(for creating lambda function) and IAMFullAccess
(perfoming iam:getUser
). (I think Amazon provides more narrow polices)
Summary,
Before run lambda deploy
aws_access_key_id
and aws_secret_access_key
~/.aws/credentials
AWSLambdaFullAccess
and IAMFullAccess
lambda deploy
Regards
I have a locally install package, built like so:
pip install . --upgrade
I check disk, and its there in the virtualenv.
However, when I run lambda build
I get:
Could not find a version that satisfies the requirement apt-commons==1.0 (from versions: )
No matching distribution found for apt-commons==1.0
I've tried the local-package argument, and it works, but then the module is not picked up in intellij, so, I'd prefer to just have it picked up from virtualenv.
Great project, @nficano.
Have you considered allowing AWS credentials to be defined from environmental variables?
My use case is automating a deploy from Travis. This is the easiest solution I can think of.
gettiing below error while deploying the function
botocore.exceptions.ClientError: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defi
ned for the function cannot be assumed by Lambda.
deployment fails after this message
Starting new HTTPS connection (1): lambda.us-east-1.amazonaws.com
Creating your new Lambda function
Starting new HTTPS connection (1): sts.amazonaws.com
Starting new HTTPS connection (1): lambda.us-east-1.amazonaws.com
To reproduce just
import boto3
somewhere in the code, for example in service.py
and deploy for python3.6
Removing concurrent directory from the zip package solves the problem.
Exception details:
s3_client = boto3.client('s3')
File "/var/task/boto3/__init__.py", line 83, in client
return _get_default_session().client(*args, **kwargs)
File "/var/task/boto3/session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "/var/task/botocore/session.py", line 836, in create_client
client_config=config, api_version=api_version)
File "/var/task/botocore/client.py", line 65, in create_client
cls = self._create_client_class(service_name, service_model)
File "/var/task/botocore/client.py", line 89, in _create_client_class
base_classes=bases)
File "/var/task/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/var/task/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/var/task/boto3/utils.py", line 61, in _handler
module = import_module(module)
File "/var/task/boto3/utils.py", line 52, in import_module
__import__(name)
File "/var/task/boto3/s3/inject.py", line 15, in <module>
from boto3.s3.transfer import create_transfer_manager
File "/var/task/boto3/s3/transfer.py", line 127, in <module>
from s3transfer.exceptions import RetriesExceededError as \
File "/var/task/s3transfer/__init__.py", line 134, in <module>
import concurrent.futures
File "/var/task/concurrent/futures/__init__.py", line 8, in <module>
from concurrent.futures._base import (FIRST_COMPLETED,
File "/var/task/concurrent/futures/_base.py", line 357
raise type(self._exception), self._exception, self._traceback
^
SyntaxError: invalid syntax
I'm getting an error when trying to deploy:
botocore.exceptions.ClientError: An error occurred (RequestEntityTooLargeException) when calling the UpdateFunctionCode operation: Request must be smaller than 69905067 bytes for the UpdateFunctionCode operation
It could be avoided by uploading the lambda function code first to S3 and then deploying it to lambda from there.
Do you plan to support this?
Tnx
I'm not sure if this is a bug in the library, or how I've setup AWS / Lambda.
But everytime I run "lambda deploy" the lambda function I'm deploying to is set to "No VPC", dropping the subnets and security group.
Any ideas why this might be happening?
I'm constantly getting this error while trying to deploy to AWS..
21:00 $ lambda deploy --use-requirements
No dependency packages installed!
Bundling: 'event.json'
Bundling: 'service.py'
Creating your new Lambda function
Traceback (most recent call last):
File "/Users/R.Thomas/.pyenv/versions/3.6.0/bin/lambda", line 62, in <module>
cli()
File "/Users/R.Thomas/.pyenv/versions/3.6.0/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/R.Thomas/.pyenv/versions/3.6.0/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/R.Thomas/.pyenv/versions/3.6.0/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/R.Thomas/.pyenv/versions/3.6.0/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/R.Thomas/.pyenv/versions/3.6.0/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/R.Thomas/.pyenv/versions/3.6.0/bin/lambda", line 47, in deploy
aws_lambda.deploy(CURRENT_DIR, use_requirements, local_package)
File "/Users/R.Thomas/.pyenv/versions/3.6.0/lib/python3.6/site-packages/aws_lambda/aws_lambda.py", line 89, in deploy
create_function(cfg, path_to_zip_file)
File "/Users/R.Thomas/.pyenv/versions/3.6.0/lib/python3.6/site-packages/aws_lambda/aws_lambda.py", line 341, in create_function
byte_stream = read(path_to_zip_file)
File "/Users/R.Thomas/.pyenv/versions/3.6.0/lib/python3.6/site-packages/aws_lambda/helpers.py", line 15, in read
return fh.read()
File "/Users/R.Thomas/.pyenv/versions/3.6.0/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 11: invalid start byte
Lambda supports execution of binary programs within a lambda function, and python-lambda is a convenient way to package and upload these wrapped in a python interface. This almost works perfectly as is, except that python zip doesn't preserve executable bits when creating the package, and AWS requires these to be set or else the binary will fail with a permissions error.
I worked around this for my own use case by hacking helpers.py based on code in this stackexchange question, hardcoding special handling for my executable file. A more complete solution might be to check the executable flags on the files when packaging and preserve when building the zip archive, or alternatively allow the config file to specify a list of executables for special handling during packaging.
Thanks for the useful package!
The current version is 0.7.0 but the version available on pip is 0.5.0
https://pypi.python.org/pypi/python-lambda/
When building or deploying the use of --local-package
will only use the last specified local package.
When I do lambda deploy
it does include these packages:
boto3 (1.4.4)
botocore (1.5.62)
click (6.6)
docutils (0.12)
futures (3.0.5)
jmespath (0.9.0)
pip (9.0.1)
pyaml (15.8.2)
python-dateutil (2.5.3)
python-lambda (2.2.0)
PyYAML (3.11)
s3transfer (0.1.11)
setuptools (38.2.3)
six (1.10.0)
wheel (0.30.0)
They are all NOT part of my project but parts of python-lambda.
They are therefore not needed on AWS Lambda to run my project.
How can I exclude these from deployment?
I would like to deploy lambda function in multiple environments, like development, staging, and production.
Is there is a way to setup config.yaml for different working environments?
Hi everyone,
I'm working on a lambda that will makes use of LXML. When running the lambda in AWS I noticed the following error message: Unable to import module 'service': cannot import name 'etree'
. It seems this is because "LXML must be built with C extensions for libxml2 and libxslt in a way that plays well with the Amazon Lambda execution environment."
Has anyone else run into this issue?
source: https://www.azavea.com/blog/2016/06/27/using-python-lxml-amazon-lambda/
My current usage of python-lambda uses a common requirements.txt file for all my lambda functions. Instead of placing the requirements.txt file in the function directory. Can I just add the ability to pass my own requirements.txt file instead of being forced to put the requirements.txt file in the function directory.
The build command in scripts/lambda uses keyword arguments that are inconsistent with the build function in aws_lambda.py
def build(use_requirements, local_package, config_file): aws_lambda.build( CURRENT_DIR, use_requirements=use_requirements, local_package=local_package, config_file=config_file, )
aws_lambda build function signature looks like this:
def build( src, requirements=False, local_package=None, config_file='config.yaml'):
I have a lambda that imports a local module (i.e. another Python file in the same folder as the main lambda module, as opposed to a pip installed package). Doing a lambda deploy
works, and the resulting lambda on AWS works also. However, when I do lambda invoke
for local testing, I get an ImportError: No module named foo
where foo
is the module I'm trying to import.
Hi,
I want to use Jenkins for packaging our AWS Lambda project. I use ShiningPanda for managing virtual environment. When I execute lambda build
command, it gives error. It creates a zip but it does not have some dependencies. Lines with +
are my commands.
[TEST] $ /usr/bin/python2.7 /var/lib/jenkins/shiningpanda/jobs/033bd94b/virtualenv.py /var/lib/jenkins/shiningpanda/jobs/033bd94b/virtualenvs/945f3fc4
New python executable in /var/lib/jenkins/shiningpanda/jobs/033bd94b/virtualenvs/945f3fc4/bin/python2.7
Also creating executable in /var/lib/jenkins/shiningpanda/jobs/033bd94b/virtualenvs/945f3fc4/bin/python
Installing setuptools, pip, wheel...done.
[TEST] $ /bin/sh -xe /tmp/shiningpanda3458767265880341612.sh
+ pip install -r requirements.txt
Collecting boto3==1.3.1 (from -r requirements.txt (line 1))
Using cached boto3-1.3.1-py2.py3-none-any.whl
Collecting botocore==1.4.32 (from -r requirements.txt (line 2))
Using cached botocore-1.4.32-py2.py3-none-any.whl
Collecting click==6.6 (from -r requirements.txt (line 3))
Collecting docutils==0.12 (from -r requirements.txt (line 4))
Collecting futures==3.0.5 (from -r requirements.txt (line 5))
Using cached futures-3.0.5-py2-none-any.whl
Collecting jmespath==0.9.0 (from -r requirements.txt (line 6))
Using cached jmespath-0.9.0-py2.py3-none-any.whl
Collecting Pillow==3.3.1 (from -r requirements.txt (line 7))
Using cached Pillow-3.3.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyaml==15.8.2 (from -r requirements.txt (line 8))
Collecting python-dateutil==2.5.3 (from -r requirements.txt (line 9))
Using cached python_dateutil-2.5.3-py2.py3-none-any.whl
Collecting python-lambda==0.4.0 (from -r requirements.txt (line 10))
Using cached python_lambda-0.4.0-py2.py3-none-any.whl
Collecting PyYAML==3.11 (from -r requirements.txt (line 11))
Collecting six==1.10.0 (from -r requirements.txt (line 12))
Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: futures, jmespath, six, python-dateutil, docutils, botocore, boto3, click, Pillow, PyYAML, pyaml, python-lambda
Successfully installed Pillow-3.3.1 PyYAML-3.11 boto3-1.3.1 botocore-1.4.32 click-6.6 docutils-0.12 futures-3.0.5 jmespath-0.9.0 pyaml-15.8.2 python-dateutil-2.5.3 python-lambda-0.4.0 six-1.10.0
+ which python
/var/lib/jenkins/shiningpanda/jobs/033bd94b/virtualenvs/945f3fc4/bin/python
+ which pip
/var/lib/jenkins/shiningpanda/jobs/033bd94b/virtualenvs/945f3fc4/bin/pip
+ pip list
boto3 (1.3.1)
botocore (1.4.32)
click (6.6)
docutils (0.12)
futures (3.0.5)
jmespath (0.9.0)
Pillow (3.3.1)
pip (8.1.2)
pyaml (15.8.2)
python-dateutil (2.5.3)
python-lambda (0.4.0)
PyYAML (3.11)
setuptools (26.1.1)
six (1.10.0)
wheel (0.29.0)
+ lambda build
Exception:
Traceback (most recent call last):
File "/var/lib/jenkins/shiningpanda/jobs/033bd94b/virtualenvs/945f3fc4/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/var/lib/jenkins/shiningpanda/jobs/033bd94b/virtualenvs/945f3fc4/local/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run
for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmpxo3fWo/lib/python'
Exception:
Traceback (most recent call last):
File "/var/lib/jenkins/shiningpanda/jobs/033bd94b/virtualenvs/945f3fc4/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/var/lib/jenkins/shiningpanda/jobs/033bd94b/virtualenvs/945f3fc4/local/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run
for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmpMeHaOI/lib/python'
Gathering pip packages
Installing boto3==1.3.1
Installing botocore==1.4.32
Installing click==6.6
Installing docutils==0.12
Installing futures==3.0.5
Installing jmespath==0.9.0
Installing Pillow==3.3.1
Installing pip==8.1.2
Installing pyaml==15.8.2
Installing python-dateutil==2.5.3
Installing python-lambda==0.4.0
Installing PyYAML==3.11
Installing setuptools==26.1.1
Installing six==1.10.0
Installing wheel==0.29.0
Installing wsgiref==0.1.2
Nice Lib. So much simpler than Zappa. It's really what I need, but I am having hard times to deploy. It got stuck when calling lambda deploy.
Found credentials in shared credentials file: ~/.aws/credentials
Starting new HTTPS connection (1): lambda.us-west-2.amazonaws.com
Creating your new Lambda function
Starting new HTTPS connection (1): sts.amazonaws.com
Starting new HTTPS connection (1): lambda.us-west-2.amazonaws.com
Starting new HTTPS connection (2): lambda.us-west-2.amazonaws.com
Starting new HTTPS connection (3): lambda.us-west-2.amazonaws.com
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.