Comments (7)
gcampionpae
commented on July 18, 2024
1
Thanks so much for the help with this! Including the setuptools package in the requirements did make this work. I will open an issue with the serverless team to see if this is how it should function or if this is a bug.
from aws-xray-sdk-python.
haotianw465
commented on July 18, 2024
Hi there, sorry for any inconvenience. pkg_resources is introduced in this PR https://github.com/aws/aws-xray-sdk-python/pull/6/files#diff-b28c74eefc290d02eb40fd81f4e26b58R5 which added support for async capture and aioboto. Per http://setuptools.readthedocs.io/en/latest/pkg_resources.html it is distributed with setuptools . How is your CI/CD setup different than your dev box when packaging and deploying?
from aws-xray-sdk-python.
gcampionpae
commented on July 18, 2024
Both deployments happen from a docker container with the Serverless Framework. The only real difference would be that when we run the docker container locally it's on a mac and on our gitlab runner it's a linux box. Otherwise there really isn't much of a difference.
I've also tried following the exact steps locally on my mac as in our CI/CD pipeline and it still works correctly.
Here is our dockerfile:
FROM node
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
build-essential \
python \
python-dev \
python-pip \
curl \
git-core && \
rm -rf /var/lib/apt/lists/* && \
apt-get clean
RUN pip install --upgrade --ignore-installed pip
RUN pip install virtualenv
RUN npm install [email protected] -g
RUN npm install --prefix /sls_plugins [email protected]
RUN npm install --prefix /sls_plugins [email protected]
RUN npm install --prefix /sls_plugins [email protected]
RUN npm install --prefix /sls_plugins [email protected]
RUN npm install [email protected] -g
COPY .pydistutils.cfg /root/.pydistutils.cfg
COPY serverless_wrapper /usr/local/bin/serverless_wrapper
RUN chmod a+x /usr/local/bin/serverless_wrapper
COPY setup_ci_environment /usr/local/bin/setup_ci_environment
RUN chmod a+x /usr/local/bin/setup_ci_environment
COPY requirements_adfs.txt /root/requirements_adfs.txt
RUN mv /root/.pydistutils.cfg /root/.pydistutils.cfg_old && \
pip install -r /root/requirements_adfs.txt && \
rm -f /root/requirements_adfs.txt && \
mv /root/.pydistutils.cfg_old /root/.pydistutils.cfg
COPY samlapi_adfs3.py /usr/local/bin/samlapi_adfs3.py
RUN chmod a+x /usr/local/bin/samlapi_adfs3.py
WORKDIR /workspace
CMD ["serverless"]
We pull this docker image to the machine and then run the setup ci script which is the following:
#!/bin/bash
mv /root/.pydistutils.cfg /root/.pydistutils.cfg_old
virtualenv venv
source venv/bin/activate
pip install -r requirements-dev.txt --quiet
mv /root/.pydistutils.cfg_old /root/.pydistutils.cfg
ln -s -f /sls_plugins/node_modules ./.serverless_plugins
from there we just run sls deploy for our staging environment
Running with gitlab-ci-multi-runner 9.5.0 (413da38)
on nue-docker-01 (ba60b53f)
Using Docker executor with image serverless-docker-image:serverless-1.20 ...
Using docker image sha256:1bbf0506e176d248a2ee0db39ef78ff9775b0f54a0afb5dded89d7d12155dda3 for predefined container...
Pulling docker image ....serverless-docker-image:serverless-1.20 ...
Using docker image ....serverless-docker-image:serverless-1.20 ID=sha256:ae8671f18ef92124361445893cef52fac9f569ef861ce48887f378b69fb12c55 for build container...
Running on runner-ba60b53f-project-51-concurrent-0 via nue-docker-01...
Fetching changes...
Removing .cache/
Removing .requirements/
Removing .serverless/
Removing .serverless_plugins
Removing lib/__init__.pyc
Removing lib/models/__init__.pyc
Removing lib/models/auth0_service.pyc
Removing lib/models/plan_service.pyc
Removing lib/models/subscription_service.pyc
Removing tests/__init__.pyc
Removing tests/__pycache__/
Removing tests/integration/__init__.pyc
Removing tests/integration/__pycache__/
Removing venv/
HEAD is now at 2102595 Merge branch 'pyup-update-boto3-1.5.4-to-1.5.6' into 'master'
Checking out 748e1d40 as master...
Skipping Git submodules setup
$ source setup_ci_environment
New python executable in /var/gitlab-builds/pod/sls-subscriptions/venv/bin/python
Installing setuptools, pip, wheel...done.
$ sls deploy -s staging
Serverless: Parsing Python requirements.txt
Serverless: Installing required Python packages for runtime python2.7...
Serverless: Linking required Python packages...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Unlinking required Python packages...
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-get-subscription"
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-create-subscription"
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-list-subscriptions"
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-update-subscription"
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-cancel-subscription"
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-cancelTrialSubscription"
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-syncAuth0CanHaveTrialFlag"
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-list-plans"
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-get-plan"
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-get-estimates"
Serverless: Tracing ENABLED for function "sls-subscriptions-staging-validate-subscription-vat"
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (2.23 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
............................................................................
Serverless: Stack update finished...
from aws-xray-sdk-python.
gcampionpae
commented on July 18, 2024
After looking at it more today, it seems that I must have been deploying from my local computer with version .94 because now no matter what I do, I cannot deploy .95 with serverless and not get this error.
from aws-xray-sdk-python.
haotianw465
commented on July 18, 2024
I'm trying to reproduce this issue. Before that I would like to confirm a few things. Is https://serverless.com the one your are using for deploying to Lambda? Your Lambda runtime is Python2.7 is that correct?
from aws-xray-sdk-python.
gcampionpae
commented on July 18, 2024
yes we are using the Serverless Framework for deploying our functions and are on Version 1.20. Our runtime is Python2.7. Thanks for looking into this!!
from aws-xray-sdk-python.
haotianw465
commented on July 18, 2024
Sorry for the delay. I have some findings and I would like to share them with you.
In my mac I first ran virtualenv -p /usr/bin/python venv to create the python2.7 environment. Then put my lambda function code and installed all requirements. Then I packaged everything under venv/lib/python2.7/site-packages along with my function as a Lambda deployment package, as Lambda official document instructs. The function can run successfully. This is because when I ran command virtualenv by default it installed pip, wheels, setuptools under venv/lib/python2.7/site-packages. And the module pkg_resources also became available as it is distributed with setuptools.
Then I repeat the above step but just before packaging I explicitly deleted pkg_resources and deployed to Lambda. This time I got a similar message "Unable to import module 'main': No module named pkg_resources".
By looking at your CI setup script I don't see any issue. After the following commands
virtualenv venv
source venv/bin/activate
pip install -r requirements-dev.txt --quiet
you should have everything you need to build your Lambda deployment package.
So the question comes down to how Serverless does packaging. Looking at the output of sls deploy I can't see the packaging logic behind it. I would suggest two things. First is to download your deployed function code and unzip it. Check if any of the modules installed by virtualenv is present. It could be that Serverless omit those modules. The second thing you could do is to add setuptools to your requirements-dev.txt and see if that module is packaged. Please let me know if that works for you.
On the other side we will look into whether to revert the change on depending on pkg_resources at runtime or fix our setup.py to add this new dependency. But I can not guarantee the latter option works for you until we know more about Serverless packaging logic.
from aws-xray-sdk-python.
Related Issues (20)
- aws-xray-sdk 2.11.0 removes causes from exceptions HOT 1
- ERROR: cannot find the current segment/subsegment when segment is open and uploading file to s3. HOT 3
- Installing SDK without botocore and other transitive dependencies HOT 4
- Custom emitter based on boto3 creates an infinite loop in the SDK HOT 3
- Bug: nested subsegments don't work across threads HOT 4
- IndexError when using AWS X-Ray SDK with SQLAlchemy HOT 1
- EKSPlugin HOT 1
- Very rare ReferenceError HOT 2
- Support for psycopg3 HOT 2
- Link a Textract async operation with downstream process HOT 1
- Current tox versions do not like how testenv.passenv is set
- aws_xray_sdk.core.exceptions.exceptions.SegmentNameMissingException: Segment name is required. HOT 1
- Using psycopg2 connection_factory throws exceptions HOT 4
- Documentation for patch_all HOT 2
- Sampling configuration should discuss DefaultSampler vs. LocalSampler HOT 1
- Flask middleware errors when an earlier Flask extension throws an exception in a before_request method HOT 1
- Patched DB cursor and template may record outside of XRayMiddleware HOT 2
- Segment not propgated when subsegment metadata is sufficiently long HOT 4
- sqlalchemy_core patch errors for unencoded special characters in db url HOT 1
- Error when using AsyncContext in python 3.11.5 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-xray-sdk-python.