Comments (15)
Okay so with the combination of @bigadsoleiman fix and also changing langchain==0.0.288 everywhere, I was able to deploy the Kendra and Opensearch stacks successfully. PGVector still fails with the size > 10 GB. Also changed the Dockerfile for Opensearch to the file below with @koushal2018 suggestion
FROM public.ecr.aws/lambda/python:3.11
RUN yum update -y && \
yum install -y git wget gcc rustc rustup cargo libxml2-devel libxslt-devel libmagic-dev poppler-utils tesseract-ocr libreoffice pandoc && \
yum clean all && \
rm -rf /var/cache/yum
COPY dependencies/* ./
COPY requirements.txt ./
COPY index.py ./
RUN pip install --upgrade pip && \
pip install --no-cache-dir unstructured[feature1,feature2] && \
pip install --no-cache-dir -r requirements.txt --upgrade
RUN python -c "import nltk;nltk.download('punkt', download_dir='/home/sbx_user1051/nltk_data')" && \
python -c "import nltk;nltk.download('averaged_perceptron_tagger', download_dir='/home/sbx_user1051/nltk_data')"
CMD ["index.lambda_handler"]
from aws-genai-llm-chatbot.
Hi, i enabled only pgvector RAG source and the deployment is stuck during the publish process. Have you experienced something similar? It just takes hours and then runs into a timeout. It does retrying but without success..
it could be maybe because of the large size of the container?
UPDATE:
I switched to a faster internet connection and the deployment went through. It took 30 min and there are 3 containers in ECR the largest has 6150.83 MB but this is only for the pgvector RAG. I have not tried out the open search vector stuff
from aws-genai-llm-chatbot.
I hit the same error when attempting to use OpenSearch:
34/53 Currently in progress: AwsGenaiLllmChatbotStack, OpenSearchVectorSearchApiHandler6B246D88, OpenSearchVectorSearchCreateIndex7EED590A, OpenSearchVectorSearchIndexDocument36B9A569
AwsGenaiLllmChatbotStack | 34/53 | 2:52:03 PM | CREATE_FAILED | AWS::Lambda::Function | OpenSearchVectorSearch/IndexDocument (OpenSearchVectorSearchIndexDocument36B9A569) Resource handler returned message: "Lambda function AwsGenaiLllmChatbotStack-OpenSearchVectorSearchInd-7ZOZdnMIEY1f reached terminal FAILED state due to InvalidImage(SizeLimitExceeded: Uncompressed container image size exceeds 10 GiB limit) and failed to stabilize" (RequestToken: f8e7a35a-da2e-bc26-b3d6-af6c6f55add2, HandlerErrorCode: NotStabilized)
AwsGenaiLllmChatbotStack | 34/53 | 2:52:04 PM | CREATE_FAILED | AWS::Lambda::Function | OpenSearchVectorSearch/ApiHandler (OpenSearchVectorSearchApiHandler6B246D88) Resource creation cancelled
AwsGenaiLllmChatbotStack | 34/53 | 2:52:04 PM | CREATE_FAILED | AWS::Lambda::Function | OpenSearchVectorSearch/CreateIndex (OpenSearchVectorSearchCreateIndex7EED590A) Resource creation cancelled
Any clues about what's happening? I assume one of the packages has grown recently, and that's causing the build to exceed 10 GB...
from aws-genai-llm-chatbot.
I also hit the same error when creating PGVector during deplyment due to the container size (over 10GiB).
1:57:39 PM | CREATE_FAILED | AWS::Lambda::Function | AuroraPgVector/Doc...g/DocumentIndexing Resource handler returned message: "Lambda function AwsGenaiLllmChatbotStack-AuroraPgVectorDocumentInd-GSLQpWzr9KTr reached terminal FAILED state due to InvalidImage(SizeL imitExceeded: Uncompressed container image size exceeds 10 GiB limit) and failed to stabilize" (RequestToken: aba5d2f6-c030-ba0a-d5e9-189946108dbc, HandlerErrorCode: NotSt abilized)
from aws-genai-llm-chatbot.
Same error.
Resource handler returned message: "Lambda function AwsGenaiLllmChatbotStack-OpenSearchVectorSearchInd-jlhdiMLnzFoE reached terminal FAILED state due to Inv
alidImage(SizeLimitExceeded: Uncompressed container image size exceeds 10 GiB limit) and failed to stabilize" (RequestToken: c9434bc1-73a0-2b42-b9a5-9ab1993
71783, HandlerErrorCode: NotStabilized)
from aws-genai-llm-chatbot.
same issue
from aws-genai-llm-chatbot.
FYI, a colleague of mine found that locking to previous versions is a workaround:
aws_lambda_powertools
aws_xray_sdk
boto3-1.28.21-py3-none-any.whl
botocore-1.31.21-py3-none-any.whl
langchain==0.0.288
opensearch-py
requests_aws4auth
unstructured[all-docs]==0.10.14
However, I will say that this is NOT a good solution, since you never get later versions of the libraries in question. But if you need to get something working right away, this might help.
from aws-genai-llm-chatbot.
Thanks @gregsomm.
unstructured paired with langchain is breaching the current containerised lambda limit size of 10G.
In the next release we expect to publish in a few days we are moving the workflow for handling documents with unstructured away from lambda so that the solution can keep benefit from future features and new document types support.
Until then locking the version is the way to go
from aws-genai-llm-chatbot.
Hi @bigadsoleiman I tried pulling the latest commit and building it. It still fails when deploying the pgvector stack with the error Lambda function AwsGenaiLllmChatbotStack-AuroraPgVectorDocumentInd-m9rbn5M19Wdn reached terminal FAIL ED state due to InvalidImage(SizeLimitExceeded: Uncompressed container image size exceeds 10 GiB limit) and failed to stabilize" (Request Token: 7a72dca0-4d75-d02e-a14e-f3163f4d4478, HandlerErrorCode: NotStabilized)
from aws-genai-llm-chatbot.
Okay so with the combination of @bigadsoleiman fix and also changing langchain==0.0.288 everywhere. Also changed the Dockerfile for Opensearch to
FROM public.ecr.aws/lambda/python:3.11 RUN yum update -y && \ yum install -y git wget gcc rustc rustup cargo libxml2-devel libxslt-devel libmagic-dev poppler-utils tesseract-ocr libreoffice pandoc && \ yum clean all && \ rm -rf /var/cache/yum COPY dependencies/* ./ COPY requirements.txt ./ COPY index.py ./ RUN pip install --upgrade pip && \ pip install --no-cache-dir unstructured[feature1,feature2] && \ pip install --no-cache-dir -r requirements.txt --upgrade RUN python -c "import nltk;nltk.download('punkt', download_dir='/home/sbx_user1051/nltk_data')" && \ python -c "import nltk;nltk.download('averaged_perceptron_tagger', download_dir='/home/sbx_user1051/nltk_data')" CMD ["index.lambda_handler"]
@Rionel-Dmello did this resolve the size issue or were you just clarifying this is what you used that generated the error?
from aws-genai-llm-chatbot.
@QuinnGT Thanks! Edited my original post, but this helped me deploy both the Kendra and the Opensearch stacks. PGVector still throws an error.
from aws-genai-llm-chatbot.
Same. for pgVector & version freeze (and git pull master to latest with freeze) container is > 10GB, rollback:
Lambda function AwsGenaiLllmChatbotStack-AuroraPgVectorDocumentInd-ybhM3EDcYGDg reached terminal FAILED state due to InvalidImage(SizeLimitExceeded
: Uncompressed container image size exceeds 10 GiB limit)
from aws-genai-llm-chatbot.
Same issue as the above reports
Lambda function AwsGenaiLllmChatbotStack-AuroraPgVectorDocumentInd-27OdlB7gdQ0g reached terminal FAILED state due to InvalidImage(SizeLimitExceeded: Uncompressed container image size exceeds 10 GiB limit) and failed to stabilize
from aws-genai-llm-chatbot.
I can confirm that changing both (like @Rionel-Dmello mentioned), the requrements.txt and the Dockerfile, fixes the issue for me as well
Okay so with the combination of @bigadsoleiman fix and also changing langchain==0.0.288 everywhere, I was able to deploy the Kendra and Opensearch stacks successfully. PGVector still fails with the size > 10 GB. Also changed the Dockerfile for Opensearch to the file below with @koushal2018 suggestion
FROM public.ecr.aws/lambda/python:3.11 RUN yum update -y && \ yum install -y git wget gcc rustc rustup cargo libxml2-devel libxslt-devel libmagic-dev poppler-utils tesseract-ocr libreoffice pandoc && \ yum clean all && \ rm -rf /var/cache/yum COPY dependencies/* ./ COPY requirements.txt ./ COPY index.py ./ RUN pip install --upgrade pip && \ pip install --no-cache-dir unstructured[feature1,feature2] && \ pip install --no-cache-dir -r requirements.txt --upgrade RUN python -c "import nltk;nltk.download('punkt', download_dir='/home/sbx_user1051/nltk_data')" && \ python -c "import nltk;nltk.download('averaged_perceptron_tagger', download_dir='/home/sbx_user1051/nltk_data')" CMD ["index.lambda_handler"]
from aws-genai-llm-chatbot.
We released a new version v.3.0.0
in which document ingestion is handled with AWS Batch instead of AWS Lambda to accomodate growing dependencies and avoid possible lambda timeout issues.
V2 won't be supported so we suggest to move to V3 for a more long term solution along with several new features around RAG.
from aws-genai-llm-chatbot.
Related Issues (20)
- Fatal error during cdk deploy: selectedItems is declared but its value is never read.
- Lack of Knowledge Database Support in Vision Mode for Bedrock Claude 3 HOT 2
- Add Amazon Q Knowledge base as a RAG Engine
- "Uploaded file must be a non-empty zip HOT 2
- how to run the deployment 'npx cdk deploy' without additional confirmation HOT 6
- command line arguments for 'cloud9-resize.sh' script HOT 6
- S3 Bucket Name is Empty when deploying UI Bucket HOT 4
- PDF upload fails with spaces in filename HOT 2
- Error: Command failed: npm --silent --prefix "/home/ubuntu/environment/aws-genai-llm-chatbot/lib/user-interface/react-app" ci HOT 1
- current credentials could not be used to assume the role HOT 1
- Feature Request: Private Workspaces HOT 1
- npx cdk deploy fails when trying to create common shared layer HOT 4
- Add interleaving for multi-source retrieval
- Website documents (crawled) do not have titles in the metadata
- previously working HF sagemaker based mistral models are now failing
- OpenAI basic prompts are not returning a response HOT 2
- Typo? Should this be 'npm run create'? HOT 1
- Installer does not prompt for Secrets Manager Secret ARN containing HF API token HOT 2
- Custom adaptor doesnt stream when RAG is enabled. Streaming is okay when no workspace is selected.
- Condition disappear for Idefics activation HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-genai-llm-chatbot.