Comments (16)
bennnym
commented on June 5, 2024
2
GOOD NEWS!
I upped my agents resources as I thought maybe that could be a possible
issue... The agent was running with 512 CPU and 1024 memory.
I changed to 1024 CPU and 4096 memory.
The orchestration seems to be working again! Just ran 9 flows that were all
scheduled on the hour.
Maybe we can add something to the docs about this?
…On Fri, 18 Nov 2022, 4:30 pm Michael Adkins, ***@***.***> wrote:
Ah interesting. Why are they in Pending states instead of Scheduled, the
work queue should not even be returning these. We allow Scheduled ->
Pending transitions but not Pending -> Pending as that would indicate two
agents attempted to submit the same run at once.
—
Reply to this email directly, view it on GitHub
<#156 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALGEVNYMNR64MJCBD4AC253WI4H7VANCNFSM6AAAAAASECXGQQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from prefect-aws.
anna-geller
commented on June 5, 2024
2
Closing the issue because we clarified the issue and added docs, thanks for all the help here, Ben
from prefect-aws.
zanieb
commented on June 5, 2024
1
Ah interesting. Why are they in Pending states instead of Scheduled, the work queue should not even be returning these. We allow Scheduled -> Pending transitions but not Pending -> Pending as that would indicate two agents attempted to submit the same run at once.
from prefect-aws.
bennnym
commented on June 5, 2024
1
I dug a bit deeper to see if this was just a UI issue. I can confirm it is
not a UI issue. It is continuing to happen today.
…On Fri, 18 Nov 2022 at 16:30, Michael Adkins ***@***.***> wrote:
Ah interesting. Why are they in Pending states instead of Scheduled, the
work queue should not even be returning these. We allow Scheduled ->
Pending transitions but not Pending -> Pending as that would indicate two
agents attempted to submit the same run at once.
—
Reply to this email directly, view it on GitHub
<#156 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALGEVNYMNR64MJCBD4AC253WI4H7VANCNFSM6AAAAAASECXGQQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
*Benjamin Muller*
*m: *0415 382 245
*e: ***@***.***
from prefect-aws.
anna-geller
commented on June 5, 2024
can you share how did you configure your agent and ecs task infra block? this got a little too sparse than I hoped, not even prefect version showing up here 😅 can you add that? we would want to reproduce the issue first before we can fix it
from prefect-aws.
bennnym
commented on June 5, 2024
Prefect version 2.6.7 prefect-aws 0.1.8
Agent is
FROM prefecthq/prefect:2.6.7-python3.9
RUN pip install prefect-aws==0.1.8
Then I run the command prefect agent start -q 'prod'
from prefect-aws.
bennnym
commented on June 5, 2024
ECSTask block
def build_and_save_ecs_task(repo_name: str):
account_number = Sts.get_caller_identity()
ecs = ECSTask(
name=repo_name,
image=f"{account_number}.dkr.ecr.ap-southeast-2.amazonaws.com/prefect-{repo_name}:latest",
cpu=512,
memory=256,
**{
"stream_output": True,
"configure_cloudwatch_logs": True,
"cluster": "prefect-cluster",
"execution_role_arn": "arn:aws:iam::XXX:role/prefect-execution-role",
"task_role_arn": "arn:aws:iam::XXX:role/prefect-task-role",
"launch_type": "FARGATE",
"vpc_id": "vpc-XXX",
"env": {"PYTHONPATH": "$PYTHONPATH:.", "AWS_RETRY_MODE": "adaptive", "AWS_MAX_ATTEMPTS": "100"},
"task_customizations": [
{"op": "replace", "path": "/networkConfiguration/awsvpcConfiguration/assignPublicIp", "value": "DISABLED"},
{
"op": "add",
"path": "/networkConfiguration/awsvpcConfiguration/subnets",
"value": ["subnet-XXX", "subnet-XXX", "subnet-XXX"],
},
{
"op": "add",
"path": "/networkConfiguration/awsvpcConfiguration/securityGroups",
"value": ["sg-XXX"],
},
],
},
)
ecs.save(name=repo_name, overwrite=True)from prefect-aws.
zanieb
commented on June 5, 2024
@bennnym Do you have debug level agent logs?
from prefect-aws.
bennnym
commented on June 5, 2024
No I don't.
I am about to wait and see on the hour what happens and I will paste the agent logs here.
I think they are just info, if that is what it defaults to?
from prefect-aws.
bennnym
commented on June 5, 2024
For the record I have 8 flows scheduled to run at the hour, every hour.
from prefect-aws.
bennnym
commented on June 5, 2024
04:59:54.473 \| INFO \| prefect.agent - Submitting flow run
--
'1c168b64-c542-4ecc-be86-fec3bf879a39'
04:59:54.474 \| INFO \| prefect.agent - Submitting flow run
'56e37fcc-bf15-4c81-b05f-0955f1f44f58'
04:59:54.475 \| INFO \| prefect.agent - Submitting flow run
'8fa259e7-6d51-4e61-9a4f-79f959f93783'
04:59:54.475 \| INFO \| prefect.agent - Submitting flow run
'96242353-061f-43b0-927e-fe693b539ac5'
04:59:54.476 \| INFO \| prefect.agent - Submitting flow run
'a98eeec6-d794-4084-b7ed-a2f2462d1be2'
04:59:54.476 \| INFO \| prefect.agent - Submitting flow run
'ab9d2ee5-6132-417c-8575-48746bac4a7b'
04:59:54.477 \| INFO \| prefect.agent - Submitting flow run
'bb535ea6-ec31-474e-b0d2-b0d1d4ea50f6'
04:59:54.478 \| INFO \| prefect.agent - Submitting flow run
'c874d6e6-68e6-419a-bd32-14db60def23c'
05:00:00.013 \| INFO \| prefect.agent - Aborted submission of flow run
'a98eeec6-d794-4084-b7ed-a2f2462d1be2'. Server sent an abort signal: This run
cannot transition to the PENDING state from the PENDING state.
05:00:00.016 \| INFO \| prefect.agent - Aborted submission of flow run
'c874d6e6-68e6-419a-bd32-14db60def23c'. Server sent an abort signal: This run
cannot transition to the PENDING state from the PENDING state.
05:00:00.017 \| INFO \| prefect.agent - Aborted submission of flow run
'1c168b64-c542-4ecc-be86-fec3bf879a39'. Server sent an abort signal: This run
cannot transition to the PENDING state from the PENDING state.
05:00:00.027 \| INFO \| prefect.agent - Aborted submission of flow run
'ab9d2ee5-6132-417c-8575-48746bac4a7b'. Server sent an abort signal: This run
cannot transition to the PENDING state from the PENDING state.
05:00:00.031 \| INFO \| prefect.agent - Aborted submission of flow run
'56e37fcc-bf15-4c81-b05f-0955f1f44f58'. Server sent an abort signal: This run
cannot transition to the PENDING state from the PENDING state.
05:00:00.055 \| INFO \| prefect.agent - Aborted submission of flow run
'96242353-061f-43b0-927e-fe693b539ac5'. Server sent an abort signal: This run
cannot transition to the PENDING state from the PENDING state.
05:00:00.090 \| INFO \| prefect.agent - Aborted submission of flow run
'bb535ea6-ec31-474e-b0d2-b0d1d4ea50f6'. Server sent an abort signal: This run
cannot transition to the PENDING state from the PENDING state.
05:00:00.235 \| INFO \| prefect.agent - Aborted submission of flow run
'8fa259e7-6d51-4e61-9a4f-79f959f93783'. Server sent an abort signal: This run
cannot transition to the PENDING state from the PENDING state.
05:00:53.069 \| INFO \| prefect.agent - Submitting flow run
'38a2d94b-81f5-4689-9e08-a4025075221a'
from prefect-aws.
bennnym
commented on June 5, 2024
Of these 8 flows, 6 are now permanently in pending and 2 ran successfully
from prefect-aws.
bennnym
commented on June 5, 2024
I also get quite a few of these errors - but dont think they are related
05:10:41.627 \| ERROR \| prefect.agent - An error occured while monitoring flow
--
run '5e1eface-72b5-4040-9be3-fe8c2c1b585f'. The flow run will not be marked as
failed, but an issue may have occurred.
from prefect-aws.
bennnym
commented on June 5, 2024
I added an extra agent to see if that was the issue. George told me that I
can have multiple agents and queue's with the same name?
So maybe those error logs are unrelated to the issue?
…On Fri, 18 Nov 2022, 4:30 pm Michael Adkins, ***@***.***> wrote:
Ah interesting. Why are they in Pending states instead of Scheduled, the
work queue should not even be returning these. We allow Scheduled ->
Pending transitions but not Pending -> Pending as that would indicate two
agents attempted to submit the same run at once.
—
Reply to this email directly, view it on GitHub
<#156 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALGEVNYMNR64MJCBD4AC253WI4H7VANCNFSM6AAAAAASECXGQQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from prefect-aws.
bennnym
commented on June 5, 2024
So I reverted back to one agent, and I can confirm that the error in
regards to PENDING was between agents. I am not able to test the primary
issue again because I have had to run my prefect 1.0 stuff again so that
our business operates over the weekend.
I might ba able to get a POC of the issue on Monday morning.
…On Fri, 18 Nov 2022 at 16:30, Michael Adkins ***@***.***> wrote:
Ah interesting. Why are they in Pending states instead of Scheduled, the
work queue should not even be returning these. We allow Scheduled ->
Pending transitions but not Pending -> Pending as that would indicate two
agents attempted to submit the same run at once.
—
Reply to this email directly, view it on GitHub
<#156 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALGEVNYMNR64MJCBD4AC253WI4H7VANCNFSM6AAAAAASECXGQQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
*Benjamin Muller*
*m: *0415 382 245
*e: ***@***.***
from prefect-aws.
anna-geller
commented on June 5, 2024
btw Ben, you could leverage work queue concurrency limits here to ensure that those runs triggered simultaneously will be queued up without overwhelming the agent
from prefect-aws.
Related Issues (20)
- allow define only aws region name through AwsCredentials HOT 1
- Rate limiting on task registration using ECS Workpools
- Add a Lambda function block HOT 1
- Accept `None` as an argument to Launch Type for ECSTask with `publish_as_work_pool` HOT 1
- Make credentials optional for S3Bucket block HOT 2
- Task definition caching does not work if the task definitions come from separate deployments. HOT 2
- Error in version 0.4.8: TypeError: unhashable type: 'dict' HOT 3
- Feature Request: Support for ExtraArgs on S3Bucket
- Credential use examples are in correct in documentation HOT 6
- Add distinct style to links in docs
- ECS Works Pools Should Support Specifying Volumes for Flow Runs HOT 1
- `S3Bucket.copy_object`: Only resolve target path with self if `to_bucket` is not defined
- ECSTask block `publish_as_work_pool` does not set `network_configuration` HOT 2
- Worker mistakenly marks flow runs as Crashed HOT 1
- Improvement to ECS worker setup guide
- ECS worker: Updating family setting to template flow name/deployment
- External luanch type not supported
- Change logging prefix to avoid unnecessary task definition registrations
- Add Python 3.12 to test matrix
- ECS task definition ephemaralStorage overrides do not work
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prefect-aws.