Comments (7)
sgotti
commented on May 27, 2024
@milosivanovic since it's just probably something looping forever to see what's happening you should provide a stack trace of the processes (SIGQUIT) and perhaps the logs of when this is happening. Syscalls aren't useful in such case.
from agola.
milosivanovic
commented on May 27, 2024
Sure - here is the raw output after sending SIGQUIT to the main process which was running for just a few seconds and had the 120% CPU issue:
from agola.
milosivanovic
commented on May 27, 2024
I re-ran agola for 8 seconds with debug: true set in each service config. The CPU immediately spiked to 120% (as expected) and agola produced a small chunk of log output to stdout every 2 seconds:
comet ~/d/agola # ./bin/agola serve --config /root/swarm/agola/config_nondocker.yml --components all-base,executor
2024-02-15T22:43:57.134413215-08:00 DBG internal/services/notification/webhooks.go:132 > webhookCleanerLoop
2024-02-15T22:43:57.134659182-08:00 DBG internal/services/executor/executor.go:1089 > executorStatusSenderLoop
2024-02-15T22:43:57.134613827-08:00 DBG internal/services/notification/commitstatus.go:138 > commitStatusesCleanerLoop
2024-02-15T22:43:57.134726757-08:00 DBG internal/services/executor/executor.go:1106 > executorTasksStatusSenderLoop
2024-02-15T22:43:57.134820864-08:00 DBG internal/services/executor/executor.go:1142 > tasksUpdater
2024-02-15T22:43:57.134728233-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:43:57.134824973-08:00 DBG internal/services/executor/executor.go:1270 > tasksDataCleaner
2024-02-15T22:43:57.134825894-08:00 DBG internal/services/executor/executor.go:1576 > tasksTimeoutCleaner
2024-02-15T22:43:57.135324643-08:00 DBG internal/services/runservice/scheduler.go:1345 > fetcher
2024-02-15T22:43:57.135325424-08:00 DBG internal/services/runservice/scheduler.go:1495 > finished run archiver loop
2024-02-15T22:43:57.135344569-08:00 DBG internal/services/runservice/scheduler.go:1361 > fetcher
2024-02-15T22:43:57.135351647-08:00 DBG internal/services/runservice/scheduler.go:1511 > finished run archiver
2024-02-15T22:43:57.135353153-08:00 DBG internal/services/runservice/scheduler.go:1658 > objectsCleaner
2024-02-15T22:43:57.135360984-08:00 DBG internal/services/runservice/scheduler.go:1450 > runsSchedulerLoop
2024-02-15T22:43:57.135397426-08:00 DBG internal/services/runservice/scheduler.go:1466 > runsScheduler
2024-02-15T22:43:57.135363169-08:00 DBG internal/services/runservice/scheduler.go:1599 > cacheCleaner
2024-02-15T22:43:57.135359596-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:43:57.13536822-08:00 DBG internal/services/runservice/scheduler.go:970 > runTasksUpdater
2024-02-15T22:43:57.135413945-08:00 DBG internal/services/runservice/scheduler.go:986 > runTasksUpdater
2024-02-15T22:43:57.135353614-08:00 DBG internal/services/runservice/scheduler.go:1641 > logCleanerLoop
2024-02-15T22:43:57.135452566-08:00 DBG internal/services/runservice/scheduler.go:1658 > objectsCleaner
2024-02-15T22:43:57.135505688-08:00 INF internal/services/runservice/runservice.go:43 > maintenance mode watcher: maintenance mode enabled: false
2024-02-15T22:43:57.135547184-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:43:57.136755848-08:00 DBG internal/services/executor/executor.go:1169 > ets: []*types.ExecutorTask{}
2024-02-15T22:43:57.136923357-08:00 INF internal/services/configstore/configstore.go:44 > maintenance mode watcher: maintenance mode enabled: false
2024-02-15T22:43:57.136951925-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:43:57.137265074-08:00 DBG internal/services/executor/executor.go:734 > send executor status: &types.ExecutorStatus{
ListenURL: "http://192.168.88.3:4001",
Archs: []types.Arch{
"amd64",
},
Labels: map[string]string{},
AllowPrivilegedContainers: false,
ActiveTasksLimit: 2,
ActiveTasks: 0,
Dynamic: false,
ExecutorGroup: "515ba71a-a58a-4d58-b699-f77d060acc1a",
SiblingsExecutors: []string{
"515ba71a-a58a-4d58-b699-f77d060acc1a",
},
}
2024-02-15T22:43:58.136955321-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:43:58.136976599-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:43:58.137037368-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:43:58.13734305-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:43:59.13493095-08:00 DBG internal/services/executor/executor.go:1270 > tasksDataCleaner
2024-02-15T22:43:59.134930006-08:00 DBG internal/services/executor/executor.go:1576 > tasksTimeoutCleaner
2024-02-15T22:43:59.13492976-08:00 DBG internal/services/executor/executor.go:1106 > executorTasksStatusSenderLoop
2024-02-15T22:43:59.135540026-08:00 DBG internal/services/runservice/scheduler.go:1345 > fetcher
2024-02-15T22:43:59.135573187-08:00 DBG internal/services/runservice/scheduler.go:1361 > fetcher
2024-02-15T22:43:59.1361108-08:00 DBG internal/services/runservice/scheduler.go:1495 > finished run archiver loop
2024-02-15T22:43:59.136138183-08:00 DBG internal/services/runservice/scheduler.go:1511 > finished run archiver
2024-02-15T22:43:59.136899799-08:00 DBG internal/services/executor/executor.go:1142 > tasksUpdater
2024-02-15T22:43:59.136898747-08:00 DBG internal/services/runservice/scheduler.go:1450 > runsSchedulerLoop
2024-02-15T22:43:59.136961853-08:00 DBG internal/services/runservice/scheduler.go:1466 > runsScheduler
2024-02-15T22:43:59.137112258-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:43:59.137111236-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:43:59.137170212-08:00 DBG internal/services/executor/executor.go:1169 > ets: []*types.ExecutorTask{}
2024-02-15T22:43:59.137140897-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:43:59.137784528-08:00 DBG internal/services/executor/executor.go:1089 > executorStatusSenderLoop
2024-02-15T22:43:59.139432447-08:00 DBG internal/services/executor/executor.go:734 > send executor status: &types.ExecutorStatus{
ListenURL: "http://192.168.88.3:4001",
Archs: []types.Arch{
"amd64",
},
Labels: map[string]string{},
AllowPrivilegedContainers: false,
ActiveTasksLimit: 2,
ActiveTasks: 0,
Dynamic: false,
ExecutorGroup: "515ba71a-a58a-4d58-b699-f77d060acc1a",
SiblingsExecutors: []string{
"515ba71a-a58a-4d58-b699-f77d060acc1a",
},
}
2024-02-15T22:43:59.139706655-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:00.137285332-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:00.137378263-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:00.137470329-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:00.141127454-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:01.134987177-08:00 DBG internal/services/executor/executor.go:1106 > executorTasksStatusSenderLoop
2024-02-15T22:44:01.1349885-08:00 DBG internal/services/executor/executor.go:1576 > tasksTimeoutCleaner
2024-02-15T22:44:01.134989389-08:00 DBG internal/services/executor/executor.go:1270 > tasksDataCleaner
2024-02-15T22:44:01.135794413-08:00 DBG internal/services/runservice/scheduler.go:1345 > fetcher
2024-02-15T22:44:01.135806459-08:00 DBG internal/services/runservice/scheduler.go:1361 > fetcher
2024-02-15T22:44:01.136311023-08:00 DBG internal/services/runservice/scheduler.go:1495 > finished run archiver loop
2024-02-15T22:44:01.136324857-08:00 DBG internal/services/runservice/scheduler.go:1511 > finished run archiver
2024-02-15T22:44:01.137071986-08:00 DBG internal/services/runservice/scheduler.go:1450 > runsSchedulerLoop
2024-02-15T22:44:01.137086227-08:00 DBG internal/services/runservice/scheduler.go:1466 > runsScheduler
2024-02-15T22:44:01.13723224-08:00 DBG internal/services/executor/executor.go:1142 > tasksUpdater
2024-02-15T22:44:01.137388638-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:01.137477751-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:01.13748769-08:00 DBG internal/services/executor/executor.go:1169 > ets: []*types.ExecutorTask{}
2024-02-15T22:44:01.137704348-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:01.139839714-08:00 DBG internal/services/executor/executor.go:1089 > executorStatusSenderLoop
2024-02-15T22:44:01.141516077-08:00 DBG internal/services/executor/executor.go:734 > send executor status: &types.ExecutorStatus{
ListenURL: "http://192.168.88.3:4001",
Archs: []types.Arch{
"amd64",
},
Labels: map[string]string{},
AllowPrivilegedContainers: false,
ActiveTasksLimit: 2,
ActiveTasks: 0,
Dynamic: false,
ExecutorGroup: "515ba71a-a58a-4d58-b699-f77d060acc1a",
SiblingsExecutors: []string{
"515ba71a-a58a-4d58-b699-f77d060acc1a",
},
}
2024-02-15T22:44:01.142895198-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:02.137454546-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:02.137522325-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:02.13775333-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:02.144428202-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:03.135157315-08:00 DBG internal/services/executor/executor.go:1270 > tasksDataCleaner
2024-02-15T22:44:03.13515986-08:00 DBG internal/services/executor/executor.go:1576 > tasksTimeoutCleaner
2024-02-15T22:44:03.135160018-08:00 DBG internal/services/executor/executor.go:1106 > executorTasksStatusSenderLoop
2024-02-15T22:44:03.135953122-08:00 DBG internal/services/runservice/scheduler.go:1345 > fetcher
2024-02-15T22:44:03.135979465-08:00 DBG internal/services/runservice/scheduler.go:1361 > fetcher
2024-02-15T22:44:03.136412754-08:00 DBG internal/services/runservice/scheduler.go:1495 > finished run archiver loop
2024-02-15T22:44:03.136424991-08:00 DBG internal/services/runservice/scheduler.go:1511 > finished run archiver
2024-02-15T22:44:03.137201763-08:00 DBG internal/services/runservice/scheduler.go:1450 > runsSchedulerLoop
2024-02-15T22:44:03.137211015-08:00 DBG internal/services/runservice/scheduler.go:1466 > runsScheduler
2024-02-15T22:44:03.13750215-08:00 DBG internal/services/executor/executor.go:1142 > tasksUpdater
2024-02-15T22:44:03.137595098-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:03.13762987-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:03.137645347-08:00 DBG internal/services/executor/executor.go:1169 > ets: []*types.ExecutorTask{}
2024-02-15T22:44:03.13782317-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:03.141838273-08:00 DBG internal/services/executor/executor.go:1089 > executorStatusSenderLoop
2024-02-15T22:44:03.143436024-08:00 DBG internal/services/executor/executor.go:734 > send executor status: &types.ExecutorStatus{
ListenURL: "http://192.168.88.3:4001",
Archs: []types.Arch{
"amd64",
},
Labels: map[string]string{},
AllowPrivilegedContainers: false,
ActiveTasksLimit: 2,
ActiveTasks: 0,
Dynamic: false,
ExecutorGroup: "515ba71a-a58a-4d58-b699-f77d060acc1a",
SiblingsExecutors: []string{
"515ba71a-a58a-4d58-b699-f77d060acc1a",
},
}
2024-02-15T22:44:03.146057005-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:04.137649644-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:04.137690427-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:04.137936275-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:04.147450947-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
2024-02-15T22:44:05.135229251-08:00 DBG internal/services/executor/executor.go:1576 > tasksTimeoutCleaner
2024-02-15T22:44:05.135229394-08:00 DBG internal/services/executor/executor.go:1270 > tasksDataCleaner
2024-02-15T22:44:05.135228478-08:00 DBG internal/services/executor/executor.go:1106 > executorTasksStatusSenderLoop
2024-02-15T22:44:05.136136144-08:00 DBG internal/services/runservice/scheduler.go:1345 > fetcher
2024-02-15T22:44:05.136160683-08:00 DBG internal/services/runservice/scheduler.go:1361 > fetcher
2024-02-15T22:44:05.136520172-08:00 DBG internal/services/runservice/scheduler.go:1495 > finished run archiver loop
2024-02-15T22:44:05.136545153-08:00 DBG internal/services/runservice/scheduler.go:1511 > finished run archiver
2024-02-15T22:44:05.137328518-08:00 DBG internal/services/runservice/scheduler.go:1450 > runsSchedulerLoop
2024-02-15T22:44:05.137352317-08:00 DBG internal/services/runservice/scheduler.go:1466 > runsScheduler
2024-02-15T22:44:05.137705609-08:00 DBG internal/services/executor/executor.go:1142 > tasksUpdater
2024-02-15T22:44:05.137743451-08:00 DBG internal/services/runservice/runservice.go:46 > maintenanceModeWatcherLoop
2024-02-15T22:44:05.137770757-08:00 DBG internal/services/configstore/configstore.go:47 > maintenanceModeWatcherLoop
2024-02-15T22:44:05.137868515-08:00 DBG internal/services/executor/executor.go:1169 > ets: []*types.ExecutorTask{}
2024-02-15T22:44:05.138083771-08:00 DBG internal/services/runservice/scheduler.go:849 > executorTasksCleaner
2024-02-15T22:44:05.143809241-08:00 DBG internal/services/executor/executor.go:1089 > executorStatusSenderLoop
2024-02-15T22:44:05.144960861-08:00 DBG internal/services/executor/executor.go:734 > send executor status: &types.ExecutorStatus{
ListenURL: "http://192.168.88.3:4001",
Archs: []types.Arch{
"amd64",
},
Labels: map[string]string{},
AllowPrivilegedContainers: false,
ActiveTasksLimit: 2,
ActiveTasks: 0,
Dynamic: false,
ExecutorGroup: "515ba71a-a58a-4d58-b699-f77d060acc1a",
SiblingsExecutors: []string{
"515ba71a-a58a-4d58-b699-f77d060acc1a",
},
}
2024-02-15T22:44:05.148948977-08:00 DBG internal/services/executor/executor.go:1033 > podsCleaner
^C
comet ~/d/agola #
from agola.
sgotti
commented on May 27, 2024
@milosivanovic Thanks for the stack trace. I think I know what's going on and will open a PR soon.
from agola.
sgotti
commented on May 27, 2024
Should be fixed by #487
from agola.
milosivanovic
commented on May 27, 2024
@sgotti wow, thank you so much - I checked out the #487 PR locally and confirmed that it fixed the bug!
from agola.
sgotti
commented on May 27, 2024
@milosivanovic Thanks for reporting and testing. Merged #487.
from agola.
Related Issues (20)
- Access Run ID from task/step HOT 2
- What happened to the DockerHub images? HOT 1
- Optional authentication for internal services HOT 1
- Clone step: use internal git HOT 1
- notification service: implement logic to start from the last run event triggered HOT 2
- notification service: save webhook and deliverys HOT 2
- Optimize run scheduling algorithm for improved CI/CD efficiency
- gateway service: order projects by name
- Add pagination to list APIs. HOT 10
- notification: add apis to manage run webhooks retry logic
- Organization rules for runs start/stop HOT 10
- notification service: save commit status and deliveries
- Global view of running runs HOT 3
- Support obtaining OIDC ID token from Git server
- variables and secrets are not loaded in building environment HOT 8
- More convenient to use HOT 1
- Couldnt get examples/agolademo to pull from docker HOT 1
- Checkout git repo fully, without switching to specific commit HOT 1
- External git repos in workflows HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from agola.