Comments (5)
@egormcobakaster This seems to indicate the environment in which the clearml-agent running your pipeline is deployed is not properly configured. Where are you running this clearml-agent? Did you complete clearml-agent init
properly?
from clearml-server.
@egormcobakaster This seems to indicate the environment in which the clearml-agent running your pipeline is deployed is not properly configured. Where are you running this clearml-agent? Did you complete
clearml-agent init
properly?
i am running clearml-agent on the same machine as the clearml-server.
when I start a new agent with a new queue:
clearml-agent daemon --queue 6c86514d67014415967bc1d319f03fac
this error disappears and individual tasks are launched from the ui, but when I start pipline, the first task gets queued and does not leave the queue
from clearml-server.
Hi @egormcobakaster, Can you share the log of the pipeline task and your pipeline code?
Also, do you only have a single clearml-agent running? and what is the queue name it listens to?
from clearml-server.
Hi @jkhenning, pipeline log:
Environment setup completed successfully
Starting Task Execution:
ClearML results page: http://172.21.0.98:8080/projects/6072ec75526e493f917e5e770f24319d/experiments/abf2370a46bc4844984d98643e995ff4/output/log
ClearML pipeline page: http://172.21.0.98:8080/pipelines/6072ec75526e493f917e5e770f24319d/experiments/abf2370a46bc4844984d98643e995ff4
2023-12-11 10:03:05,217 - clearml.util - WARNING - 2 task found when searching for {'project_name': 'data process', 'task_name': 'Pipeline step 2 create clearml dataset', 'include_archived': True, 'task_filter': {'status': ['created', 'queued', 'in_progress', 'published', 'stopped', 'completed', 'closed']}}
2023-12-11 10:03:05,217 - clearml.util - WARNING - Selected task Pipeline step 2 create clearml dataset
(id=adad180edd364cb1b8cedcb77e0a7712)
Launching the next 1 steps
Launching step [anotation]
Cloning Task id=8e7aac5e6f004730a0a3088f6fb0e327 with parameters: {'General/dataset_path': '/mnt/ext2/datasets/DataSet/Casia_images'}
Launching step: anotation
Parameters:
{'General/dataset_path': '${pipeline.path}'}
Configurations:
{}
Overrides:
{}
pipeline code:
from clearml import Dataset
import argparse
import sys
from clearml import Task
from clearml.automation import PipelineController
def pre_execute_callback_example(a_pipeline, a_node, current_param_override):
# type (PipelineController, PipelineController.Node, dict) -> bool
print(
"Cloning Task id={} with parameters: {}".format(
a_node.base_task_id, current_param_override
)
)
# if we want to skip this node (and subtree of this node) we return False
# return True to continue DAG execution
return True
def post_execute_callback_example(a_pipeline, a_node):
# type (PipelineController, PipelineController.Node) -> None
print("Completed Task id={}".format(a_node.executed))
# if we need the actual executed Task: Task.get_task(task_id=a_node.executed)
return
parser = argparse.ArgumentParser()
parser.add_argument('--path', default='', action='store',
help='path to dataset')
args = parser.parse_args()
if args.path == '':
print("empty path to dataset")
sys.exit()
pipe = PipelineController(
name="Pipeline demo", project="data process", version="0.0.1", add_pipeline_tags=False
)
pipe.add_parameter(
"path",
args.path,
"path_to_dataset",
)
pipe.set_default_execution_queue("default")
pipe.add_step(
name="anotation",
base_task_project="data process",
base_task_name="Pipeline step 1 create anotation",
parameter_override={"General/dataset_path": "${pipeline.path}"},
pre_execute_callback=pre_execute_callback_example,
post_execute_callback=post_execute_callback_example,
)
pipe.add_step(
name="create dataset",
parents=["anotation"],
base_task_project="data process",
base_task_name="Pipeline step 2 create clearml dataset",
parameter_override={
"General/dataset_path": "${pipeline.path}",
},
pre_execute_callback=pre_execute_callback_example,
post_execute_callback=post_execute_callback_example,
)
pipe.start()
print("done")
the first task only gets queued and is not executed:
from clearml-server.
@jkhenning , @ainoam Thanks for the answers, it helped me to create another queue. one for the pipeline and the other for tasks
from clearml-server.
Related Issues (20)
- Feature Request: Get server configuration parameters from AWS Secrets Manager [security]
- [Customising web-ui] - Projects are loading tasks in web ui of self hosting server but i want them to show datasets HOT 3
- generating clearml-reports HOT 13
- How to write artifacts to S3 from server side? HOT 1
- Nginx Not Loading Plotly.js Resource: ClearML Self-Hosted Docker HOT 7
- Failed Navigate From Overview to Experiments Details HOT 4
- Async Delete Always Failed when Removing Experiments (using Minio)
- nginx 0.6.x < 1.20.1 1-Byte Memory Overwrite RCE vulnerability HOT 2
- ElasticSearch UI and Redis UI? HOT 2
- The problem with scalars HOT 12
- Curl 7.69 < 8.4.0 Heap Buffer Overflow vulnerability HOT 2
- OpenSSL 1.1.1 < 1.1.1x Vulnerability HOT 1
- Elasticsearch image tag 7.17 does not exist HOT 4
- Git package is not installed by default in node:20-bookworm-slim HOT 1
- SERVER UNAVAILABLE HOT 4
- APP Credentials disapper in webapp HOT 20
- Scalar graphs legend is too narrow for experiments with long names HOT 9
- Update from 1.14.1 to 1.15.0 leads to several fatal issues when booting HOT 3
- AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. HOT 3
- Web server Ipv6 error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clearml-server.