saagie / api-saagie Goto Github PK

Describe the bug
There is no consistency on the instance information we can get as we go by the job or by the instance.
When I called job.get_instance, it didn't return the start date and end date of the instance.
When I called job.get_info, I had the dates but not the version information.

Expected behavior
I want the same level of information on the instance as I use one or another method.

Timezone can now be configured in Job & Pipeline scheduling

See https://gitlab.com/saagie-group/product/product/-/blob/master/projects-and-jobs/server/src/main/resources/schema.graphqls

Update saagie api accordingly

Based on https://github.com/saagie/gradle-saagie-dataops-plugin/wiki/projectsCreateGraphPipeline

Describe the bug
A clear and concise description of what the bug is.

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

This is a test

To ensure maintainability of the current codebase, the following refactoring should be done

• Remove the manager(V1) code
• All the code under projects should be moved directly under the saagieapi directory
• Code in the auth.py file should be put in saagieapi.py file
• Code regarding jobs, pipelines, projects should be separated by functionality, same for the gql queries. The new code structure would be something like :

saagieapi
    saagie_api
    pipelines
        pipelines.py
        gql.py
    jobs
    apps
    projects
    env_vars
    docker_credentials

This would mean creating new classes for each main functionnality (pipelines, jobs...) that inherits the SaagieApi main class.
Let's take an example with the create_job function that would be now in the jobs.py file, and named create so that calling this function would mean calling saagieapi.jobs.create. The original create_job in saagieapi would be tagged as deprecated and will call the new function.

As in the doc of gql : https://gql.readthedocs.io/en/stable/usage/variables.html
We can use variables with $ instead of doing string.format()
See gql_create_graph_pipeline in gql_template for example

Currently editing a project is not possible

Currently, when creating a job, only the technology name is specified. If this technology name is used in different catalogs, the saagie-api will get the first one available.
It would be nice to be able to specify the catalog (if no catalog specified, take the official) to deal with that.

Change the graphql schema for the test

When creating a project, only one group can be assigned to it. It should be possible to specify a list with multiple groups and their corresponding roles on the project.

When using get_project_web_apps to get the list of applications, if the technology no longer exists, the function give an error instead of the list.

Is your feature request related to a problem? Please describe.
In project's jobs list, get the current version

When I tried to list the app of a project, I got the following error message :
TransportQueryError: {'message': 'io.saagie.projectsandjobs.domain.exception.NonExistingTechnologyException: Technology <technology_id> does not exist', 'locations': [], 'extensions': {'classification': 'DataFetchingException'}}
In Saagie UI, I saw that one of the app has a deleted technology. In the developper console, I saw there is 2 results for the graphql request: an error and some data.
It will be great if the package can do the same.

The correction can be performed in utils>gqlclient.py>execute() link this:
except TransportQueryError as transport_error: #if data not null #return data and print error or warning to show the error message print(transport_error.data)

• create pipeline
• get pipeline instance information
• create job without archive

tester et documenter le endpoint "/api/open/refresh" pour prolonger la vie d'un token

get the project id by the project name

For jobs, pipelines and apps, add a create_or_upgrade method that will check if a given job, app or pipeline already exists inside a project (based on its name) and decide if it needs to be created or updated

Adding deprecation warnings for deprecated warnings such as linear pipelines

Upgrade to gql 3.0 that now supports file uploads (instead of using requests library)

apps.get_info does not return infoirmation about current instance id

This method should return a list of instances for a given app (with a max instance parameter, as it's done for the jobs and pipelines)

Describe the bug
There is no option to configure the print in the different methods

Expected behavior
A parameter in all methods which prints results to configure it, and a global parameter to turn everything on or off

If a try to add a job with a technology not allowed inside my project/technology, the create_job fiunction does not return an error, and the API returns
{'errors': [{'message': 'Job not valid', 'extensions': {'technology': 'not exists', 'classification': 'ValidationError'}}], 'data': None}

Update jobs doesn't work from version 0.8.4 and up

Project name used for integration tests is always the same and if an integration test is launched while another one is already running, it will fail.

Jobs can now be configured with CPU & memory requests/limits

See https://gitlab.com/saagie-group/product/product/-/blob/master/projects-and-jobs/server/src/main/resources/schema.graphqls

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Currently, it's only possible to edit the following parameters of a job :
job_name
description
is_scheduled
cron_scheduling
schedule_timezone
resources

It should also be possible to upgrade a job (creating a now job version) :
runtime_version,
command_line
release_note
file

The upgrade should be able by passing either :
-the name of the job and its project (name is unique inside a project)
-the id of the job and its project

Add retry number as parameter of RequestsHTTPTransport

Improve logging:

• have the same logging format everywhere
• have info logs when jobs, apps, pipelines....are successfully created
• in jobs with callbacks, remove logs every x seconds an replace them wuith a status in the console only

Impossible to launch the edit_pipeline function when one parameters changed between 2 versions of this pipeline.
Example : if we didn't set the "alerting" parameter in the first version of the pipeline and we set this parameter to None in the second version, the function doesn't work.

Add get, create, update, delete operations for dockerCredentials

Add the features for creating an app :

• creating an app from a pre-configured app (mlflow, jupyter....)
• creating a custom app by entering docker information

Add a new parameter for the function create_project in order to have technology of other catalogs

Handle alerting for jobs, pipelines and apps

When the project creation is taking too much tiome on Saagie, the integration tests are failing without explicit messages.
To do :

• increase the timeout
• when timeout is reached, raise a Timeout error with an explicit message

Add requests_toolbelt librairy as dependencies

requests_toolbelt is needed when we import the package saagieapi, it will be better if we include this package in the dependencies

Is your feature request related to a problem? Please describe.
Our platform got a breaking change in its API. I wanted to know which version of saagieapi I should use, but the release description does not specify which version of Saagie is compatible with the releases of saagieapi.

Describe the solution you'd like
In each release note, it should be specified which version(s) of Saagie is supported.
Additionnaly, saagieapi package should check directly with the Saagie platform if it is compatible.
For example, When creating an object from SaagieApi class, the init should query /version (or something similar) to check if the version of the API is handled by the package. If not, raise Exception with specific message.

Function to return pipeline ID with pipeline name argument.

get_pipeline_id(self, project_id, pipeline_name) :
"""
"""
return id

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
Add field instances in the query GQL_GET_PIPELINE

To ensure consistency on code convention, code linter (isort and black) should be run before every commit made on this repository.

Apps available within a project can now be configured

See https://gitlab.com/saagie-group/product/product/-/blob/master/projects-and-jobs/server/src/main/resources/schema.graphqls

Is your feature request related to a problem? Please describe.
Possibility to import/export a project to a new platform

Is your feature request related to a problem? Please describe.
Currently, there is no feature to change an environment variable

Describe the solution you'd like
Add two methods to up either global or porject environment variable

Import projects, jobs, pipelines, envs and apps from json files

It should be possible to have a folder containing json with jobs and pipelines definitions and a function that gets this definitions and create the corresponding jobs and pipelines (or update them if they exist)

Severals functions use instances_limit as parameter in order to limit the number of instance in results. This parameter could be optional : if specified, you request the n instances; if not, you request all the instances