Code Monkey home page Code Monkey logo

magniv-core's Introduction

One line data science infra.

Build Python Docs Demo with Loom


Magniv is the easiest way to build and deploy data applications, pipelines, or cron-like jobs. A Python decorator based orchestration library at the core, Magniv allows data scientists and developers to schedule arbitrary functions in one line. Monitoring, CI/CD, and compute management all come out of the box using the Magniv in a fully-managed context.

Quickstart ⚡

To get Magniv running with a simple "Hello, World" example, follow these three steps:

1. Install Magniv from pip.

pip install magniv

2. Create /tasks/requirements.txt every Magniv project requires at least one requirements.txt

3. Run the code below as a Python script or in a Python notebook (or in a colab notebook).

from magniv.core import task


@task(schedule="@hourly")
def hello_world():
    print("Hello world")

Why Magniv?

Scheduling is not a new problem. Magniv is designed with data scientists and hackers in mind. No more DAG files and bash scripts. Observability and logging comes out of the box.

Monitoring

Using the Magniv dashboard makes it easy to monitor running jobs. Each Magniv task is displayed with its previous run information, logs, code snippets, and other details. Users also have the ability to disable tasks and manually trigger new runs.

Git CI/CD

Similar to tools like Heroku or Netlify, Magniv sets up a CI/CD pipeline from your GitHub repository. After connecting a Magniv workspace to a GitHub repo, every new commit will trigger a new build and update your tasks.

Code Quality

Before commits, please run:

black . --line-length=98

isort . --profile black --line-length=98

magniv-core's People

Contributors

jped avatar seanabreau avatar gusrasch avatar jzone3 avatar

Stargazers

Tonic avatar Charlie Cortial avatar  avatar Sigrid Jin (ง'̀-'́)ง oO avatar Ruslan Konviser avatar Andrejs Agejevs avatar Nikolaus Schlemm avatar  avatar Madhav Jha avatar Tucker Wash avatar Florian Hoppe avatar Kobi Felton avatar Vyatcheslav Mogilevsky avatar William Shiao avatar mahdi tavasoli avatar Alex Remedios avatar  avatar TM Data avatar Reuben Cummings avatar HarryZhu avatar Doug Hudgeon avatar  avatar Diogo Magalhães Machado avatar Chengxi Zang avatar  avatar Thomas Martin avatar guoxue avatar 爱可可-爱生活 avatar Dani El-Ayyass avatar  avatar Theodor Dimache avatar Mayur Bhangale avatar Chung-Ping Huang avatar Sriram Somasundaram avatar Kartik Modi avatar  avatar Kirill Klimov avatar Roman Tezikov avatar dongpil seo avatar Madson Dias avatar Hossein Taghi-Zadeh avatar vignesh avatar Edu Magá avatar Emre Sevinç avatar Suhaib Affan avatar Koushik Vikram avatar Jim Ryu avatar Xu Chao avatar Chris avatar Matt Owen avatar Junbum Lee avatar Dhvanil Shah avatar Nikita avatar Davide Uberti avatar Maksim Gramin avatar  avatar Edward Bunt avatar  avatar Martin Skow Røed avatar Alexandar Dimcevski avatar Xinbin Huang avatar Moein Bbp avatar David Gidwani avatar Leonardo avatar George Ho avatar Arunprasad Rajkumar avatar  avatar Gabriel Almeida avatar Noah Frazier-Logue avatar  avatar  avatar Sachin Kesiraju avatar Chris Goddard avatar  avatar Michael Salaverry avatar  avatar

Watchers

 avatar Katie Hemus avatar

magniv-core's Issues

Verify that the resource requests are valid

Description

Right now we verify if the resources that are being requested are resources that we support; however, we do not check if the amount is valid.

The request needs to be valid based on this

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

LICENSE needed

Description

Maybe it's better to include a license for this repo.

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Cron Tab Error Handling Seems to be too Restrictive

Describe the bug

The following CRONs raise an error
0 0 * * SUN
30 20 * * 7

Based on the source code for Apache Airflow (which is what we are using in the backend), these should be allowed.
The way Airflow does it
is to check using croniter:

try:
      dag.is_subdag = False
      self.bag_dag(dag, parent_dag=dag, root_dag=dag)
      if isinstance(dag.normalized_schedule_interval, six.string_types):
          croniter(dag.normalized_schedule_interval)
      found_dags.append(dag)
      found_dags += dag.subdags
except (CroniterBadCronError,
              CroniterBadDateError,
              CroniterNotAlphaError) as cron_e:
      self.log.exception("Failed to bag_dag: %s", dag.full_filepath)
      self.import_errors[dag.full_filepath] = \
          "Invalid Cron expression: " + str(cron_e)
      self.file_last_changed[dag.full_filepath] = \
          file_last_changed_on_disk

Is there an existing issue for this?

  • I have searched the existing issues

Reproduction

Try
0 0 * * SUN
30 20 * * 7

Screenshot

No response

Logs

Traceback (most recent call last):
  File "tasks/main.py", line 30, in <module>
    def friendship_creation():
  File "/Users/jonathanpedoeem/opt/anaconda3/envs/.vybe/lib/python3.8/site-packages/magniv/core/task.py", line 84, in wrapper
    task_instance = Task(function, schedule=schedule, description=description, key=key)
  File "/Users/jonathanpedoeem/opt/anaconda3/envs/.vybe/lib/python3.8/site-packages/magniv/core/task.py", line 24, in __init__
    raise ValueError("{} is not a valid cron schedule".format(schedule))
ValueError: 0 0 * * SUN is not a valid cron schedule

System Info

magniv version 0.1.40

Severity

serious, but I can work around it

Nested requirements.txt logic is not fully complete

Describe the bug

We should be looking for a requirements.txt in the folder of the task, if it does not exist there we go up one parent directory until the root tasks directory.

What is currently happening is that it looks for a requirements.txt in the current directory and then if it cant find it looks for one in the root ./tasks directory.

It also makes the requirements.txt in the root directory a requirement which it doesn't necessarily have to be.

Is there an existing issue for this?

  • I have searched the existing issues

Reproduction

You can create a tasks structure like this:

tasks
|__dir1
     |-requirements.txt
     |_dir2
        |_tasks.py

It will raise an error bc there is no root requirements.txt which it shouldn't, and then if you add a root req file it will use that for the tasks in tasks.py when it should be using ./tasks/dir1/requirements.txt

Screenshot

No response

Logs

None

System Info

Current

Severity

serious, but I can work around it

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.