Code Monkey home page Code Monkey logo

merlin-spellbook's Introduction

merlin-spellbook

Utilities for merlin workflows!

merlin-spellbook's People

Contributors

ben-bay avatar bobcorey avatar lucpeterson avatar nkeilbart avatar ryannova avatar rydtran avatar ymubarka avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

merlin-spellbook's Issues

Parallelize Collector

Upgrade collector to use concurrent futures and parallelize over loading conduit nodes

type annotations

  • Add type annotations to function calls
  • Add a 3rd party linter to makefile to check for them (eg mypy)

Reduce Verbiage in Serialize Command

The current serialize command can become verbose when converting a large serialized JSON string with nested subtrees into a JSON file. I propose adding an additional syntax to reduce the verbiage. Or even introduce a new command called deserialize that will convert a serialized tree into a JSON file.

Proposal:
  A new syntax added to serialize.
    or
  A new command called deserialize. I'm more inclined to this option since the name accurately describes the proposed behavior.

We will use the following JSON to demonstrate the differences;

{
  "people" : {
    "ryan" : {
      "age" : 23,
      "hobbies" : {
        "soccer" : "good",
        "running" : "bad"
      },
      "status" : "insomina"
    },
    "john" : {
      "age" : 45,
      "hobbies" : {
        "soccer" : "bad",
        "running" : "good"
      },
      "status": "sad"
    }
  }
}

Old Syntax:

spellbook serialize --output myfile.json --vars \
people/ryan/age=23 \
people/ryan/hobbies/soccer=good \
people/ryan/hobbies/running=bad \
people/ryan/status=insomania \
people/john/age=45 \
people/john/hobbies/soccer=bad \
people/john/hobbies/running=good \
people/john/status=sad\

New Syntax:

spellbook deserialize --output myfile.json \
people>\
ryan>age=23&hobbies>soccer=good&running=bad<status=insomania<\
john>age=45&hobbies>soccer=bad&running=good&status=sad\

Breaking Down the Syntax:

  • ">" tells the program that the next value to read is a child node.
  • "<" tells the program to return to the parent node.
  • "&" tells the program that the next value to read is a sibling node.

Things to Note:

  • After specifying "hobbies" and its child nodes, a "<" symbol is appended. The first instinct is to append "&," but a "<" symbol is needed to tell the program to return to the parent node of "hobbies." This will correctly place "status" as a sibling of "age" and "hobbies."
  • The optional variable "--vars" is not needed since the serialized tree is one continuous string with no spaces.

[BUG] make-samples with rounding triggers error in Merlin

🐛 Bug Report

Describe the bug
When running make-samples with rounding in a merlin workflow, the following error occurs:

[2023-03-21 11:36:34: INFO] Generating samples...
[2023-03-21 11:36:37: INFO] Generating samples complete!
[2023-03-21 11:36:37: INFO] Loading samples from 'samples.npy'...
[2023-03-21 11:36:37: ERROR] Object arrays cannot be loaded when allow_pickle=False

To Reproduce
Steps to reproduce the behavior:

  1. Create a merlin workflow and use the make-samples command with roudning in the "samples" step.
  2. Run merlin workflow in dry mode.

Expected behavior
Merlin should be able to load the samples.

Additional context
When not using the -round option, the workflow is able to load the samples.

I'm using the following packages:

merlin-spellbook==0.8.1
merlin==1.8.3 (not using 1.9.1 b/c it is unable to correctly resubmit itself when doing an iterative workflow)

Command
spellbook make-samples -dims 6 -n 8 -sample_type lhs -outfile=samples.npy -scale "[[-0.5,0.5],[-0.5,0.5],[430,452],[60,70],[50,60],[0,1]]" -seed 10 -round "[False,False,False,False,False,round]"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.