Public facing API about pyblish-magenta HOT 21 CLOSED

I came to think of one more disadvantage to this approach, which is initial set-up.

This..

import pyblish_magenta
pyblish_magenta.setup()

Would need to become..

import pyblish_magenta.api
pyblish_magenta.api.setup()

Which means it'll differ from other packages and integrations. It's not necessarily a bad thing, but something to keep in mind, the documentation will need to make this difference clear.

from pyblish-magenta.

Yes, I have thought about this too!
I was thinking solely making the inegration of pyblish_magenta to be available in the standard way and having only the custom magenta functionality available from the pyblish_magenta.api.

So __init__.py would hold solely:

# pseudocode
def setup():
    register_plugins()

def register_plugins():
    """Register accompanying plugins"""
    pass

def deregister_plugins():
    """Deregister accompanying plugins"""
    pass

Especially since this isn't really the API, but more the integration (or is it extension?) part?

I'm thinking the API of pyblish_magenta will end up having methods available for retrieving information about the assets. (Like checking whether a referenced asset is the latest version). So having an accessible API separated from __init__.py could become beneficial and feel a bit more light-weight?

from pyblish-magenta.

I'm thinking the API of pyblish_magenta will end up having method available for retrieving information about the assets. (Like checking whether a referenced asset is the latest version).

Good thinking. This would be a very good place for it, and something that could be very interesting to provide. And yeah, as a separate API does smell more light weight, and possibly interchangeable with another, external, API later on, if needed.

from pyblish-magenta.

Any ideas for a simple API? I think it should cover querying the list of asset/shots and their accompanying published version to find a specific published file in a simple way. Currently we have versions per task so I'm assuming something like this:

# pseudo
project = Project() # defaults to project in os.environ['PROJECT']

asset = project.get_asset(name='ben')
tasks = asset.tasks()
for task in tasks:
    version = task.get_highest_version()
    version.files() # not sure how to differentiate into what file is important here?

Or to get it for the current environment?

import os
asset = Asset(os.environ['PROJECT'], os.environ['ITEM'])
task = asset.get_task(os.environ['TASK'])

Listing all assets/shots

project = Project()
print project.get_assets()
print project.get_shots()

from pyblish-magenta.

Maybe the contents of a version could be families?

from pyblish-magenta.

So something like:

# pseudo
project = Project() # defaults to project in os.environ['PROJECT']
asset = project.get_asset('ben')
task = asset.get_task('modeling')
version = asset.get_highest_version()
if version:
    family = version.get_family('model')

I remember #25 mentioning different 'variations' of a family within a version, eg. low-res vs high-res. Do we need to retrieve one level further from the family to get the exact file. And how do we easily differentiate between variation if it's solely based on name?

# Getting the latest shader from lookdev
project = Project() # defaults to project in os.environ['PROJECT']
asset = project.get_asset('ben')
task = asset.get_task('lookdev')
version = asset.get_highest_version()
if version:
    family = version.get_family('shader')
    family.get() # default
    family.get('red') # red-variation of the shader?
    family.get('blue') # blue-variation of the shader?

Then the question is, if we have a published version of the model. How do we know which version of the lookdev to apply?

from pyblish-magenta.

I'm not sure we really need "variations", additional tasks should work just fine.

asset.get_task("redLookdev")

A few more notes.

1. getters and setters are a tad non-Pythonic, I'd vouch for properties or a simple function.

   import server
   print server.ls()
   # /server/projects/thedeal
   # /server/projects/hulk
   print server.ls("thedeal", "ben", "modeling")
   # /server/projects/thedeal/assets/ben/modeling/published/v001
   # /server/projects/thedeal/assets/ben/modeling/published/v002
   # /server/projects/thedeal/assets/ben/modeling/published/v003

   Every asset ultimately represents an absolute path each, after all.

2. Assets and shots could be collectively known as "items", saving some cognitive space and keeping the door open for other types, like the "collection" you talked about for sets.

3. There's no need for classes of each type. Keep it simple, it's all paths. Having functions that operates on paths means functionality remains flexible, reusable and portable, even with manually entered paths.

   topic = ("thedeal", "ben", "modeling")
   server.highest_version(*topic)
   # 3

Before you start thinking more about the particular interface, how about we list the things we want the API to enable? This is a good start.

I'm thinking the API of pyblish_magenta will end up having methods available for retrieving information about the assets. (Like checking whether a referenced asset is the latest version).

This in particular could be as simple as:

import scene
import server
path = scene.path("ben")
topic = server.topic(path)
# ("thedeal", "ben", "modeling", "v03")
version = topic[-1]
latest = server.ls(*topic[:-1])[-1]
if not version == latest:
  print "time to update"

from pyblish-magenta.

I've been trying to type a comment on this because I just don't see how it works. There's too many if this, if that going through my mind to see clearly how this would work without hiccups. Probably similar to what I had with be.

I think the formatting on how it's on disk should solely be an outcome of a schema and the data you retrieve with these tools does not rely on that disk location whatsoever, yet solely has a way to resolve (format) those file locations... again based on the schema?

To me an asset and shot are both just Item (or Entity?) types that contain similar type of metadata, name, etc. yet only store different elements in the projects (which is basically shots vs. assets?). The more complicating thing here is (like we had with be is that for shots we need more information to resolve the actual shot, since it's contained in a sequence?)

Here's another way, not sure if better. ;)

project = Project()
item = project.item('ben')
task = item.task('lookdev')
version = task.versions[-1]
shader = version.get('shader')
filepath = shader.path()

print project.items()

What would that resolve to? How would it list out both the assets and shots?

Or in your case:

print server.topics()

from pyblish-magenta.

Ok, I think I have some terminology down now.

# pseudocode
class Project(object):
    """The Project instance is the master container of all items that are to be
    produced for the given project. It's usually identified by its `code` which
    is a (preferrably short) unique string identifier for the project.
    """
    def items(self,
              code=None,
              item_types=None):
        """Return the items in this project"""
        pass


class Item(object):
    """ An Item is an element within a project that needs to be produced and
    goes through a set of Tasks that create the output Versions.
    """
    def tasks(self, name=None):
        """Return the tasks of this item"""
        pass


class ItemType(object):
    """ The Type that can be defined for an Item.

    Example types would be: Asset, Shot or Sequence.
    """
    pass


class Task(object):
    """ A Task is a single stage in your production.

    For example modeling, rigging and lookdev are all tasks.

    A Task is assigned to an Item to allow the item to progress through that part of
    production, for example for a model to be modeled.
    """
    def versions(self):
        """Return the output versions of this task"""
        pass


class Version(object):
    """An output version from a task.

    A single `Version` can have multiple `Components`.
    For example a mayaBinary and alembicCache.
    Or a high-res mayaBinary and low-res mayaBinary.

    eg.
    >> version.components('model')
    # Just to show the hierarchy in a verbose manner, this is roughly returned:
    # Component('model', Version(1, Task('modeling', Item('Ben', Project('thedeal')))))
    """
    def components(self, name=None):
        """Return the components in this Version"""
        pass


class Component(object):
    """A component is a single file/resource that belongs to version.

    To ease retrieving all components within a version of a single 'type' you
    can categorize by `family`.

    """
    def family(self):
        pass

So for example:

# Get the Alembic caches for all shots
project = Project()
for shot in project.items(item_type='shot'):
    latest_anim = shot.tasks('animation').versions()[-1]
    alembic_cache = latest_anim.components('alembic')
    path = alembic_cache.path()

from pyblish-magenta.

Why must everything be a class? :)

from pyblish-magenta.

Ah, no specific reason at this point. Just made more sense to me, plus allowed me to lay it out as building blocks together with a docstring. At the same time the code snippet at the end made direct more sense alongside the classes?

It also allows to easily retrieve the parent Task from a Version if the Version information would only hold information about itself like a dict in JSON probably would? (Or would you hold hierarchal references?)

What other way would you prefer/recommend in this scenario?

from pyblish-magenta.

Here's a variation holding it in a dictionary:

# A project
project = {'code': "thedeal",
           'name': "The Deal",
           'items': []}

item = {'code': "ben",
        'name': "Ben",
        'item_type': "asset",
        'tasks': []}

task = {'task_type': "modeling",
        'versions': []}

version = {'num': 1,
           'components': []}

component = {'code': "model",
             'name': "Model"}


# Add them into a hierarchy
project['items'].append(item)
item['tasks'].append(task)
task['versions'].append(version)
version['components'].append(component)

# Have a look
import pprint
pprint.pprint(project)

# {'code': 'thedeal',
#  'items': [{'code': 'ben',
#             'item_type': 'asset',
#             'name': 'Ben',
#             'tasks': [{'task_type': 'modeling',
#                        'versions': [{'components': [{'code': 'model',
#                                                      'name': 'Model'}],
#                                      'num': 1}]}]}],
#  'name': 'The Deal'}

Probably missing something there.
And maybe some Item Types:

# Item types
item_type_shot = {'code': "shot",
                  'name': "Shot",
                  'description': "A shot"}

item_type_asset = {'code': "asset",
                   'name': "Asset",
                   'description': "An asset"}

item_types = [item_type_shot,
              item_type_asset]

Since this is a dictionary it won't hold methods like to retrieve a filtered subset of items directly or construct the path of the component. The code snippet would turn into something like this:

# Get the models for all assets
for asset in [item in project['items'] if item['item_type'] == 'asset']:
    anim_task = next(task for task in asset['tasks'] if task['task_type' == 'modeling')
    versions = anim_task['versions]
    for version in versions:
        alembic_component = next(c for c in version['components'] if c['code'] == 'model')
        path = construct_path(project, asset, anim_task, version, alembic_component)

from pyblish-magenta.

Ok, we haven't yet nailed down what it is you want this API to provide functionality for.

• Is it searching for an asset, e.g. by name?
• Searching for a group of assets, e.g. by having the color red?
• Is it listing the latest version of a published asset?
• Is it for looking up a development directory of a shot?

An API is suppose to be the interface for functionality, not the functionality itself. I think we should first focus on the functionality, make it stable. And then we could consider how to provide an API for it.

plus allowed me to lay it out as building blocks together with a docstring.

The reason I'm a bit sceptical to classes, and the dictionaries above, is precisely because it looks like building blocks; tightly coupled one-way objects that could ever only be used in that order in that way. It will force either classes with a tiny amount of functionality, or classes with a wide variety of functionality outside of the object's responsibility. Like a search function in every object, perhaps as a superclass to them all?

What I'm suggesting is that we treat each "object" like an absolute path, and build functionality that operates on plain absolute paths. Like a search() function, that takes a path and keyword as input.

That way we have an abstraction we know works and is relevant (a path) and can build functionality that can remain applicable to any directory layout. The user won't necessarily have to deal with paths. As in, the user won't ever have to type one or know one, but the inputs and outputs of each function is a path.

But, like I said, we first need to find out what we want the API to wrap. Currently, we have functionality to register and de-register plug-ins. Should we build something to also list assets?

from pyblish-magenta.

From our chat conversation, core Magenta functionality.

Legend

project: A collection of items
item: A subdivision of project, e.g. asset, shot
task: A subdivision of item, e.g. modeling, rigging
version: A subdivision of task, e.g. v001, v002
representation: A subdivision of version, e.g. "ben_turntable.mov", "ben_turntable.gif"

Functionality

• List items, filter by name, e.g. all items starting with "10"
• List tasks of an item
• List versions of a task
• List representations of a version
• Enter a new environment from a running session

Details

So it seems Project is the cornerstone and top-most abstraction, everything starts here. A project is defined as a collection of items, which as far as humans are concerned may be divided into assets and shots, but to code they are equal in terms of responsibility and functionality.

A task then is merely another subdivision of an item, the reasoning behind which being that it represents a body of work related to both a project and item, such as a "model" or "rig" to an "asset". All work and all content shared amongst artists are directly related to a task.

A version is the result of a task at a given point in time, is immutable and infinitely incremental. I.e. once 005 is produced, it is never overwritten. The next version if 006.

Finally, a representation is the physical file/folder on disk, carrying a single serialisation of a task for it's given version, e.g. version 002 of a "rig" as a "Maya file". "Maya file" then is a representation of this version, and there may be more than one, such as a "Quicktime file" or "FBX file of locators".

• project is always a folder
• item is always a folder
• task is always a folder
• version is always a folder
• representation is either a folder or file

from pyblish-magenta.

Lets use this as our highest priority features, yes. I have some additional functionality in mind, but I think the stage where we are now it's better left simple and see how far we can stretch its uses. If we have this functionality down I'll also start with some simple tools in Pyside to ease newcomers who are less technical. Sounds good Marcus!

from pyblish-magenta.

I had a look at implementing the bulk of this in be earlier today with great success. The point at which i hit a wall was when it was time to list versions, and published items. be doesn't deal with published material, only development files.

Now I'm thinking of either (A) providing the bulk of functionality via the be API, and supplement it via Magenta or (B) rewrite it for Magenta, only for published files, no development files, where both are used together for their strenghts.

Each come with its pros and cons, I think the safest and even fastest thing to do is to implement both A and B and discover where things shine and where they faulter.

The interface in both cases work like above, ls("topic1", "topic2", "topic3"), returning plain absolute paths as string.

In short, an immediate benefit of inplementing the bulk of it through be was the reuse of templates and items from the inventory. The disadvantage being that templates only define development directories, and we're looking to list published items, and their versions.

from pyblish-magenta.

I'm not sure whether be is the right place for such an API? I would say grab what you have for be working now and lets set up the important features we need in Magenta?

Do you have a reference to the implemented code?

The disadvantage being that templates only define development directories, and we're looking to list published items, and their versions.

I think it's about both. We want to list Items in general (whether published or not). Once it comes to version as output of Tasks that's when we want the published information. Don't you think?

from pyblish-magenta.

The tab-completion in be is exactly this. You can access it through be ls as well. As a starting point of content, I think it's a good spot for it.

On the road again, I'll push the implementation tomorrow so we can take a look.

from pyblish-magenta.

I did push the implementation but didn't feel it was enough to talk about, here are some more thoughts on the topic that I discovered while working on it.

In a nutshell, initialisation - e.g. creating an environment along with directories for an asset - is the opposite of publishing - e.g. serialising an environment along with writing content into those directories.

Put another way, it's input versus output.

Input is complex enough for independent development and growth, as is output - as we've seen with Pyblish. Output, or publishing, clearly requires a lot of thought and development for it to reach stability and I believe the same to be true about input - i.e. be.

The problem then is how be should not know about Pyblish and vice versa, even though they both relate to the same assets, only different parts of it. The initialisation of an asset, and the serialisation of it.

To work around it, it may be ideal to implement a third party. Something that oversees them both, or provides a bridge; a mediator.

In another nutshell, this third party would only deal with queries and to bridge differences between input and output; such as where assets are initialised - /project/assets/MyAsset/work - and where they are published - /project/assets/MyAsset/published/v001

My solution

As a side-car file enthusiast, my version of solving this problem would be to make MyAsset responsible for the location of it's public and private files, and the mediator responsible for directories up till that point. be queries MyAsset for private files whereas Pyblish queries it for public files, both of which are simply suffixes off of the asset itself.

I think it needs a prototype to fully make sense, but in the end it's just an opinion.

from pyblish-magenta.

Anything that can hit the feature list I think is a good starting point. I'm at a crossroad and don't really know which way to take so I guess we'll just take one and steer as we go.

Let's sprint into a working implementation and go from there.

from pyblish-magenta.

A lot has changed since opening this issue and the actual "problems" have somewhat shifted. There's also been no requirement anymore for using be but only to have a Collector that collects the required data to let Magenta do its thing.

As such I feel what we're tackling now is much less about a public API yet more about documenting what "data" is relevant for instances so they could be integrated using Magenta plug-ins.

Similarly "utilties" have been reduced in the package. At this moment there is a api.py but I don't think it houses anything that's really used (or would be useful) outside of Magenta itself. For now let's keep it as simple as it is and close this issue unless we will find it "lacking" again.

from pyblish-magenta.