Encapsulating Collection about pyblish-magenta HOT 12 CLOSED

The reason for encapsulation would solely be making Validators more software-agnostic. Or possibly alloe threading?

The benefit has to be weighed up against how much data could differ between hosts. For example when Validating locked normals can other DCCs have locked normals, or is this a Maya feature?

Collecting relevant data

The hardest thing for me is that you'll either have a very long Collector that gets 'everything' about the scene creating a bloated context/instance or many small ones that stuff each node in the Context with additional data for a specific host. Both don't reduce duplicitiy of code per host.

What is relevant data? Normal directions? Shader assignments? Naming? Attribute values? Vertex Colors?

Validation Algorithms

This becomes even more problematic with Validations that use complex algorithms in a single function call from the DCC. For example checking non-manifold edges/vertices or lamina faces.

You would either have to run the check in the Selector storing just a True/False value where each Validator solely becomes if not data: raise

Or you'll be recreating complex algorithms that are internally present in DCCs?

from pyblish-magenta.

The reason for encapsulation would solely be making Validators more software-agnostic.

The reason would be to validate the appropriate things.

Consider a modeling scene in which there is 1 final output, a single mesh, but where the modeler has 10 additional meshes used to create this mesh. They could have been combined into the final output, for example. In addition, there are 50 utility nodes, including history on those nodes.

In order to properly validate such a scene, a validator would have to judge which nodes to validate, and which to consider out-of-scope. This is exactly the kind of thing a collector is meant for.

Agnosticism and parallelism is an added bonus.

What is relevant data? Normal directions? Shader assignments? Naming? Attribute values? Vertex Colors?

Relevant data depends on what you collect. Collecting a model would likely involve normals, attribute values, and others.

For example checking non-manifold edges/vertices or lamina faces.

This would also be collected into the instance.

MyModelInstance

# Attributes per node
normalDirection: "out"
laminaFaces: false
nonManifoldEdges: true
color: (0.51, 0, 0)

The collector simply collects this data, without making any judgement call on whether e.g. laminaFaces is appropriate or not. That's the responsibility of a validator, and in some cases might be ok, others not.

from pyblish-magenta.

For example, in the case of a rig and a number of utility nodes, such as measurements, the utility nodes aren't included in the Instance and will not get extracted, yet they are still subject to validation as each Validator handles their own collection.

In the current implementation this isn't actually true. Note that all Validators are getting the nodes from the instance, they are just filtering types. Like so:

cmds.ls (instance, type='mesh')

from pyblish-magenta.

Ah, I didn't notice that. That's a very good start, it would likely need to do something similar with "fully loaded" instances.

But I'm not sure it solves the problem in the example above.

from pyblish-magenta.

The Validators only check what the Collector puts in the instance. We put the minimum amount of data in the instance so it doesn't grow huge in size plus requires us to duplicate functionality that's often already available (plus optimized) in the DCC. That minimum data is only that which allows us to identify exactly what contributes to the Extraction.

Having the Collector put only related data in might even limit the possibilities of Validators. Say we don't include keyframes in the Collection since we won't extract them, then we couldn't validate whether the mesh originally was keyframed. Which we might want to do as discussed in #1

from pyblish-magenta.

I see where you're coming from, Roy and I agree that many things would seem unfit to store in an Instance, like vertex positions and animation curves. But these are problems in optimisation and performance and we haven't really gotten to a point where we can safely judge what will be fast or consume much memory.

I think we should consider performance, but not until we can stare it in the eye.

Vertex information is a good example of what should probably be left out during collection, and looked up during validation. Maya would basically take the form of an external database of information, where Collection merely collects identifiers to relevant portions of it. We could do that, it would make collection faster and instances lighter, at the cost of making validation more complex. To know which make our lives easier, we'll need to consider both options.

I would collect what can be collected during collection, make as many decisions as possible about what an Instance is in the Collector, and conversely do as many decisions as possible about what is valid to a Validator. When a performance or memory bottle-neck is apparent, it will be much easier to talk about and to solve.

from pyblish-magenta.

There is one more thing to consider here. The more work you do in collectors, the more frustrated artists, you're going to have, because it will take much longer to even populate the UI. We have very light collectors now, but even so sometimes it can take a lot of time (for example in houdini) to filter through the types of nodes and data we are collecting. This is not pyblish's fault but I think that collectors should stay clear of anything that is not very fast to run. I had artists refreshing pyblish, when they thought it was frozen, while it was only collecting lots of render nodes from the scene. It feels sluggish and artists hate sluggish things, when they don't expect it. Validation can take long time, but they see it's doing something, they have feedback. Collection is just grey blank.

from pyblish-magenta.

Validation can take long time, but they see it's doing something, they have feedback. Collection is just grey blank.

You should be seeing a "spinning wheel" during collection, but other than that I think you're right, this is important to consider as well.

I've been thinking about how to facilitate this. It wouldn't be terribly difficult to get instances to appear as they are discovered and being worked on, or to un-hide selectors from list of plug-ins. The terminal could also output things during collection.

Lengthy collection isn't something I've put much thought into initially, but it's becoming more and more relevant, and I wouldn't the performance of the GUI to have a negative impact on their design and intent.

from pyblish-magenta.

You should be seeing a "spinning wheel" during collection, but other than that I think you're right, this is important to consider as well.

It might help if it would be clearer that it's actually already collecting data through Collectors. Maybe it could even show which Collectors are processing. This clarifies to the artists that it's already processing as opposed to just initializing the GUI.

from pyblish-magenta.

Yeah, it might make more sense. Right now, they've intentionally been hidden away because of not seeming important enough for artists to care about. But in these situations, it does make sense to display them.

Sounds like a good time to start thinking about adding customisations to the GUI.

from pyblish-magenta.

I think we've found "common ground" in a solution where it offers flexibility but also the speed needed for production. The core collector is now collecting customizable data from the artist and I think it's a good fit.

Shall we close this?

from pyblish-magenta.

👍

from pyblish-magenta.