Multiple repositories about stalker HOT 38 CLOSED

To clarify, from within my applications, I'm saving down two files:

1. Version file (e.g. Maya Ascii)
2. Published file (e.g. Maya binary)

I'm tracking the version file, just like you would expect, using the Version object. However, I would like to store this file on a "cheaper" storage media (a "versions" repository), and outside of the actual project folder. The reason for this is because we generate a massive amount of versions and this becomes a disk usage problem for us, not only during production but also later on when we want to archive the project (cloud storage, LTO tape). Example name of version file: scene1_v001.ma

The published file resides in the actual project (on the "production" repository) and is the actual file being used in rendering, referencing etc. This filename does not change inbetween versions and is constantly overwritten by newer iterations of the same Version.take. Example name of published file: scene1.mb

So this of course poses a series of issues, which I have written supporting methods for in my GUIs, so that I can yank this functionality into stalker. This is far from ideal, and I'm struggling with how to find a long-term solution to this approach. So that is why I am bringing this question up.

This is what I have done in order to yank this functionality into stalker:

1. The Version.full_path leads to the actual project directory on the "production" repository and matches the path to the published file (although the published file does not have the version number in the filename). I will have to manually remove e.g. "_v001" from the returned Version.full_path everytime I wish to fetch the path to the published file. Example return value: //10.0.1.200/prod/proj1/maya/scene1.mb
2. On the Version.generic_text attrubute, I store a Repository.id, which represents the "versions" repository. Whenever I wish to access the version file, I fetch Version.full_pathand manually replace the repository in the returned filepath into the "versions" repository. Example return value: //10.0.1.100/vers/proj1/maya/scene1_v001.ma

I'm not sure if there is much that can be done to the design of stalker to support this kind of behavior or if there is any interest in discussing this further. But I thought I would just bring it up and see what kind of response I would get. Please let me know what you think!

from stalker.

It would be great to implement multiple repository support, but I can't simply find a good solution to resolve Version paths.

Currently the version paths are stored as relative to the repository root, and if a project is going to have more than one repository we need to include the repository information to the path.

So a path like:

/path/to/the/version_v001.ma

should become:

$REPO1977/path/to/the/version_v001.ma

This is actually possible, and we are using this kind of paths (with environment variables) a lot in our pipeline due to the fact that we are using more than one OS (Windows and Linux namely). And it will merge the way we handle paths in anima and stalker.

Let's think about how we will define the FilenameTemplates:

    ft = FilenameTemplate(
            name='Task Filename Template',
            target_entity_type='Task',
            path='{%if version.is_published%}{{project.repos[1]}}{%else%}{{project.repos[0]}}{%endif%}/{{project.code}}/{%- for parent_task in parent_tasks -%}'
                 '{{parent_task.nice_name}}/{%- endfor -%}',
            filename='{{task.nice_name}}_{{version.take_name}}'
                     '_v{{"%03d"|format(version.version_number)}}{{extension}}'
        )

actually I liked the idea of defining the repository with:

    {%if version.is_published%}{{project.repos[1]}}{%else%}{{project.repos[0]}}{%endif%}

this is pretty good.

Ok, so I'll try to implement this in the next release.

from stalker.

Let's do a quick correction, it should be:

    {%if version.is_published%}$REPO{{project.repos[1].id}}{%else%}$REPO{{project.repos[0].id}}{%endif%}

from stalker.

Or you can directly use the Repo.path (instead of $REPO{{id}}) if you're using only one OS in your studio actually.

from stalker.

That is an absolutely awesome implementation for what I intend to use multiple repositories for. I love the fact that you can use one repository for published versions and another repository for any other version. That's exactly what I would like to do.

We use multiple OS's in our studio, but we use the same paths on all OS's!
The path //10.0.0.10/share/folder/file.ext works on all platforms here. We mount the network share in OSX/Linux at /10.0.0.10 and on Windows we use UNC paths. It's a neat little trick.

But what if you access Project.repository?
Perhaps it should be Project.repositories or Project.repos (like you've used in your examples above) instead, which returns a list of Repository objects.
Or have Project.repository return the repos[0] for backwards compatibility and have Project.repositories return all repos?

Anyway, I wouldn't use Project.repository anymore if this multiple repos feature will make it into stalker.

from stalker.

Ok, so I've implemented it and I'll push the changes as soon as I'm confident about the tests (if we need any new one or not etc).

Now with this upcoming commit:

• We have a Project.repositories attribute which accepts a list of Repository instances.
• The Project.repositories attribute is not a read-only attribute any more (which were not actually but the documentation and the tests were implying it).
• It is now possible to create a Project without a Repository connected to it
• As you've suggested we still have a Project.repository attribute/property which returns the first repository or None if there is no repository.
• I had to update the Version.absolute_full_path attribute to not to merge the Repository.path with the Version.full_path anymore, but instead return the variable expanded version of the Version.full_path attribute value. This will be helpful if you were using environment variables in your FilenameTemplate (which we'll use in our studio, and this reminds me that I need to update the studio database so the Version.full_path should include the $REPO{id} part and also upgrade anima library).
• I've added the Alembic revision which seamlessly migrates the data without any data loss back and forth.

Hopefully I'll submit it tomorrow.

from stalker.

Looks good to me. Let me know when it's available and I will download and try it out here in our pipeline.

from stalker.

ok it is in there
just pull the latest revision from the development branch

from stalker.

Pulling now...

from stalker.

Everything works great here so far.

A quick fix to get everything up and running was to do a simple search and replace in my frontend code:
Changing project.repository_id into project.repositories[0].id

And in my FilenameTemplate I had to add the repository path in to the path argument.

Also, noticed another thing that I had to change just to get my old code working. When I created a project, I previously used p = Project(repository=r) but I had to change that into p = Project(repositories=[r]).

So, now begins the work to change the code into stopping grabbing my secondary repository from the Project.generic_text attribute, where I have been storing this information.

I hope to be done with this before the weekend and will chime back in and let you know of any findings I might have.

This indeed is a great addition to stalker. Thanks so much for implementing it!

from stalker.

Minor issue:

Version.path and Version.full_path returns different results:

version = Version.query.first()
print version.path
print version.absolute_path

Results in:

//192.168.0.225/prod//proj1/nuke/scripts
//192.168.0.225/prod/proj1/nuke/scripts

I've got a slash between the repo and the project code in my FilenameTemplate:

path='{{project.repositories[0].path}}/{{project.code}}/ ...... '

from stalker.

I'm noticing that Version.full_path also adds in an extra slash for me, just like with Version.path.

from stalker.

I've now fully implemented this latest development build of stalker here and everything seems to work well.

One thing that I had to make workarounds for was the fact that Version.path and Version.full_path returns an extra slash between the repo and the project code for me, when comparing with the output of Version.absolute_path and Version.absolute_full_path.
My FilenameTemplate starts with {{project.repositories[0].path}}/{{project.code}}.

Besides from this, I would say this is ready to be merged with the master branch.

from stalker.

Great, I've fixed the extra slash issue and I think I'm nearly ready to publish the 0.2.13 version.

from stalker.

One thing left though in my mind is to test what happens if you define 3 or more repositories for one project, and see if they return in the same order every time, or are we going to have a problem with the order so that referencing the repositories as in project.repositories[2] will not work properly.

from stalker.

I'm storing two repositories on the project and I thought that they were
returned in the right order every time. But I have this one bug that I did
not understand why I was getting which could be related. Let me check and
see if it is related to a case where they are no longer returned in the
right order.

from stalker.

Hey guys, just a quick thought, but wouldn't it make sense to store those repositories in a dict, where your first repo is always called "master" or "primary" and all the other repos may have arbitrary names?

from stalker.

Hey Oli, it is good to know that you were following Stalker.

Yes actually I should check the SQLAlchemy tutorials for dictionary
collections which can solve this problem.

Anyway, if we can't solve it so that we keep having a list of repositories
I would probably prefer reverting back to a scalar repository attribute
and add a new attribute for the secondary repository (ex:
project.secondary_repository).

Having two repositories or may be three is already more than enough in most
cases.

But I would definitely prefer having the flexibility of being able to
define a list of repositories per project.

And a quick look to the SQLAlchemy docs reveals that it is possible to keep
the order of items in a list or dictionary.

Erkan Ozgur Yilmaz
On Feb 10, 2015 11:19 AM, "Oliver Markowski" [email protected]
wrote:

Hey guys, just a quick thought, but wouldn't it make sense to store those
repositories in a dict, where your first repo is always called "master" or
"primary" and all the other repos may have arbitrary names?

—
Reply to this email directly or view it on GitHub
#21 (comment).

from stalker.

Okay, I've verified my code and I have this:

project.repositories[0] = repository_prod
project.repositories[1] = repository_ver
print project.repositories
db.DBSession.add(project)
print project.repositories
db.DBSession.commit()
print project.repositories

and I can see that in some cases, the repositories are being switched around after the db.DBSession.commit(), see output of the above code:

[<Jabba Production (Repository)>, <Mothership Versions (Repository)>]
[<Jabba Production (Repository)>, <Mothership Versions (Repository)>]
[<Mothership Versions (Repository)>, <Jabba Production (Repository)>]

I've got four repositories in total:

<Mothership Production (Repository)>
<Mothership Versions (Repository)>
<Jabba Production (Repository)>
<Jabba Versions (Repository)>

Could it be that the Project.repositories is getting sorted somehow?

I like the idea of storing a dictionary with arbitrary names, like Oli suggested, but a list will work for me as well.

Our projects will need to take advantage of three repositories;

• Published files
• Old versions
• Project Archive

from stalker.

I found that it is possible to use the order_by keyword in relationships to define the sort order. This is going to be enough to keep a consistent order, but it will not be the order that they've added to the list.

Also there is ordering_list factory that does exactly what we want. I'll try to implement it.

from stalker.

but it will not be the order that they've added to the list.

I don't see how defining a sort order will help (me) if the order is different from the one when added to the list initially.

If the repositories do not get stored in the order they were added, then in my scenario I can never assume that my repositories are stored in the same sequence for all projects, e.g. [ repo_prod, repo_vers, repo_arch ] and I cannot always fetch the archive repository at Project.repositories[2].
Instead, I would have to always loop over the Project.repositories and identify which one I need (e.g. using Tag).

Also, I am currently defining my FilenameTemplate with {{project.repositories[0].path}} at the moment, which is not going to work if I cannot rely on that the initial order when added to list is kept.

from stalker.

Also there is ordering_list factory that does exactly what we want. I'll try to implement it.

That sounds much better, of course :)

from stalker.

In case this cannot be solved with ordering_list would an option be to bring back Project.repository and then also provide a new Project.secondary_repositories argument of type dict?

I think a dictionary will increase readability of the code:
project.secondary_repositories[2] vs project.secondary_repositories['archive']

from stalker.

Ok I've implemented it. The repositories attribute value is now returning back in correct order every time.

I had to setup the Project <-> Repository relation with an AssociationProxy and had to use ordered_list factory on it. The association object has a third column called position which is updated by the ordered_list factory automatically.

The usage is same, but the internals are changed.

from stalker.

That sounds fantastic. I'd love to pull that.

from stalker.

Ok it's gone, you can pull it

from stalker.

Excellent, pulling now.

from stalker.

ok...so this means no named repos for now right?

and btw: yes i am watching stalker all the time and i will try using it
with some freelance projects as soon as possible!

2015-02-10 15:18 GMT+01:00 Fredrik Averpil [email protected]:

Excellent, pulling now.

—
Reply to this email directly or view it on GitHub
#21 (comment).

from stalker.

Yes no named repos for now.

But it will be fairly easy to implement later on if we think that is what
we need.

Erkan Ozgur Yilmaz
On Feb 10, 2015 4:57 PM, "Oliver Markowski" [email protected]
wrote:

ok...so this means no named repos for now right?

and btw: yes i am watching stalker all the time and i will try using it
with some freelance projects as soon as possible!

2015-02-10 15:18 GMT+01:00 Fredrik Averpil [email protected]:

Excellent, pulling now.

—
Reply to this email directly or view it on GitHub
#21 (comment).

—
Reply to this email directly or view it on GitHub
#21 (comment).

from stalker.

@eoyilmaz did you successfully manage to perform an alembic upgrade head?

I successfully upgraded these:

INFO  [alembic.migration] Context impl PostgresqlImpl.
INFO  [alembic.migration] Will assume transactional DDL.
INFO  [alembic.migration] Running upgrade 2252e51506de -> 583875229230, Added Tasks.good_id column

but I got an error on:

INFO  [alembic.migration] Running upgrade 583875229230 -> eaed49db6d9, Added position column to Project_Repositories table

Partial Traceback error:

sqlalchemy.exc.IntegrityError: (IntegrityError) column "repository_id" contains
null values
 'ALTER TABLE "Project_Repositories" ADD COLUMN repository_id INTEGER NOT NULL'
{}

from stalker.

With a fresh database, I can see that the repos now stay in the order in which they were initially added to Project.repositories. Great!

from stalker.

Yes I've the same error. Will try to fix it asap.

from stalker.

Ohh, it is because I've renamed the column from repo_id to repository_id and the automatically generated alembic upgrade script picked it as a column remove/add which causes the data to be lost.

Fixin it quickly

from stalker.

Ok pushed the fix

from stalker.

Great, I'm giving it a try.

from stalker.

Very nice!
No problems here. Alembic upgrade went smooth and the repos now stay in their initial order of Project.repositories. I am fine with closing this issue.

from stalker.

@ultra-sonic it could be worth implementing named repos down the line. Perhaps open a new issue for it?

from stalker.

I'm closing this issue now, the requested feature is completely implemented in d12553c

from stalker.