Comments (38)
To clarify, from within my applications, I'm saving down two files:
- Version file (e.g. Maya Ascii)
- Published file (e.g. Maya binary)
I'm tracking the version file, just like you would expect, using the Version
object. However, I would like to store this file on a "cheaper" storage media (a "versions" repository), and outside of the actual project folder. The reason for this is because we generate a massive amount of versions and this becomes a disk usage problem for us, not only during production but also later on when we want to archive the project (cloud storage, LTO tape). Example name of version file: scene1_v001.ma
The published file resides in the actual project (on the "production" repository) and is the actual file being used in rendering, referencing etc. This filename does not change inbetween versions and is constantly overwritten by newer iterations of the same Version.take
. Example name of published file: scene1.mb
So this of course poses a series of issues, which I have written supporting methods for in my GUIs, so that I can yank this functionality into stalker. This is far from ideal, and I'm struggling with how to find a long-term solution to this approach. So that is why I am bringing this question up.
This is what I have done in order to yank this functionality into stalker:
- The
Version.full_path
leads to the actual project directory on the "production" repository and matches the path to the published file (although the published file does not have the version number in the filename). I will have to manually remove e.g. "_v001" from the returnedVersion.full_path
everytime I wish to fetch the path to the published file. Example return value://10.0.1.200/prod/proj1/maya/scene1.mb
- On the
Version.generic_text
attrubute, I store aRepository.id
, which represents the "versions" repository. Whenever I wish to access the version file, I fetchVersion.full_path
and manually replace the repository in the returned filepath into the "versions" repository. Example return value://10.0.1.100/vers/proj1/maya/scene1_v001.ma
I'm not sure if there is much that can be done to the design of stalker to support this kind of behavior or if there is any interest in discussing this further. But I thought I would just bring it up and see what kind of response I would get. Please let me know what you think!
from stalker.
It would be great to implement multiple repository support, but I can't simply find a good solution to resolve Version paths.
Currently the version paths are stored as relative to the repository root, and if a project is going to have more than one repository we need to include the repository information to the path.
So a path like:
/path/to/the/version_v001.ma
should become:
$REPO1977/path/to/the/version_v001.ma
This is actually possible, and we are using this kind of paths (with environment variables) a lot in our pipeline due to the fact that we are using more than one OS (Windows and Linux namely). And it will merge the way we handle paths in anima
and stalker
.
Let's think about how we will define the FilenameTemplates:
ft = FilenameTemplate(
name='Task Filename Template',
target_entity_type='Task',
path='{%if version.is_published%}{{project.repos[1]}}{%else%}{{project.repos[0]}}{%endif%}/{{project.code}}/{%- for parent_task in parent_tasks -%}'
'{{parent_task.nice_name}}/{%- endfor -%}',
filename='{{task.nice_name}}_{{version.take_name}}'
'_v{{"%03d"|format(version.version_number)}}{{extension}}'
)
actually I liked the idea of defining the repository with:
{%if version.is_published%}{{project.repos[1]}}{%else%}{{project.repos[0]}}{%endif%}
this is pretty good.
Ok, so I'll try to implement this in the next release.
from stalker.
Let's do a quick correction, it should be:
{%if version.is_published%}$REPO{{project.repos[1].id}}{%else%}$REPO{{project.repos[0].id}}{%endif%}
from stalker.
Or you can directly use the Repo.path
(instead of $REPO{{id}}) if you're using only one OS in your studio actually.
from stalker.
That is an absolutely awesome implementation for what I intend to use multiple repositories for. I love the fact that you can use one repository for published versions and another repository for any other version. That's exactly what I would like to do.
We use multiple OS's in our studio, but we use the same paths on all OS's!
The path //10.0.0.10/share/folder/file.ext works on all platforms here. We mount the network share in OSX/Linux at /10.0.0.10 and on Windows we use UNC paths. It's a neat little trick.
But what if you access Project.repository
?
Perhaps it should be Project.repositories
or Project.repos
(like you've used in your examples above) instead, which returns a list of Repository
objects.
Or have Project.repository
return the repos[0]
for backwards compatibility and have Project.repositories
return all repos?
Anyway, I wouldn't use Project.repository
anymore if this multiple repos feature will make it into stalker.
from stalker.
Ok, so I've implemented it and I'll push the changes as soon as I'm confident about the tests (if we need any new one or not etc).
Now with this upcoming commit:
- We have a
Project.repositories
attribute which accepts a list ofRepository
instances. - The
Project.repositories
attribute is not a read-only attribute any more (which were not actually but the documentation and the tests were implying it). - It is now possible to create a
Project
without aRepository
connected to it - As you've suggested we still have a
Project.repository
attribute/property which returns the first repository orNone
if there is no repository. - I had to update the
Version.absolute_full_path
attribute to not to merge theRepository.path
with theVersion.full_path
anymore, but instead return the variable expanded version of theVersion.full_path
attribute value. This will be helpful if you were using environment variables in yourFilenameTemplate
(which we'll use in our studio, and this reminds me that I need to update the studio database so the Version.full_path should include the $REPO{id} part and also upgradeanima
library). - I've added the Alembic revision which seamlessly migrates the data without any data loss back and forth.
Hopefully I'll submit it tomorrow.
from stalker.
Looks good to me. Let me know when it's available and I will download and try it out here in our pipeline.
from stalker.
ok it is in there
just pull the latest revision from the development branch
from stalker.
Pulling now...
from stalker.
Everything works great here so far.
A quick fix to get everything up and running was to do a simple search and replace in my frontend code:
Changing project.repository_id
into project.repositories[0].id
And in my FilenameTemplate I had to add the repository path in to the path
argument.
Also, noticed another thing that I had to change just to get my old code working. When I created a project, I previously used p = Project(repository=r)
but I had to change that into p = Project(repositories=[r])
.
So, now begins the work to change the code into stopping grabbing my secondary repository from the Project.generic_text
attribute, where I have been storing this information.
I hope to be done with this before the weekend and will chime back in and let you know of any findings I might have.
This indeed is a great addition to stalker. Thanks so much for implementing it!
from stalker.
Minor issue:
Version.path and Version.full_path returns different results:
version = Version.query.first()
print version.path
print version.absolute_path
Results in:
//192.168.0.225/prod//proj1/nuke/scripts
//192.168.0.225/prod/proj1/nuke/scripts
I've got a slash between the repo and the project code in my FilenameTemplate:
path='{{project.repositories[0].path}}/{{project.code}}/ ...... '
from stalker.
I'm noticing that Version.full_path
also adds in an extra slash for me, just like with Version.path
.
from stalker.
I've now fully implemented this latest development build of stalker here and everything seems to work well.
One thing that I had to make workarounds for was the fact that Version.path
and Version.full_path
returns an extra slash between the repo and the project code for me, when comparing with the output of Version.absolute_path
and Version.absolute_full_path
.
My FilenameTemplate starts with {{project.repositories[0].path}}/{{project.code}}
.
Besides from this, I would say this is ready to be merged with the master branch.
from stalker.
Great, I've fixed the extra slash issue and I think I'm nearly ready to publish the 0.2.13 version.
from stalker.
One thing left though in my mind is to test what happens if you define 3 or more repositories for one project, and see if they return in the same order every time, or are we going to have a problem with the order so that referencing the repositories as in project.repositories[2] will not work properly.
from stalker.
I'm storing two repositories on the project and I thought that they were
returned in the right order every time. But I have this one bug that I did
not understand why I was getting which could be related. Let me check and
see if it is related to a case where they are no longer returned in the
right order.
from stalker.
Hey guys, just a quick thought, but wouldn't it make sense to store those repositories in a dict, where your first repo is always called "master" or "primary" and all the other repos may have arbitrary names?
from stalker.
Hey Oli, it is good to know that you were following Stalker.
Yes actually I should check the SQLAlchemy tutorials for dictionary
collections which can solve this problem.
Anyway, if we can't solve it so that we keep having a list of repositories
I would probably prefer reverting back to a scalar repository
attribute
and add a new attribute for the secondary repository (ex:
project.secondary_repository
).
Having two repositories or may be three is already more than enough in most
cases.
But I would definitely prefer having the flexibility of being able to
define a list of repositories per project.
And a quick look to the SQLAlchemy docs reveals that it is possible to keep
the order of items in a list or dictionary.
Erkan Ozgur Yilmaz
On Feb 10, 2015 11:19 AM, "Oliver Markowski" [email protected]
wrote:
Hey guys, just a quick thought, but wouldn't it make sense to store those
repositories in a dict, where your first repo is always called "master" or
"primary" and all the other repos may have arbitrary names?—
Reply to this email directly or view it on GitHub
#21 (comment).
from stalker.
Okay, I've verified my code and I have this:
project.repositories[0] = repository_prod
project.repositories[1] = repository_ver
print project.repositories
db.DBSession.add(project)
print project.repositories
db.DBSession.commit()
print project.repositories
and I can see that in some cases, the repositories are being switched around after the db.DBSession.commit()
, see output of the above code:
[<Jabba Production (Repository)>, <Mothership Versions (Repository)>]
[<Jabba Production (Repository)>, <Mothership Versions (Repository)>]
[<Mothership Versions (Repository)>, <Jabba Production (Repository)>]
I've got four repositories in total:
<Mothership Production (Repository)>
<Mothership Versions (Repository)>
<Jabba Production (Repository)>
<Jabba Versions (Repository)>
Could it be that the Project.repositories
is getting sorted somehow?
I like the idea of storing a dictionary with arbitrary names, like Oli suggested, but a list will work for me as well.
Our projects will need to take advantage of three repositories;
- Published files
- Old versions
- Project Archive
from stalker.
I found that it is possible to use the order_by
keyword in relationships to define the sort order. This is going to be enough to keep a consistent order, but it will not be the order that they've added to the list.
Also there is ordering_list
factory that does exactly what we want. I'll try to implement it.
from stalker.
but it will not be the order that they've added to the list.
I don't see how defining a sort order will help (me) if the order is different from the one when added to the list initially.
If the repositories do not get stored in the order they were added, then in my scenario I can never assume that my repositories are stored in the same sequence for all projects, e.g. [ repo_prod, repo_vers, repo_arch ]
and I cannot always fetch the archive repository at Project.repositories[2]
.
Instead, I would have to always loop over the Project.repositories and identify which one I need (e.g. using Tag
).
Also, I am currently defining my FilenameTemplate with {{project.repositories[0].path}}
at the moment, which is not going to work if I cannot rely on that the initial order when added to list is kept.
from stalker.
Also there is ordering_list factory that does exactly what we want. I'll try to implement it.
That sounds much better, of course :)
from stalker.
In case this cannot be solved with ordering_list
would an option be to bring back Project.repository
and then also provide a new Project.secondary_repositories
argument of type dict?
I think a dictionary will increase readability of the code:
project.secondary_repositories[2]
vs project.secondary_repositories['archive']
from stalker.
Ok I've implemented it. The repositories attribute value is now returning back in correct order every time.
I had to setup the Project <-> Repository relation with an AssociationProxy and had to use ordered_list
factory on it. The association object has a third column called position
which is updated by the ordered_list
factory automatically.
The usage is same, but the internals are changed.
from stalker.
That sounds fantastic. I'd love to pull that.
from stalker.
Ok it's gone, you can pull it
from stalker.
Excellent, pulling now.
from stalker.
ok...so this means no named repos for now right?
and btw: yes i am watching stalker all the time and i will try using it
with some freelance projects as soon as possible!
2015-02-10 15:18 GMT+01:00 Fredrik Averpil [email protected]:
Excellent, pulling now.
—
Reply to this email directly or view it on GitHub
#21 (comment).
from stalker.
Yes no named repos for now.
But it will be fairly easy to implement later on if we think that is what
we need.
Erkan Ozgur Yilmaz
On Feb 10, 2015 4:57 PM, "Oliver Markowski" [email protected]
wrote:
ok...so this means no named repos for now right?
and btw: yes i am watching stalker all the time and i will try using it
with some freelance projects as soon as possible!2015-02-10 15:18 GMT+01:00 Fredrik Averpil [email protected]:
Excellent, pulling now.
—
Reply to this email directly or view it on GitHub
#21 (comment).—
Reply to this email directly or view it on GitHub
#21 (comment).
from stalker.
@eoyilmaz did you successfully manage to perform an alembic upgrade head
?
I successfully upgraded these:
INFO [alembic.migration] Context impl PostgresqlImpl.
INFO [alembic.migration] Will assume transactional DDL.
INFO [alembic.migration] Running upgrade 2252e51506de -> 583875229230, Added Tasks.good_id column
but I got an error on:
INFO [alembic.migration] Running upgrade 583875229230 -> eaed49db6d9, Added position column to Project_Repositories table
Partial Traceback error:
sqlalchemy.exc.IntegrityError: (IntegrityError) column "repository_id" contains
null values
'ALTER TABLE "Project_Repositories" ADD COLUMN repository_id INTEGER NOT NULL'
{}
from stalker.
With a fresh database, I can see that the repos now stay in the order in which they were initially added to Project.repositories
. Great!
from stalker.
Yes I've the same error. Will try to fix it asap.
from stalker.
Ohh, it is because I've renamed the column from repo_id
to repository_id
and the automatically generated alembic upgrade script picked it as a column remove/add which causes the data to be lost.
Fixin it quickly
from stalker.
Ok pushed the fix
from stalker.
Great, I'm giving it a try.
from stalker.
Very nice!
No problems here. Alembic upgrade went smooth and the repos now stay in their initial order of Project.repositories
. I am fine with closing this issue.
from stalker.
@ultra-sonic it could be worth implementing named repos down the line. Perhaps open a new issue for it?
from stalker.
I'm closing this issue now, the requested feature is completely implemented in d12553c
from stalker.
Related Issues (20)
- date_updated attribute off by one hour (we are UTC+1) HOT 7
- "pip install" fails when stalker exists as dependency HOT 2
- Querying all projects, sorted by client name and then by project name HOT 7
- Postgres pubsub for near-realtime updates HOT 10
- Error in main Readme? HOT 1
- Delete a project with tasks and versions HOT 5
- Cockroach db HOT 2
- Help with SQL query HOT 16
- project creation from help file not working anymore HOT 9
- stalker_pyramid 0.1.0.b1 install error HOT 2
- Please, recreate the WorkingHours for all Studio instances! HOT 2
- Require psycopg2-binary rather than psycopg2 HOT 1
- Will Stalker run fine on postgres v11.5? HOT 2
- ``Version.take_name`` should be renamed to ``Version.variant``
- ``Scene`` should derive from ``Task``
- ``Shot.sequences`` should be ``Shot.sequence``
- Inherited arguments of ``__init__`` functions are not IDE friendly.
- Remove setting the status attribute with int.
- Remove ``Project.is_active`` attribute.
- Rename Task.depends to Task.depends_on
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stalker.