Add support for shortened type names about helm HOT 19 CLOSED

+1. Was just about to file something like this, since I can do a dm describe replicatedservice:v2, but I can't turn around and use it in my template.

from helm.

Actually, this might be higher priority than P2. If you see #93, I think there's a real issue writing PRs that need to change existing types and write new ones. I think if you use this issue also to provide a local registry within the git repo (that shortened type names can resolve to), that issue goes away. I might be missing something abotu the mecahnics of the system, though.

from helm.

I'm going to start implementing the following change:
Today you would import:
type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v1/replicatedservice.py

After this change:
type: github.com/kubernetes/deployment-manager/templates/replicatedservice/v1/replicatedservice.py

If this is not what's expected here, yell loudly shortly :)

from helm.

If we assume structure in the registry, could this be
github.com/kubernetes/deployment-manager/replicatedservice:v1?

Otherwise where is the value we get?

On Wed, Nov 25, 2015 at 10:46 AM, vaikas-google [email protected]
wrote:

I'm going to start implementing the following change:
Today you would import:
type:
https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v1/replicatedservice.py

After this change:
type:
github.com/kubernetes/deployment-manager/templates/replicatedservice/v1/replicatedservice.py

If this is not what's expected here, yell loudly shortly :)

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

from helm.

Well, the latest discussion that was about the registry ended up having things like:
storage/redis/v1/redis.jinja and
common/replicatedservice/v1/replicatedservice.py

But, yes, your point about the replicatedservice:v1 at the end is a good one and will be a good first step. I'll take a look at that. Though, i think that might be blocked by the gorilla/mux issue since we'll need an encoded one.

from helm.

aaaand, I see that's what @zmerlynn wanted also and I had just overlooked that part.

from helm.

Ah yes definitely love it:

github.com/kubernetes/deployment-manager/storage/redis:v1

On Wed, Nov 25, 2015 at 10:51 AM, vaikas-google [email protected]
wrote:

Well, the latest discussion that was about the registry ended up having
things like:
storage/redis/v1/redis.jinja and
common/replicatedservice/v1/replicatedservice.py

But, yes, your point about the replicatedservice:v1 at the end is a good
one and will be a good first step. I'll take a look at that. Though, i
think that might be blocked by the gorilla/mux issue since we'll need an
encoded one.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

from helm.

The side effect of this however is that we can't assume a structure about
the registry when listing the types, we have to walk the directory tree all
the way until we hit the leaves and construct the type from that. I suppose
part of this discussion could/should take place there, but since we are
going from:
/type/version/type[py|jinja]
to:
/what/ever/can/gohere/type/version/type[py|jinja]

I really don't see an easy way to construct the list of types. Or we have
to have some other way [metadata at the root?] to list the available types.

Restricting to only one level [like storage in
storage/redis/v1/redis.jinja] seems like a wonky restriction too.

So, while going from a short to long with simple mapping rules works pretty
well, the listing of the short types is going to be more work...

But, perhaps I'm missing something :)

On Wed, Nov 25, 2015 at 10:53 AM, Brendan Melville <[email protected]

wrote:

Ah yes definitely love it:

github.com/kubernetes/deployment-manager/storage/redis:v1

On Wed, Nov 25, 2015 at 10:51 AM, vaikas-google [email protected]
wrote:

Well, the latest discussion that was about the registry ended up having
things like:
storage/redis/v1/redis.jinja and
common/replicatedservice/v1/replicatedservice.py

But, yes, your point about the replicatedservice:v1 at the end is a good
one and will be a good first step. I'll take a look at that. Though, i
think that might be blocked by the gorilla/mux issue since we'll need an
encoded one.

—
Reply to this email directly or view it on GitHub
<
#25 (comment)

.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

from helm.

Good point, @vaikas-google.

We now let people organize their types any way they like, since a flat list won't scale, and we aren't likely to successfully define a taxonomy that works for everyone.

However, putting a metadata file at the root may help, to you point. It was with this idea in mind that I put a registry.yaml at the root initially.

I removed it later, however, since it wasn't yet clear what should go in it or how it should be used. Following the YAGNI principle, it felt like something we should revisit when we knew what to do with it. Maybe that time is now.

from helm.

Ok, so @jackgr and I had a quick offline chat about this. The proposal on the table that we're proceeding with is the following. A root of the templates directory always only contains types like redis/v1/redis.jinja. In order to allow for uniquely qualified or to allow for more structure in the repository in the future, a name for a type in a github repo will always be specified like so:
github.com/owner/repository/qualifier/type:version
[qualifier must not contain slashes]

As our previous example with storage:
github.com/storage/redis:v1

This means that later on we can introduce a virtual root concept by providing a mapping in the metadata at the root which provides mapping from things like:
org.example.staging => org/example/staging
storage => org/example/staging

or something like that. If no mapping exists in the metadata file (which there won't be from day one), it's assumed to simply be a path component.

Point of this is to ensure we do not end up with arbitrary tree walking for listing the types. Providing a short hand, yet providing enough flexibility later on to ensure richer structure of the tree. This also means that one can move things around the tree, so if we later on want to reorg the repo from:
application-dm-templates/storage/redis/v1/redis.jinja
to something like:
application-dm-templates/org/example/examples/redis/v1/redis.jinja
we could achieve this by adding a metadata at the application-dm-templates which would create the mapping between qualifier 'storage' and 'org/example/examples'.

On the listing of the types side, this will be a two step process:

1. list qualifiers [for now just the directories at the root of the tree]
2. list types for each qualifier [this is what exists today without the qualifier]

Does that jive with what we decided on?

from helm.

The one downside to doing more stuff with github is that there are restrictions on number of operations you can do against the API.
https://developer.github.com/v3/#rate-limiting

Without doing any auth, you get 60/hour, yes, you read that right. That seems like a super small and very limiting in the usability of this without breaking into using OAuth (and then requiring users to create a github account before utilizing shared repositories).

from helm.

Yes, with one minor correction. Our previous example with storage will still be github.com/kubernetes/application-dm-templates/storage/redis:v1, since we haven't removed the requirement for the user to specify the owner and repository (i.e., kubernetes/application-dm-templates).

from helm.

Oops, yep. that's right. Thanks :)

On Wed, Nov 25, 2015 at 2:42 PM, Jack Greenfield [email protected]
wrote:

Yes, with one minor correction. Our previous example with storage will
still be github.com/kubernetes/application-dm-templates/storage/redis:v1,
since we haven't removed the requirement for the user to specify the owner
and repository (i.e., kubernetes/application-dm-templates).

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

from helm.

While we're at it... here's one more idea.

Since we default to Github, why not assume it for any type reference that isn't a complete URL (i.e., that doesn't have a schema component and a host component)?

So, the example above becomes: kubernetes/application-dm-templates/storage/redis:v1.

Note that we can still pull from any repository addressable by a URL by specifying a complete URL. We simply make it easier to pull from Github than from anywhere else, since that's the repository we expect to use the most.

from helm.

I don't like assuming github (well, actually assuming the missing repo
specifies this) given the new found limitations with rates, etc. Seems more
prudent to just have github.com there and be more futureproof. Just like
with go, it seems better to just plan for future repos rather than assume
it's github.com if omitted. It's harder to add it back in later than just
assume it has to be there from the get-go.

On Wed, Nov 25, 2015 at 2:46 PM, Jack Greenfield [email protected]
wrote:

While we're at it... here's one more idea.

Since we default to Github, why not assume it for any type reference that
isn't a complete URL (i.e., that doesn't have a schema component and a host
component)?

So, the example above becomes:
kubernetes/application-dm-templates/storage/redis:v1.

Note that we can still pull from any repository addressable by a URL by
specifying a complete URL. We simply make it easier to pull from Github
than from anywhere else, since that's the repository we expect to use the
most.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

from helm.

Re: the API rate-limiting issue, the best work around is to provide basic auth, which is what the git command line tool does. We can get the user's credentials from the git credential store and put them in the URL. It's as secure as git based access over https, which should be good enough.

from helm.

Regardless, I think it would be prudent to specify the github.com as a
prefix :)

On Wed, Nov 25, 2015 at 2:55 PM, Jack Greenfield [email protected]
wrote:

Re: the API rate-limiting issue, the best work around is to provide basic
auth, which is what the git command line tool does. We can get the user's
credentials from the git credential store
https://git-scm.com/docs/git-credential-store and put them in the URL.
It's as secure as git based access over https, which should be good enough.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

from helm.

OK. Good argument re: go's usage. Let's keep the github.com prefix.

Still worth doing basic auth, though, to work around the rate limiting issue for everyday use, IMO.

from helm.

Ok, this should now work.

from helm.