Allow custom metadata on each version about braid-spec HOT 20 CLOSED

You can do this already in the Headers for the PUT. We don't have to define this.

I think this is also the reason why also merge versions (so versions without payload) would still have to be present and not just "virtual".

I don't know what you mean.

from braid-spec.

You can do this already in the Headers for the PUT.

See, having an abstract protocol would help here. Because then we would be able to define metadata in the abstract protocol and then explain how one provides this metadata through HTTP headers. The point is that you want to be able to specify them in a way that they are remembered by the peer and send to other peers. Not just that you can specify the metadata.

I don't know what you mean.

"virtual" == the version clients didn't send over the network.

from braid-spec.

I don't think we should put "custom metadata" in the abstract protocol. I think signatures should be specified directly in the protocol. Not as "optional metadata", but as a part of the transport spec. It's just a feature we haven't gotten to yet, because right now everything goes over TLS, a transport that provides its own encryption, but doesn't support P2P. When we go to a P2P transport, we'll need a new encryption and signing scheme. I think of this as a feature in the transport; not the version.

from braid-spec.

"virtual" == the version clients didn't send over the network.

I specifically don't understand what merge-versions have to do with specifying metadata on a version.

from braid-spec.

So every version is an object to me, so having those objects being explicit is useful because you can attach metadata to them.

When we go to a P2P transport, we'll need a new encryption and signing scheme. But I think of this as a feature in the transport; not the version.

I see this as something you cannot prescribe. So what types of metadata people want to use/add. But yes, once we have metadata concept in place, we can standardize some uses of it for P2P purposes.

from braid-spec.

But yes, once we have metadata concept in place

I don't know why phrase this "Yes, ..." and then go on and say something I disagree with. :)

I think that once you look into the issues in defining a transport, you'll see it's not as simple as putting metadata on versions. I suspect that the idea of metadata on versions is a premature abstraction. Every time I've considered something that might be metadata, it's turned out to be better make it either (1) a protocol feature or (2) just put it into the state being synchronized. I challenge you to come up with an example of something that should be metadata before trying to add a metadata feature. :)

from braid-spec.

I challenge you to come up with an example of something that should be metadata before trying to add a metadata feature. :)

So metadata is something you want to associate with the patch and not the state. For example, user who made a particular version. Or timestamp of a particular version.

from braid-spec.

You can put a timestamp and author in the state itself:

{
  value: 3,
  last_edited_by: 'mitar',
  last_edited: 23482384
}

So I think you need an example of something can't do by adding metadata in the state, but you can do if the timestamp/author is metadata on the patch.

from braid-spec.

It might sound like I'm being really strict, but consider that any feature we add to HTTP is going to be seen as "well somebody must have thought this through, so there must be a really good reason to use it."

So people will use the feature, and put stuff there. Sometimes they will put a last_edited timestamp in the patch metadata. But sometimes they put it on the state directly. So imagine you're a beginning programmer, and you're trying to access the timestamp of an edit that some other programmer added. You know how to access the state, because it's the first thing you learned when web programming. But the timestamp of the last edit isn't there. And you have to actually use this OTHER api, that you access via the req.session.versions.last.patches[0].metadata object, and you don't know what all those things mean, because you're just learning how to code, but now you have two APIs, both can set a string, but only the state one supports patches to the string, and the whole system is extra complicated, and you feel like an idiot for now knowing when you should store things on the state or the req.session.version.last.patches[0].metadata object, and then you get holy wars where some camps are arguing for the former, and some for the latter, and nothing makes sense, and life is hell and I start screaming cusswords at the designers of the protocol for making life more complicated than it needed to be. Oh wait, now I'm yelling at myself.

from braid-spec.

On the other hand, let's consider signatures for state-mutations. You want these to exist everywhere, in a standard way, consistently, so that you can guarantee that the whole p2p network is encrypted, signed, and secure. And you don't want to have to think about it when programming in state. You want all state to by signed and encrypted. So you want to put this directly in the transport. And make it a required spec. You don't want it to be optional. (You don't want metadata.)

from braid-spec.

You can put a timestamp and author in the state itself

I find this really ugly. Because you are mixing app's state with audit log, for example. Of course you can make everything into the state. But the separation of semantics is important.

And you have to actually use this OTHER api

No, to me there are already two APIs:

• An API to get a state at particular version + its metadata (schema associated with the state, content type of the state).
• An API to get a patch which made a particular version + its metadata (timestamp of that patch, author of that patch).

So I disagree that the programmer experience must be hard here. In a way you already mentioned that you could see HTTP headers as such metadata: so when you do GET on the state of the resource, you get state metadata in headers of that request. And when you do GET on the patch for the version of the resource, you also get patch metadata in headers of that request. I think this is completely normal. What I am saying is that we have to think about this metadata in abstract API as well. It is not just an artifact of us using HTTP to have that metadata, but the opposite. We have metadata, and we have also a way to add it to HTTP, but even more, HTTP adds more metadata as well for that particular HTTP request.

I also think that developers in general are quite used to think about separation between data and metadata. I do not think this is too rocket science.

What I am asking is that we define this extension point. If you do not see use of it, I do.

from braid-spec.

@mitar Is there anything unresolved in this issue for you?

from braid-spec.

I still think it would be useful to have metadata. Like version, content type, content length are such metadata. We use that in patches already, we put all that in headers. Our serialization format even explicitly has this, e.g.,:

         Version: "ej4lhb9z78"
         Parents: "oakwn5b8qh", "uc9zwhw7mf"
         Content-Type: application/json
         Merge-Type: sync9
         Content-Length: 64

         [{"text": "Hi, everyone!",
           "author": {"link": "/user/tommy"}}]

So we have already payload + metadata. So I would suggest we make this explicit. Especially if we talk about abstract protocol. Then it is obvious that we have those. And then why not also allow in same abstraction also user-provided metadata?

from braid-spec.

Can you propose a concrete change the spec that would resolve this issue for you?

from braid-spec.

I do not have time at the moment, sorry.

from braid-spec.

Ok. The protocol already allows custom metadata on versions, as far as I can see.

So if your issue is that the protocol should allow custom metadata, then I think we can close this issue, because it already does.

from braid-spec.

So doing a PUT with X-Custom-Key: value will be stored by implementations so when I subscribe, that key will be send to all peers?

from braid-spec.

That behavior is implementation-specific.

In general, we do not require a server to send versions verbatim to all subscribed clients. They can compress multiple versions into one, for instance, or rebase edits onto different client histories.

Is that good enough? Or do you have a use-case where it is important to force a server to forward headers verbatim to all clients?

from braid-spec.

I see.

My original motivation was if you have signatures over patches. Such patches could not be collapsed though.

OK, I understand limitations now.

from braid-spec.

I think these will be good issues to take up later when we standardize P2P Messaging and Transport. We will need to standardize signatures on patches then too.

I gave an overview of these different aspects of standardization at the last braid meeting, btw, in the talk "How to extend HTTP to a fully P2P protocol". Slides and video are available here: https://braid.news/meeting-2

from braid-spec.