dependencies on CESR 2.0

[dhh1128]

Yesterday's implementer's call had this topic on its agenda, but we didn't have time to cover it. @2byrds suggested that I create a discussion to pursue the topic asynchronously.

Here is my initial comment from Discord, which was intended as a seed for the discussion:

All: I heard through other Provenant folks that there's pressure building for a more definitive upgrade to version 2.0 (of CESR specifically, I believe). I think some of it may be connected to issues/842 and Phil's fix in pull/844? Maybe other things as well; I heard Sam mention this as an important objective of a new KERI initiative, too.

I think I know some of the reasons for wanting this, but I don't think I know all of them. I'm also worried that this could have ramifications for existing ACDCs and wallets and could be a challenge that takes a lot of management. Moving from cmdline wallets to the keria/signify architecture took Provenant almost a year, and we made some unfair pleas for help of Phil and others along the way. It was so painful and disruptive to our business that it makes me wonder if Provenant should just plan to backport all fixes to a 1.x branch for a while -- though I'm sensitive to how bad it would be to divide our energies. I'm sure Kevin Griffin and crew at GLEIF are counting the cost of a 2.0 upgrade, and I'd be curious to hear their analysis. And of course, the analysis of others as well.

This is not advocacy against a migration, but rather a bookmark in my desire to discuss further. I have missed 2 or 3 community meetings in a row, but other Provenant people have been there. I'll try to make the next one; maybe we can discuss further there? Or do we need a dedicated meeting?

[dhh1128]

Since I made this comment in Discord, I had a discussion with @m00sey where imagined that it would be possible to identify a 1.x "freeze version" of the whole KERI stack that all QVIs must be compatible with. I think this is a very positive step, and I welcome it. However, I feel a need to for a 1.x version that is not frozen, but rather is receiving bug fixes (possibly not all, but at least those that are for bugs disruptive to production use of vLEIs) and enhancements (likely not all, but perhaps a few that are required to make production vLEI usage practical). I don't feel like I have a practical path to an update to a CESR 2.0 dependency, if I am just looking for a bug fix or a critical enhancement.

My concern about upgrading to CESR 2.0 is mostly not about code compatibility. Rather, it is about data compatibility. I don't want to have to re-issue credentials, or rebuild witness history. Possibly, the data compatibility issue could be made much less of a concern by making CESR 2.0 code capable of handling 1.x data structures?

[SmithSamuelM]

These are great concerns. So let me detail how the upgrade is designed to minimize data concerns. There are the main parts of CESR that are affected by CESR 2.0 changes.

None of the changes in CESR 2.0 require reissuing credentials or key events. All previously issued key events and ACDCs formats are forward compatible with CESR 2.0

1. Primitives: there are no backward breaking primitive code changes for CESR 2.0. All the CESR 2.0 does is add new primitives not change the meaning, syntax etc of existing primitives. This continues the policy that when a new primitive is needed it can be added to the primitive table. Those implementations that don't yet support the new primitive will not recognize it and should error out. The major addition of primitives were to support encrypted sealed-box ciphers of three different types of plain text. These are needed to support the TSP protocol. So the TSP protocol has CESR 2.0 as a dependency.

2. New Count Codes. The curent (v1) count codes were inconsistently designed and do not support pipelining universally nor support CESR native field maps or data structures or ESSR formatted messages needed for the TSP. The list of needed changes mean that attempting any sort of backwards compatibility mode without versioning was going to be cumbersome.

As a result CESR 2.0 supports versioned CESR streams. In a versioned CESR streams, both CESR 1.0 count codes and CESR 2.0 count codes are supported by switching versions. A new count code was added for versioning CESR streams. A parser that supports CESR 2.0 must support the versioning which means it may maintain backwards compatiblity when the stream includes the version code. This means that for version 1 to be forwards compatible, version 1 implementation need to start adding the version count code to their streams.

3. CESR Native event types. This provides support for Key Events and ACDCs that are expressed as CESR native compositions of count codes and primitives. This also provides support for the messages for TSP (ESSR). Since these messages (key events, ACDCs, TSP) are either signed or anchored and then the anchors signed. They cannot be exchanged verifiably with a version 1 only implementation. This is backwards breaking. But there is no rush for any ecosystem to start using CESR native events until its ready to support it.

The message types have their own versioning. So CESR v2 backwards compatible supports version 1 messages embedded in a v2 stream.

Intermediate migration.
The breaking changes from version 1 to version 2 are all in the count codes. In version 1 the only place count codes are used are for attachments to messages which messages are serialized as either JSON/CBOR/MGPK. The attachments themselves are not signed or anchored/signed so they can be converted after the fact between version 1 and version 2. One could send a version 1 json message in a version 2 cesr stream of attachments. There is no breaking data change.

So an intermediate path would be to continue using version 1 messages while upgrading the count codes to version 2. This means a replay of a KEL for example would use version 1 messages but the attachments could be attached with version 2 count codes.

As some future point, new events on the same KEL would be issued using version 2 messages. In that case a replay would include both version 1 and version 2 message and version 2 attachments.

Always in the future any KERI implementation must support version 1 messages otherwise all legacy version key state would be unverifiable.

But attachments because they are not signed or anchored/signed do not have to stay in version 1 format they can be either.

[SmithSamuelM]

One more clarificaton.

In the keripy implementation, attached primitives are stored in the database not the enclosing count codes. So there are no database changes when going to CESR 2.0 count codes. When replaying a KEL the code that generates the attachments newly generates the associated count codes on the fly to create the attachments. So an implementation can change the attachement generation code to use a different version. The databases which store the primitives don't change and the primitive codes don't change as they are forwards compatible by design.

This means that a given implementation could differentially stream the same KEL to different destination with different version for each destination where the version matches the version supported by the destination.

This would allow an API to have a query parameter that indicates which CESR version to use for attachments when streaming the response.

[dhh1128]

This is wonderful news, @SmithSamuelM , and it alleviates most of my concerns.

This means that for version 1 to be forwards compatible, version 1 implementation need to start adding the version count code to their streams.... When replaying a KEL the code that generates the attachments newly generates the associated count codes on the fly to create the attachments.

I'm just restating to make sure I understood correctly. If I have a witness that's serving KELs with events that were sealed with v1 code, and I upgrade the witness to be v2-oriented, the count codes will appear in the witness's responses without any changes to the sealed data. Right?

What happens if that witness serves such data to a client that's only v1-aware?

[SmithSamuelM]

@dhh1128

What happens if that witness serves such data to a client that's only v1-aware?

Just to be clear, there are two versions. Each message in a CESR stream has a message version, this is different from the CESR stream version.

So when you say v1 aware I don't know for sure which of these versions you mean.

Messages never change their own message version at the time they were created. This is because they are signed/anchored-signed and changing their serialization would break the signatures (i.e. make them unverifiable). So no matter what any replay of those messages will alwyas preserve the message version at the origninal time of creation.

A CESR stream can support both CESR v1 and CESR v2 stream. An endpoint that someone queries (such a the host that runs a witness) may respond in whatever CESR version it chooses. The endpoint could advertise what version of CESR is supports, v1, v2 or both. The query could then indicate which one it wants.

I'm just restating to make sure I understood correctly. If I have a witness that's serving KELs with events that were sealed with v1 code, and I upgrade the witness to be v2-oriented, the count codes will appear in the witness's responses without any changes to the data. Right?

So when you say data, I want to clarify. There are several types of data.

1. Data serialized inside a message. These are signed/anchored so the serializations can't ever change. The semantics of the data should be stable so if the serialization does not change then the data should not change.

2. Data serialized as an attachment to a message. The primary data in attachments are signatures on the message to which the signatures are attached. The signatures themselves are CESR primitives and they primitive codes are stable (we plan on always having forwards compatiblity of primitives codes) We can add codes but not change codes. Attachments are also composed or framed or grouped using count codes. The count codes are changing from CESR version 1 to Version 2. These are breaking changing. So a group of signature primitives would be attached with a count code. The individual signature primitives do not change between CESR versions but the count code that groups them does.

3. Data stored in a database. An implementation can store data however it chooses. In KERYpy the serialized messages are stored and the signature primitives are stored without their enclosing count code. An implementation could store a serialized stream in toto. This would make it more difficult to support versioning of CESR streams.

[dhh1128]

So when you say v1 aware I don't know for sure which of these versions you mean

My question was about CESR streams: "serves such data" meant "serves v1 messages inside a v2 stream". I understood that messages themselves are static.

An endpoint that someone queries (such a the host that runs a witness) may respond in whatever CESR version it chooses. The endpoint could advertise what version of CESR is supports, v1, v2 or both. The query could then indicate which one it wants.

Am I right in assuming that "could", in this context, means that the data structure design keeps open the possibility to resolve this version challenge flexibly, by writing code that is capable of the feature you describe -- but that no wallets/agents/witnesses/watchers that are currently deployed in production exhibit this feature? And that therefore, we need to upgrade our software fabric, generally, before CESR 2 data in the wild avoids triggering incompatibilities? That's what I expect, and it is a remarkable achievement that CESR is this forward-thinking, so I am not being critical. I'm just trying to understand the practical ramifications of having code in the main branches of keripy or keria that begins emitting CESR 2 data streams.

[SmithSamuelM]

Yes.

Currently there are no production implementations of CESR v2 stream parsers. So one would have to upgrade production code to include CESR v2 compatible stream parsers.

Current there are no production implementation of CESR v2 stream generators (I have some preliminary ones running in unit tests)
So one would have to upgrade production code to support CESR v2 compatible stream generators.

Then the ecosystems would have to start enabling the use of those CESR v2 streams.