[chore] Schema Processor Revamp Implementation Parent PR #35248
Conversation
As the schema processor improves, this package path changes to help make sense of what is trying to do.
In order to help validate the works of the schema transformer in a future PR, these additional packages add in functionality to verify / abstract efforts.
A modifier allows mutating a signal from a previous version to the current one and vice versa.
A revision handles applying the modifiers to incoming signals and converting a signal to the next version.
A translation is an immutable representation of the schema translation file that can be applied to each imcoming signal that is then managed by the schema manager. The schema manager has a set of providers that allow the schema translation to be fetched from the remote location.
ankitpatel96
changed the title
@ankitpatel96 thank you for working on this. Are you looking for reviews on this PR or we should wait until you split into smaller ones? |
Hi @tigrannajaryan, I am definitely not trying to get a review on this entire PR haha - I am in the process of splitting it into 3-4 PRs. Part 1 - #35214 The rest of the code is largely done - I'm just working on splitting it into stacked PRs that makes sense. I am working up a mini design document so people have an idea of what they are looking at. |
|
Edit: Added link to specification One major question -> Is this not intended to support I was going through looking for the intricate "fun" code to deal with that scenario, but I didn't see anything, nor do I see tests associated with it. I think it's fine for to disallow that for an initial implementation/improvement, but you may want to consider the architecture/design for when you need to apply attirbute name changes on resources in addition to signals. Specifically -> The shuffling of Data points out of a resource and into a new one is... fun. From my SDK implementation this was much simpler, as the SDK only allows one resource. For the Collector -> I think you'll need some robust code, checks and performance tests around that. See https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/schemas/file_format_v1.1.0.md#all-section - specifically the notion there's a resource attribute rename possibility and that |
|
Hi Josh, |
Description:
This is a duplicate of #35213 - I had to reopen this because the Github Actions refused to trigger.
This is the schema processor, as originally written by @MovieStoreGuy in #11547 - refreshed and rebased. I've rewritten some parts and added significant amounts of testing to many parts - resulting in quite a few changes. I will be splitting this PR into many parts.
There are several features that need to be completed before this is ready for production use:
This only supports the Schema File Format 1.0.0 - we need to support 1.1.0
Right now only downgrades work. Upgrades need some work to enable
We should improve internal metrics and observability.