[Coral-Hive] [Coral-Trino] Make named_struct a Coral IR operator and Migrate GenericProject Function #431
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aastha25
changed the title
yiqiangin
approved these changes
Jun 30, 2023
| @@ -21,7 +21,7 @@ | |||
| * If a column, colA, has a RelDataType, relDataTypeA, with a Trino type string, trinoTypeStringA = buildStructDataTypeString(relDataTypeA), | |||
| * then the following operation is syntactically and semantically correct in Trino: CAST(colA as trinoTypeStringA) | |||
| */ | |||
| class RelDataTypeToTrinoTypeStringConverter { | |||
| public class RelDataTypeToTrinoTypeStringConverter { | |||
There was a problem hiding this comment.
May I know why these three classes including TrinoMapTransformValuesFunction and TrinoStructCastRowFunction are converted to public? I don't see any usage of this class in this PR.
There was a problem hiding this comment.
These classes are used in GenericProjectTransformer for example here. Previously GenericProjectTransformer was in the same package as RelDataTypeToTrinoTypeStringConverter but now it's moved to another package.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request, and why are they necessary?
This PR covers two migrations - (1)
named_struct()(2)generic_project()[1]
This PR uses code changes from open PR #412 and adds minor modifications on top of it to be compatible with the new API.
Summary from the PR#412:
This patch removes the transformation from
HiveConvertletTablethat converts named_struct to CAST (ROW() AS ROW()). Instead, it makesnamed_structa Coral IR operator. Engine translations on the RHS are also adapted to accommodate this change. This also eliminates the need to rewrite from CAST (ROW() AS ROW()) to named_struct on the Spark side, because named_struct is now maintained all along. CastToNamedStructTransformer on the Spark side will be removed in a future PR.This PR also introduces a Trino transformer,
NamedStructToCastTransformer, which converts the Coral IR operator:named_structto its equivalent Trino compatible operator.This PR should address #357 and also unblocks migration of CONCAT operator here #378
[2]
This PR also migrates the Rel transformer:
GenericProjectToTrinoConverterto a SqlCall transformer:GenericProjectTransformer.How was this patch tested?
./gradlew build
updated & added UTs
tested with production views for spark, avro, trino