Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] [Coral-Hive] Make named_struct a Coral IR operator #412

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

wmoustafa
Copy link
Contributor

@wmoustafa wmoustafa commented May 22, 2023

What changes are proposed in this pull request, and why are they necessary?

This patch removes HiveConvertletTable implementation of the CAST operator that convert named_struct to CAST (ROW() AS ROW()). Instead, it makes named_struct a Coral IR operator. Engine translations on the RHS are also adapted to accommodate this change. This also eliminates the need to rewrite from CAST (ROW() AS ROW()) to named_struct on the Spark side, because named_struct is now maintained all along. I have not removed that transformation yet, but will remove either in this PR or a future PR.

This PR is also introduces a Trino converter that takes the SqlValidator as an input so we can do type derivation on the RHS. This requires changes in engine integrations, and hence this PR is not ready to merge (unless we somehow are able to create the SqlValidator from scratch without having to change the API to pass it on).

Furthermore, this PR fixes the traversal logic in Coral-Trino transformer shuttle, and adds visited logic to transformers potentially resulting in infinite recursion so that they are called exactly once per SqlCall.

This PR should address #357.

How was this patch tested?

  • ./gradlew build
  • Added unit tests
  • Will run thorough integration test pipelines and update.

@wmoustafa wmoustafa changed the title [WIP [WIP] [Coral-Hive] Make named_struct a Coral IR operator May 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant