OpenTofu Support #315

[warrensbox]

Discussion related to using tfswitch to download binaries from OpenTofu

[warrensbox]

@MatthewJohn suggestion/thoughts from email thread:

I've been thinking about the OpenTofu support: #315
On the topic of the atlantis use of tfswitch as a library, I see they're using the, now private, GetTFList method and having to pre-query the URL beforehand due to the early-exits (or Fatals) that are used in tfswitch (https://github.com/runatlantis/atlantis/blob/513b57aa45bbae6e1217fb389adb0de4831279d0/server/core/terraform/terraform_client.go#L304C16-L304C43)).
This is maybe something of a discussion point

It seems that OpenTofu now has a static site containing release information, if we didn't want to start using the Github API: https://get.opentofu.org/tofu/.
This resembles Hashicorp's release page a bit (so much so that testing tfswitch using it, it does pick up a version correctly, but only shows one, I suspect because the HTML is all on one line, perhaps (need to dig into further).
Though, on another note, hashicorp do have a JSON version of this file (which should definitely be worth considering!).
Perhaps we could suggest to OpenTofu to implement a JSON-version of their static get.opentofu release list page.

I'd be happy to take a look - but my main concern would be how to add the logic and retaining compatibility with external library usage:

• From what I can tell getTFList does sort of work for their static page (though might need a tiny bit of tweaking, but hoping it could be made compatible with both TF and tofu release pages).
• In terms of library support, should install and getTfList be public methods to allow external use? If not, can the signatures of these methods be happily changed?

I'm wondering if installDir (https://github.com/warrensbox/terraform-switcher/blob/master/lib/defaults.go#L31) could be used to toggle the directory for Terraform vs OpenTofu (and thus installed versions and GPG key could be kept independent).

Then perhaps a flag for --distribution for terraform or opentofu could be added? (I know there's plenty of other things that will come out of the woods when starting to implement, though!)

[MatthewJohn]

I've updated this commented based on a slightly newer version that I posted in #315 today about the JSON support

[warrensbox]

@yermulnik suggestion/thoughts from email thread:

OpenTofu support would be really great. We can take into account groundwork that has been already done by https://github.com/tofuutils/tenv in this direction.
Re the source of releases: I think it is worth of starting with trying to accommodate existing code to get data from https://get.opentofu.org/tofu/
It may be easier to adjust existing code, as opposed to maintaining two different solutions (one for TF and the other one for OpenTofu)
It seems to me there was an effort to switch from Web to TF Releases API to get data about TF versions led by @Isaac Wilson, though I lost track of it and am not sure whether OpenTofu has something similar.
GH API rate limits is what adds value to the https://get.opentofu.org/tofu/ direction. This especially is depressing for people behind NATs. The GH API token is the relief, but IMHO it's best to keep the dependency chain as short as possible so that tfswitch just works w/o any external scaffold unless indeed necessary.
installDir: I think we can continue using the same dir as binaries there have tool-specific names and the rest of the files is just leftovers that we miss to clean up I guess.
The only exception is the RECENT file, which keeps a list of recently used TF versions. It is "tool-agnostic" at the moment and we may need to implement some logic to separate TF and OpenTofu recent versions depending on which tool user wants to use this time. Maybe just name it differently when using OpenTofu.
Yep, a cmdline (and TOML) flag to switch from default of TF to OpenTofu (or any other tool potentially in the future) is fine by me.
The rest I have no idea about as I'm not a Golang guy. I see no nays to allow others to use tfswitch as a lib, though this requires even more effort from us, while we still can't afford enough effort at present time

[warrensbox]

@MatrixCrawler suggestion/thoughts from email thread:

i second that openTofu support.
The mentioned points are valid. Let's circumvent GH APi Limits were possible.
Regarding the RECENT File: i would suggest to store recent binaries there in json format which can be easily (un)marshaled in golang. With a bit of effort we could easily make it backwards compatible with a convert function or something like that.

Oh and regarding the usage of our code in other projects: maybe we should expose some of our functions as kind of API in an explicit package for reusage (we will obviously use it ourselves)

[MatrixCrawler]

The advantage of the JSON format would be that we can keep track of terraform and opentofu binaries at the same time. Forget to add that sentence.

[MatthewJohn]

Hey @MatrixCrawler , sorry, I commented around the JSON support in the #315 comment.

If you're referring the github API providing JSON, I commented that hashicorp do provide a JSON endpoint in their releases page - I was going to suggest that I raise a ticket with OpenTofu "get.opentofu" repo to see if they could add a JSON file that we could use.
This would give us the benefit of getting a much more usable response, but not have the issues with the rate limiting with Github :)

[MatrixCrawler]

I was specifically refering to our RECENT file which @yermulnik wqas pointing out is no suitable for storing two different binary types at moment.
So i wanted to suggest to store it as json as it it easy to (un)marshal in golang structs. This would be a preparation step

[yermulnik]

JSON for RECENT sounds great to me.

[MatthewJohn]

Apologies @MatrixCrawler, I had completely misunderstood! Embarrassingly, I was completely unaware of that file/logic!

[warrensbox]

@crablab suggestion/thoughts from email thread:

I agree with the previous comments - I would like to see (and would be happy to contribute) towards Open Tofu support :)

[warrensbox]

This is what've I have gathered:

Considerations

Concerns	Type	Notes
RECENT	File	Currently, it's just a standard txt file. I should move it to a json format file
Install Directory	Directory	Can we use the same directory as the terraform install directory? Create a object class that look for either OpenTofu or Terraform binaries
lib.GetTFList	Library/Code	How do we make this publicly accessible? After refactoring, it should still work with OpenTofu and Terraform (same function)
Github API vs static list site	API	It's not easy using Github API, use can ask for a token from Git, but there are limitations. I'd rather use their site
HashiCorp json for release	API	This should be a future release

[MatthewJohn]

Ah, as I started to implement this (the changes in get.opentofu.org), I found: https://get.opentofu.org/tofu/api.json :D

[MatthewJohn]

Maybe as part of this, we should have instructions for creating a local mirror of Terraform/OpenTofu archives

[MatrixCrawler]

Regarding the usage of our methods in other golang projects we should have a look at the golang project layouts
See https://github.com/golang-standards/project-layout/tree/master/pkg/_your_public_lib_

At moment we don't reflect that and we are not really intentionally exposing any functions to reuse in other projects.

What are your thoughts regarding that? (we could split that discussion from this thread if needed)

[MatthewJohn]

Sounds like a new discussions - if that's okay?
I we need to ask the core question whether we want to allow it. Given that some people are using it, maybe it's a missed opportunity, but certainly has quickly taken a hit on refactoring :)

[MatthewJohn]

Have raised #452

[MatthewJohn]

Having started doing some work in #435, though getting the versions is working fine, for the download URL, I see some issues:

• Hashicorp hosts the download files on their releases site
• OpenTofu links to the Github release artifacts

I can see two ways around this:

• Use the version release page (e.g. https://releases.hashicorp.com/terraform/1.4.7) to obtain the download link (this feels like fewer changes, but less reliable)
• Have a secondary DownloadMirror.

For the second option (downloadmirror), since the github URLs contain two instances of the version (so the formatting is a bit off), plus the fact that we still need to pass around the name of the application (for the binary, zip name etc. etc.), I wondering about suggesting the following (and this is quite a bit of re-architecting):

• Update methods that revolve around naming of location, files etc. e.g. install, addRecent and other methods in install.go to take in products.
  For public methods e.g. InstallLatestVersion, create new methods (e.g. InstallLatestProductVersion), which take in a product and replace the original methods with wrapper for legacy integrations, e.g.:

const legacyProductId = "terraform"
func getLegacyProduct() ProductDetails {
	product := GetProductById(legacyProductId)
	if product != nil {
		logger.Fatalf("Default product could not be found")
	}
	return *product
}
func InstallLatestVersion(dryRun bool, customBinaryPath, installPath string, mirrorURL string) {
	InstallLatestProductVersion(getLegacyProduct(), dryRun, customBinaryPath, installPath, mirrorURL)
}

The main reason is that I think we're using the name terraform (and inherently using this in places where OpenTofu might differ considerably

[MatthewJohn]

Little update - #435 is getting there, is mostly working (except for GPG verification - for some reason opengpg doesn't like their key/sig files - but hopefully it won't be too difficult to work out.
The PR incorporates the idea of "product" entities, which I was previously referencing and maintains all of the previous public methods (but adds new ones that can be moved to).

Treat this as a demo - I won't update any tests at the moment until this design has been reviewed :)

Thanks all!

[warrensbox]

@yermulnik I will test this out and let you know here

[warrensbox]

@MatthewJohn
On this topic : https://github.com/runatlantis/atlantis/blob/513b57aa45bbae6e1217fb389adb0de4831279d0/server/core/terraform/terraform_client.go#L304C16-L304C43

I might be totally wrong on this topic, can atlantis still use getTFList (our latest changes)? Previously they were using our library.

[MatthewJohn]

Since the name was changed in ef175f6, I don't think so