Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy hidden directories using CopyTerraformFolderToTemp #1021

Open
ztki opened this issue Oct 26, 2021 · 3 comments · May be fixed by #1353
Open

Copy hidden directories using CopyTerraformFolderToTemp #1021

ztki opened this issue Oct 26, 2021 · 3 comments · May be fixed by #1353
Labels
enhancement New feature or request p:backlog

Comments

@ztki
Copy link

ztki commented Oct 26, 2021

Currently - CopyTerraformFolderToTemp allows us to copy around terraform modules to temp locations - which is particularly nice for running a bunch of parallel tests, however - it doesn't copy hidden directories such as .terraform which prevents me from trying to cache both my plugins and modules prior to running my tests.

Something I've run into lately when running some tests in parallel, when running InitAndApply is hitting infrequent - but enough to be annoying transient network errors for fetching the various modules used that are basically Failed to download module type errors.

I ended up hitting a wall with causing the errors to go away since it's out of my immediate control, my first approach to solving this issue was to run a single terraform init prior to running my tests to populate .terraform/plugins and .terraform/modules prior to running the tests and just use those - reducing my number of Inits, and also improving the efficiency of the pipeline slightly.

I ended up finding out however that CopyTerraformFolderToTemp doesn't end up copying the .terraform directory (and seems to exclude all hidden directories here). While ultimately I'm able to use RetryableTerraformErrors to just retry the step to get around the problem for my for specific use-case, I still wanted to make the suggestion to potentially allow CopyTerraformFolderToTemp to copy hidden directories (or just .terraform) to the temp directories.

While Terraform's native functionality does offer the ability to have a separate directory for the plugin-cache by setting a plugin_cache_dir env var - there (currently) exists no matching functionality to be able to manage a separate modules-cache type directory - so all downloaded modules are expected to be in .terraform/modules (see hashicorp/terraform#16268).

I think a solution built into CopyTerraformFolderToTemp, or a separate function to allow this would be more desirable than a hacky workaround where we copy modules around manually for those who want to make use of a module cache while still being able to use CopyTerraformFolderToTemp

I don't mind (attempting to) implement something, but wanted to put the idea out there to collect any sort of feedback prior to making any sort of attempts.

Thanks!

@brikis98 brikis98 added enhancement New feature or request p:backlog labels Nov 1, 2021
@brikis98
Copy link
Member

brikis98 commented Nov 1, 2021

This has come up a few times. We should probably add a new function called something like CopyTerraformFolderToTempWithFilter, which allows you to filter what does/doesn't get copied. Under the hood, the current CopyTerraformFolderToTempWithFilter could call CopyTerraformFolderToTempWithFilter with a filter that works as it does today. A PR to add this is very welcome!

@brikis98
Copy link
Member

brikis98 commented Nov 1, 2021

See also #963.

@AgustinRamiroDiaz
Copy link

Hi! First of all thank you very much for this great package!

I've been hitting this one, and it'd be great to add this feature. Can I work on it?

@AgustinRamiroDiaz AgustinRamiroDiaz linked a pull request Oct 5, 2023 that will close this issue
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request p:backlog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants