Comments (17)
Hi @amontalban,
Thanks for looking this issue, I have built a test case for easier testing.
You can find it here: https://github.com/amontalban/terragrunt-issue-3076Thank you, this should really help, I will try it out.
The problem is that when I run multiple parallel plans the generated token which is an UUID will be different for each of the processes and therefore causing this authentication issue.
Each Terragrunt process runs its own cache server (starts listening on a free port) and generates its own token, they shouldn't conflict. I suppose that this
rejected the given authentication credentials
error occurs when Terraform tries to connect to a server other than the one whose process run the terraform command. The logic for determining a free port is implemented in Golang code, so the likelihood of a bug is unlikely.That's all thoughts for now, I'll try your test if I encounter this error I will be able to say more.
Hi @levkohimins,
Thank you for your explanation! I have now tested this in Atlantis and the rejected the given authentication credentials.
error disappeared after using a fixed TERRAGRUNT_PROVIDER_CACHE_TOKEN
value for all processes.
However I am now getting this error in some of the plans (Here are the stats 42 failing, 1 pending, and 652 successful checks):
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider
│ hashicorp/null: could not query provider registry for
│ registry.terraform.io/hashicorp/null: the request failed after 2 attempts,
│ please try again later: Get
│ "http://127.0.0.1:36847/v1/providers/registry.terraform.io/hashicorp/null/versions":
│ dial tcp 127.0.0.1:36847: connect: connection refused
Here is the full log just in case: https://gist.github.com/amontalban/a5dbe6f5831d9ea79fd2923793a53323
Just in case I'm going to test with v0.57.6
and report back because I have been testing with v0.57.2
.
Thanks once again for the help!
from terragrunt.
Hi @amontalban,
Thanks for looking this issue, I have built a test case for easier testing.
You can find it here: https://github.com/amontalban/terragrunt-issue-3076
Thank you, this should really help, I will try it out.
The problem is that when I run multiple parallel plans the generated token which is an UUID will be different for each of the processes and therefore causing this authentication issue.
Each Terragrunt process runs its own cache server (starts listening on a free port) and generates its own token, they shouldn't conflict. I suppose that this rejected the given authentication credentials
error occurs when Terraform tries to connect to a server other than the one whose process run the terraform command.
The logic for determining a free port is implemented in Golang code, so the likelihood of a bug is unlikely.
That's all thoughts for now, I'll try your test if I encounter this error I will be able to say more.
from terragrunt.
Hi @amontalban,
Thanks for the test case. I figured out why it doesn't work, --terragrunt-provider-cache
conflicts with --terragrunt-download-dir
. I have started working on fixing this issue and will update you once it is done.
from terragrunt.
Hey @levkohimins, sorry I missed your previous message. I will try to test it out tomorrow and report back.
Thank you for the quick fix!
from terragrunt.
@sshepel, I could not reproduce the issue.
from terragrunt.
I think the init logs are missleading (so it's working). Because with terragrunt cache enabled I can see this:
Initializing provider plugins... - Finding latest version of hashicorp/template... .... - Installing hashicorp/template v2.2.0... - Installed hashicorp/template v2.2.0 (unauthenticated)
Where as with TF_PLUGIN_CACHE_DIR var set I saw something like:
used from cache
for the same provider plugin. But I have checked the.terraform
folder under.terragrunt-cache
for specific module after the init step with cache and I can see that the binaries for provider plugins are linked to the cached folder as mentioned inProvider Caching
docs.
We use provider_installation instead of plugin_cache_dir so that Terraform has read-only access to the cache directory. Since the logs you mentioned are Terrafrom's log we cannot change them in any way.
Also there is a message about incomplete lock file information for providers (which seems to be fine, but very annoying).
We are trying to solve it in #3089
from terragrunt.
@levkohimins: I am still getting
│ Error: Failed to resolve provider packages
│
│ Could not resolve provider hashicorp/time: could not query provider
│ registry for registry.opentofu.org/hashicorp/time: the request failed after
│ 2 attempts, please try again later: Get
│ "http://127.0.0.1:46489/v1/providers//registry.opentofu.org/hashicorp/time/versions":
│ dial tcp 127.0.0.1:46489: connect: connection refused
with v0.58.3
from terragrunt.
Hi @amontalban, Unfortunately, this is not enough information to understand the cause of the issue.
│ Could not retrieve the list of available versions for provider
│ hashicorp/aws: host registry.terraform.io rejected the given authentication
│ credentials
This error is in no way related to the number of concurrent processes.
Please review how caching works https://terragrunt.gruntwork.io/docs/features/provider-cache/
Most likely, the values, TF_TOKEN_*
, are somehow overwritten, which is why you cannot connect to the cache server and get rejected the given authentication credentials
.
from terragrunt.
I would suggest also checking the credentials file.
from terragrunt.
Hi @levkohimins,
Thanks for looking this issue, I have built a test case for easier testing.
You can find it here: https://github.com/amontalban/terragrunt-issue-3076
If you set your machine with those requirements (Mainly GNU Parallel, Docker and the specified Terraform/Terragrunt versions) then if you run make parallel
you will see the error I specified in the report, on the other side if you run make sequentially
it will work properly because it is executed one at a time.
Let me know if that helps or you need anything else.
Thanks once again!
from terragrunt.
@levkohimins after building this test case and running some tests, I think you were on the right track.
The problem is that when I run multiple parallel plans the generated token which is an UUID will be different for each of the processes and therefore causing this authentication issue.
If I set the TERRAGRUNT_PROVIDER_CACHE_TOKEN
env var or --terragrunt-provider-cache-token
to be the same for all processes then this works in my test case.
I'm going to test it out in Atlantis to confirm and get back to you to confirm that solves my problem.
from terragrunt.
Hi @amontalban @nsb413, the issue should be resolved in v0.57.9 release, could you please confirm this?
from terragrunt.
Still an issue for me.
TG: 0.57.9
TF: 1.5.6
TERRAGRUNT_PROVIDER_CACHE_DIR="somepath"
TERRAGRUNT_PROVIDER_CACHE="true"
TF_PLUGIN_CACHE_DIR is not set
TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE should be set automatically based on the caching docs.
Might be some different problem, because cache is there after the first run, but during the second run on terrgarunt run-all init -upgrade
I can see that provides been installed again even though that caching server is running from what I can see in logs.
from terragrunt.
@sshepel, I could not reproduce the issue.
I think the init logs are missleading (so it's working). Because with terragrunt cache enabled I can see this:
Initializing provider plugins...
- Finding latest version of hashicorp/template...
....
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (unauthenticated)
Where as with TF_PLUGIN_CACHE_DIR var set I saw something like: used from cache
for the same provider plugin.
But I have checked the .terraform
folder under .terragrunt-cache
for specific module after the init step with cache and I can see that the binaries for provider plugins are linked to the cached folder as mentioned in Provider Caching
docs.
Also there is a message about incomplete lock file information for providers (which seems to be fine, but very annoying).
from terragrunt.
Hi @amontalban, could you please confirm that the issue is resolved?
from terragrunt.
Hi @norman-zon, your issue is different, not connected to this one, this is already resolved. Please created a new one with the full log --terragrunt-log-level debug
.
from terragrunt.
I'm closing the issue since it seems to be resolved. Feel free to reopen if this is not the case.
from terragrunt.
Related Issues (20)
- Wrong default tool when both Terraform and OpenTofu are installed? HOT 8
- `get_parent_terragrunt_dir()` function behavior does not match documentation
- goroutine fatal error: concurrent map writes
- Clarify when we issue major vs. minor vs. patch releases HOT 1
- Authentication signature from unknown issue when installing registry.opentofu.org/datadog/datadog v3.39.0 HOT 9
- Unable to apply multiple modules at once. HOT 2
- There is no variable named "dependency" - terragrunt 0.58.14 (again, 3rd time) HOT 2
- (Re)run `terragrunt run-all destroy` even if some resources tore down successfully while others failed HOT 4
- [Feature Request] terragrunt hclfmt path to format (supporting wildcard or regex)
- Issue with find_in_parent_folders function
- run-all cycle error on config folder HOT 1
- Using `--terragrunt-working-dir` makes `--terragrunt-json-out-dir` not properly place output files in a relative heirarchy
- `terragrunt graph` does not work properly with `--terragrunt-working-dir`
- Terragrunt version: 'v0.59.0' causes blob doesn't exist error HOT 2
- Reduce Reliance on Global Cache HOT 1
- Mock outputs doesn't work with attribute key and/or array HOT 3
- Provider Cache cannot fetch file from `network_mirror` with data compression
- How to use array as input in terragrunt which accepts only as string in terraform HOT 2
- `--terragrunt-tf-logs-to-json` logs are out-of-order HOT 2
- Broken link to cookie policy on homepage
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from terragrunt.