Code Monkey home page Code Monkey logo

cf-terraforming's Introduction

Cloudflare Terraforming

Overview

cf-terraforming is a command line utility to facilitate terraforming your existing Cloudflare resources. It does this by using your account credentials to retrieve your configurations from the Cloudflare API and converting them to Terraform configurations that can be used with the Terraform Cloudflare provider.

This tool is ideal if you already have Cloudflare resources defined but want to start managing them via Terraform, and don't want to spend the time to manually write the Terraform configuration to describe them.

Read the announcement blog for further details on using cf-terraforming in your workflow.

Note

If you would like to export resources compatible with Terraform < 0.12.x, you will need to download an older release as this tool no longer supports it.

Usage

Usage:
  cf-terraforming [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  generate    Fetch resources from the Cloudflare API and generate the respective Terraform stanzas
  help        Help about any command
  import      Output `terraform import` compatible commands in order to import resources into state
  version     Print the version number of cf-terraforming

Flags:
  -a, --account string                      Target the provided account ID for the command
  -c, --config string                       Path to config file (default "~/.cf-terraforming.yaml")
  -e, --email string                        API Email address associated with your account
  -h, --help                                help for cf-terraforming
      --hostname string                     Hostname to use to query the API
  -k, --key string                          API Key generated on the 'My Profile' page. See: https://dash.cloudflare.com/profile
      --modern-import-block                 Whether to generate HCL import blocks for generated resources instead of terraform import compatible CLI commands. This is only compatible with Terraform 1.5+
      --provider-registry-hostname string   Hostname to use for provider registry lookups (default "registry.terraform.io")
      --resource-type string                Comma delimitered string of which resource(s) you wish to generate
      --terraform-binary-path string        Path to an existing Terraform binary (otherwise, one will be downloaded)
      --terraform-install-path string       Path to an initialized Terraform working directory (default ".")
  -t, --token string                        API Token
  -v, --verbose                             Specify verbose output (same as setting log level to debug)
  -z, --zone string                         Target the provided zone ID for the command

Use "cf-terraforming [command] --help" for more information about a command.

Authentication

Cloudflare supports two authentication methods to the API:

  • API Token - gives access only to resources and permissions specified for that token (recommended)
  • API key - gives access to everything your user profile has access to

Both can be retrieved on the user profile page.

Tip

We recommend that you store your Cloudflare credentials (API key, email, token) as environment variables as demonstrated below.

# if using API Token
export CLOUDFLARE_API_TOKEN='Hzsq3Vub-7Y-hSTlAaLH3Jq_YfTUOCcgf22_Fs-j'

# if using API Key
export CLOUDFLARE_EMAIL='[email protected]'
export CLOUDFLARE_API_KEY='1150bed3f45247b99f7db9696fffa17cbx9'

# specify zone ID
export CLOUDFLARE_ZONE_ID='81b06ss3228f488fh84e5e993c2dc17'

# now call cf-terraforming, e.g.
cf-terraforming generate \
  --resource-type "cloudflare_record" \
  --zone $CLOUDFLARE_ZONE_ID

cf-terraforming supports the following environment variables:

  • CLOUDFLARE_API_TOKEN - API Token based authentication
  • CLOUDFLARE_EMAIL, CLOUDFLARE_API_KEY - API Key based authentication

Alternatively, if using a config file, then specify the inputs using the same names the flag names. Example:

cat ~/.cf-terraforming.yaml
email: "[email protected]"
key: "<key>"
#or
token: "<token>"

Example usage

cf-terraforming generate \
  --zone $CLOUDFLARE_ZONE_ID \
  --resource-type "cloudflare_record"

will contact the Cloudflare API on your behalf and result in a valid Terraform configuration representing the resource you requested:

resource "cloudflare_record" "terraform_managed_resource" {
  name = "example.com"
  proxied = false
  ttl = 120
  type = "A"
  value = "198.51.100.4"
  zone_id = "0da42c8d2132a9ddaf714f9e7c920711"
}

Prerequisites

  • A Cloudflare account with resources defined (e.g. a few zones, some load balancers, spectrum applications, etc)
  • A valid Cloudflare API key and sufficient permissions to access the resources you are requesting via the API
  • An initialised Terraform directory (terraform init has run and providers installed). See the provider documentation if you have not yet setup the Terraform directory.

Installation

Homebrew

brew tap cloudflare/cloudflare
brew install cloudflare/cloudflare/cf-terraforming

Note

If you have installed an older version of cf-terraforming via Homebrew, you may need to first uninstall cf-terraforming and then install it to pick up the updated install process and address the signing/notarisation issues.

Go

go install github.com/cloudflare/cf-terraforming/cmd/cf-terraforming@latest

If you use another OS, you will need to download the release directly from GitHub Releases or build the Go source.

Importing with Terraform state

cf-terraforming has the ability to generate the configuration for you to import existing resources.

Depending on your version of Terraform, you can generate the import block (Terraform 1.5+) using the --modern-import-block flag or the terraform import compatible CLI output (all versions).

This command assumes you have already ran cf-terraforming generate ... to output your resources.

# All versions of Terraform
cf-terraforming import \
  --resource-type "cloudflare_record" \
  --email $CLOUDFLARE_EMAIL \
  --key $CLOUDFLARE_API_KEY \
  --zone $CLOUDFLARE_ZONE_ID
# Terraform 1.5+ only
cf-terraforming import \
  --resource-type "cloudflare_record" \
  --modern-import-block \
  --email $CLOUDFLARE_EMAIL \
  --key $CLOUDFLARE_API_KEY \
  --zone $CLOUDFLARE_ZONE_ID

Using non-standard registries

By default, we use the Hashicorp registry (registry.terraform.io) for looking up the provider to introspect the schema. If you are attempting to use another registry, you will need to provide the --provider-registry-hostname flag or CLOUDFLARE_PROVIDER_REGISTRY_HOSTNAME environment variable to query the correct registry.

Using non-standard Terraform binaries

Internally, we use terraform-exec library to run Terraform operations in the same way that the CLI tooling would. If a terraform binary is not available on your system path, we will attempt to download the latest to use it.

Should you have the binary stored in a non-standard location, want to use an existing binary, or you wish to provide a Terraform compatible binary (such as tofu), you need to provide the --terraform-binary-path flag or CLOUDFLARE_TERRAFORM_BINARY_PATH environment variable to instruct cf-terraforming which you expect to use.

Supported Resources

Any resources not listed are currently not supported.

Resource Resource Scope Generate Supported Import Supported
cloudflare_access_application Account
cloudflare_access_group Account
cloudflare_access_identity_provider Account
cloudflare_access_mutual_tls_certificate Account
cloudflare_access_policy Account
cloudflare_access_rule Account
cloudflare_access_service_token Account
cloudflare_account_member Account
cloudflare_api_shield Zone
cloudflare_api_token User
cloudflare_argo Zone
cloudflare_authenticated_origin_pulls Zone
cloudflare_authenticated_origin_pulls_certificate Zone
cloudflare_bot_management Zone
cloudflare_byo_ip_prefix Account
cloudflare_certificate_pack Zone
cloudflare_custom_hostname Zone
cloudflare_custom_hostname_fallback_origin Account
cloudflare_custom_pages Account or Zone
cloudflare_custom_ssl Zone
cloudflare_filter Zone
cloudflare_firewall_rule Zone
cloudflare_healthcheck Zone
cloudflare_ip_list Account
cloudflare_load_balancer Zone
cloudflare_load_balancer_monitor Account
cloudflare_load_balancer_pool Account
cloudflare_logpull_retention Zone
cloudflare_logpush_job Zone
cloudflare_logpush_ownership_challenge Zone
cloudflare_magic_firewall_ruleset Account
cloudflare_origin_ca_certificate Zone
cloudflare_page_rule Zone
cloudflare_rate_limit Zone
cloudflare_record Zone
cloudflare_ruleset Account or Zone
cloudflare_spectrum_application Zone
cloudflare_tiered_cache Zone
cloudflare_teams_list Account
cloudflare_teams_location Account
cloudflare_teams_proxy_endpoint Account
cloudflare_teams_rule Account
cloudflare_tunnel Account
cloudflare_turnstile_widget Account
cloudflare_url_normalization_settings Zone
cloudflare_waf_group Zone
cloudflare_waf_override Zone
cloudflare_waf_package Zone
cloudflare_waf_rule Zone
cloudflare_waiting_room Zone
cloudflare_worker_cron_trigger Account
cloudflare_worker_route Zone
cloudflare_worker_script Account
cloudflare_workers_kv Account
cloudflare_workers_kv_namespace Account
cloudflare_zone Account
cloudflare_zone_dnssec Zone
cloudflare_zone_lockdown Zone
cloudflare_zone_settings_override Zone

Testing

To ensure changes don't introduce regressions this tool uses an automated test suite consisting of HTTP mocks via go-vcr and Terraform configuration files to assert against. The premise is that we mock the HTTP responses from the Cloudflare API to ensure we don't need to create and delete real resources to test. The Terraform files then allow us to build what the resource structure is expected to look like and once the tool parses the API response, we can compare that to the static file.

Suggested local testing steps:

  1. Create a file with the basic provider configuration (do not commit this file)
cat > main.tf <<EOF
terraform {
  required_providers {
    cloudflare = {
      source = "cloudflare/cloudflare"
      version = "~> 4"
    }
  }
}
EOF
  1. Initialize terraform
terraform init
  1. Run tests (Cloudflare Install path should be path to repository)
make test

If you want to run a specific test case you can do so with the TESTARGS variable and -run flag

TESTARGS="-run '^TestResourceGeneration/cloudflare_teams_list'" make test

Updating VCR cassettes

Periodically, it is a good idea to recreate the VCR cassettes used in our testing to ensure they haven't drifted from actual responses. To do this, you will need to:

  • Create the appropriate resource in a Cloudflare account/zone you have access to. This is required as overwriting cassettes makes real API requests on your behalf.
  • Invoke the test suite with OVERWRITE_VCR_CASSETTES=true, CLOUDFLARE_DOMAIN=<real domain here>, authentication credentials (CLOUDFLARE_EMAIL, CLOUDFLARE_KEY, CLOUDFLARE_API_TOKEN) and the test you want to update. Example of updating the DNS CAA record test with a zone I own:
  OVERWRITE_VCR_CASSETTES=true \
    CLOUDFLARE_DOMAIN="terraform.cfapi.net" \
    CLOUDFLARE_EMAIL="[email protected]" \
    CLOUDFLARE_API_KEY="..." \
    TESTARGS="-run '^TestResourceGeneration/cloudflare_record_caa'"  \
    make test
  • Commit your changes and push them via a Pull Request.

cf-terraforming's People

Contributors

aristat avatar broswen avatar chriscf999 avatar davidferdinand avatar dependabot[bot] avatar devinbfergy avatar foreseaz avatar garrettgalow avatar gstotts avatar gvwirth avatar hunts avatar jacobbednarz avatar jafowler avatar janto avatar jsoref avatar lucascioffi-mb avatar manatarms avatar msiherb avatar patryk avatar paulliwog avatar piperswe avatar punkeel avatar renovate-bot avatar riuvshyn avatar rufoa avatar sean- avatar tamas-jozsa avatar ton31337 avatar yon avatar zackproser avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cf-terraforming's Issues

Unsupported fields are exported

Two fields were exported among others in zone settings:

advanced_ddos            = "on"
max_upload               = ""

Terraform did not like them:

settings.0.max_upload: cannot parse '' as int: strconv.ParseInt: parsing "": invalid syntax
settings.0: invalid or unknown key: advanced_ddos

resource names are invalid / deprecated

Resource names are recommended to be valid identifiers therefore they should not start with a number. Where you are using UUID type names as the resource identifier they can start with a digit.
Related to
#33
hashicorp/terraform#19919

I recommend that you prefix the resource names with an underscore.

Access policies fail on invalid key

The following error:

Error: cloudflare_access_policy.04872c94-46e9-4f2b-bac3-c4eaafc10bde: include.0: invalid or unknown key: group

Comes from the following config:

  include = {
    group = ["2b392a6b-c4b5-40d2-94a4-6c5c9ecbc61c"]
  }

Plus there's this error:

Error: cloudflare_access_policy.097877a2-46ca-434d-b7b9-09b483ee1a93: include.0: invalid or unknown key: gsuite

From the following config:

  include = {
    gsuite = ["49918f6d-015d-4a97-94d8-86c10beed822", "[email protected]", "01rvwp1q15uh8wr"]
  }

Inconsistent logging and error handling

I ran the tool for each type of data (see #31), and noticed that I sometimes got INFO and other times FATA messages for the same ("Insufficient permissions").

In the picture, almost all commands make no logs, except for access_application, access_policy, custom_pages and worker_script.

  • custom_pages just makes an extra message as INFO when no other does
  • access_policy and worker_script fail in a different way than access_application

image

Insufficient permissions to access zone

Latest cf-terraforming as of today, free Cloudflare account.
While I see TF code being generated, I always get the error Insufficient permissions to access zone .

export CLOUDFLARE_EMAIL=my_email CLOUDFLARE_TOKEN=global_api_token_from_profile CLOUDFLARE_ACCOUNT_ID=account_token_from_zone_dashboard

go run cmd/cf-terraforming/main.go --email $CLOUDFLARE_EMAIL --key $CLOUDFLARE_TOKEN --account $CLOUDFLARE_ACCOUNT_ID all > cloudflare.tf

INFO[0002] Insufficient permissions to access zone       ID=<my-zone-id>

Invalid `0rtt` code generated for cloudflare_zone_settings_override type

Using cf-terraforming to generate a cloudflare_zone_settings_override resource type results in output like:

resource "cloudflare_zone_settings_override" "zone_settings_override_zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz" {
	name = "mydomain.com"
	settings {
		0rtt = "off"

This results in an error like the following in terraforming:

Error: Argument or block definition required

  on mydomain.cloudflare.tf line 4, in resource "cloudflare_zone_settings_override" "zone_settings_override_zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz":
  4:           0rtt = "off"

An argument or block definition is required here.

Error when using tool - OrganizationID undefined

../../internal/app/cf-terraforming/cmd/root.go:154:13: undefined: cloudflare.UsingOrganization
../../internal/app/cf-terraforming/cmd/worker_route.go:64:53: api.OrganizationID undefined (type *cloudflare.API has no field or method OrganizationID)
../../internal/app/cf-terraforming/cmd/worker_route.go:71:40: api.OrganizationID undefined (type *cloudflare.API has no field or method OrganizationID)
../../internal/app/cf-terraforming/cmd/worker_script.go:38:9: api.OrganizationID undefined (type *cloudflare.API has no field or method OrganizationID)

rate_limit breaks on bypass_url_patterns

When using the rate_limit command the output created has uncompleted resources for those with bypass_url_patterns. This produces a result like this:

resource "cloudflare_rate_limit" "resource_1_def456" {
  zone = "someZone"
  threshold = 5
  period = 60
  match {
    request {
      url_pattern = "*some-url.com/public/v1*"
      schemes = ["HTTPS", ]
      methods = ["_ALL_", ]
    }
    response {
      statuses = []
      origin_traffic = true
    }
  }
  action {
    mode = "ban"
    timeout = 60
    
  }
  
  correlate {
    by = "nat"
  }
  
  disabled = false
  description = "some description"
  
  bypass_url_patterns = [
resource "cloudflare_rate_limit" "resource_2_abc123" {

If the first resource has a bypass url, it will then start to create the configuration for that url and then start with the next resource without finishing. This will result in terraform validation errors.

Support for Cloudflare Access groups,

cf-terraforming added support for three commands for managing Cloudflare Access last year:

access_application Import Access Application data into Terraform
access_policy Import Access Policy data into Terraform
access_rule Import Access Rule data into Terraform

We have since published documentation for three new commands:
access_groups
https://api.cloudflare.com/#access-groups-properties
curl -X GET "https://api.cloudflare.com/client/v4/accounts/9a7806061c88ada191ed06f989cc3dac/access/groups" \ -H "X-Auth-Email: [email protected]" \ -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \ -H "Content-Type: application/json"

access_service_tokens
https://api.cloudflare.com/#access-service-tokens-properties
curl -X GET "https://api.cloudflare.com/client/v4/accounts/9a7806061c88ada191ed06f989cc3dac/access/service_tokens" \ -H "X-Auth-Email: [email protected]" \ -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \ -H "Content-Type: application/json"

access_mtls
https://api.cloudflare.com/#access-mutual-tls-authentication-properties
curl -X GET "https://api.cloudflare.com/client/v4/zones/9a7806061c88ada191ed06f989cc3dac/access/certificates" \ -H "X-Auth-Email: [email protected]" \ -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \ -H "Content-Type: application/json"

Make log messages an optional flag

if someone wants to output the cf-terraforming cmd to a .tf directly the output needs to be valid .tf syntax. We should hide log messages by default unless requested with a flag, so the standard use case of putting in .tf will work

Cloudflare_access_rules are exported multiple times

I find that the whitelist ip access rules are exported multiple times with the same name also I can't figure the IPs they are not the IPs from the access config but seem to be the IP of the access target rather than the permitted source IP. I guess these are related because the same rule is applied to multiple origins

IN addition
cloudflare_access_policy includes groups but the groups are not exported.

Cache keys are not escaped

resource "cloudflare_page_rule" "REDACTED"
  zone = "REDACTED"
  target = "REDACTED"
  priority = 9

  actions = {
    cache_level = "cache_everything"
    edge_cache_ttl = 14400

    cache_key = "${header:origin}::${scheme}://${host_header}${uri_iqs}"
  }
}

This cache key is dumped literally from the CF rule however $ { and : are special characters to terraform and so this string needs to be quoted

terraform validate

Error: Error loading ./main.tf: Error reading config for cloudflare_page_rule[REDACTED]: parse error at 1:9: expected "}" but found ":"

Formatting error for `cloudflare_zone_settings_override`.

When reading output from an cloudflare_zone_settings_override object, the flag zero_rtt, referenced here formats as 0rtt:

resource "cloudflare_zone_settings_override" "zone_settings_override_12345" {
  name = "contoso.com"
  settings {
    0rtt = "off"
  ...

When requesting all resource, its formatting the headers incorrectly for your API

After having some issues with #122, I was finally able to get cf-terraforming running. When trying to fetch all resource, I get the following error:

troy.s$ 
troy.s$  go run /Users/troy.s/go/src/github.com/cloudflare/cf-terraforming/cmd/cf-terraforming --email $MYEMAIL --key $MYKEY --account $MYACCOUNTID all
   ERRO[0000] error from makeRequest: HTTP status 400: content "{\"success\":false,\"errors\":[{\"code\":6003,\"message\":\"Invalid request headers\",\"error_chain\":[{\"code\":6103,\"message\":\"Invalid format for X-Auth-Key header\"}]}],\"messages\":[],\"result\":null}" 
troy.s$ 

Unable to get started following the README

I ran into #122 but even after getting past that point, all of the API responses fail with an error like this:

$ go run $HOME/go/src/github.com/cloudflare/cf-terraforming/cmd/cf-terraforming/ --account $CLOUDFLARE_ACCOUNT_ID --email $CLOUDFLARE_EMAIL --key $CLOUDFLARE_TOKEN record
ERRO[0000] error from makeRequest: HTTP status 400: content "{\"success\":false,\"errors\":[{\"code\":6003,\"message\":\"Invalid request headers\",\"error_chain\":[{\"code\":6111,\"message\":\"Invalid format for Authorization header\"}]}],\"messages\":[],\"result\":null}" 

The actual terraform provider and cli4 both work using that API key.

DNS Record Import `name` does not match TF import

When using cf-terraforming on a dns record the record is created as:
name: "<subdomain>.<apex>"

For domain mitigateddos.net, for www:
name: "www.mitigateddos.net"

On TF import of that resource the name is imported as www

This leads to a state conflict after cf-terraforming import state and TF import of the tfstate:

-/+ cloudflare_record.mitigateddos_net_www_mitigateddos_net (new resource required)
      id:          "d09d916d059aa9fc8cb54bdd49deea5f" => <computed> (forces new resource)
      created_on:  "2018-04-05T13:43:59.54833Z" => <computed>
      domain:      "mitigateddos.net" => "mitigateddos.net"
      hostname:    "www.mitigateddos.net" => <computed>
      metadata.%:  "3" => <computed>
      modified_on: "2018-04-05T13:43:59.54833Z" => <computed>
      name:        "www" => "www.mitigateddos.net" (forces new resource)
      proxiable:   "true" => <computed>
      proxied:     "true" => "true"
      ttl:         "1" => "1"
      type:        "CNAME" => "CNAME"
      value:       "mitigateddos.net" => "mitigateddos.net"
      zone_id:     "be9827cd1241f1f259c6af218827dc7b" => <computed>

cf-terraforming should import the config as the tfstate would expect

Rate Limits are missing the closing brace

resource "cloudflare_rate_limit" "SOME_NAME" {
  zone = "SOME_ZONE"
  threshold = 10000
  period = 60
  match {
    request {
      url_pattern = "SOME_URL"
      schemes = ["_ALL_", ]
      methods = ["_ALL_", ]
    }
    response {
      statuses = []
      origin_traffic = true
    }
  }
  action {
    mode = "simulate"
    timeout = 60

  }

Needs a closing brace.

Resource identifiers are invalid

I've generated some resources which cannot be read by Terraform:

> terraform fmt
Error running fmt: In out.tf: At 1063:44: expected: IDENT | STRING | ASSIGN | LBRACE got: NUMBER

At that line, I have:

resource cloudflare_zone_settings_override 9225337f9c19b3ba890ed063b8a5e38d {

I fixed it by ensuring that the resource name is in quotes, like this:

resource cloudflare_zone_settings_override "9225337f9c19b3ba890ed063b8a5e38d" {

Installation Instruction

I think the installation instruction is not clear enough. Despite having the Go installed I cannot force the package to run.

$ go version
go version go1.10.4 linux/amd64

First attempt - follow instructions directly:

$ go get -v -u github.com/cloudflare/cf-terraforming/...
github.com/cloudflare/cf-terraforming (download)
github.com/cloudflare/cloudflare-go (download)
github.com/hashicorp/terraform (download)
github.com/mitchellh/go-homedir (download)
github.com/sirupsen/logrus (download)
Fetching https://golang.org/x/sys/unix?go-get=1
Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
get "golang.org/x/sys/unix": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
get "golang.org/x/sys/unix": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
golang.org/x/sys (download)
github.com/spf13/cobra (download)
github.com/spf13/pflag (download)
github.com/spf13/viper (download)
github.com/fsnotify/fsnotify (download)
github.com/hashicorp/hcl (download)
github.com/magiconair/properties (download)
github.com/mitchellh/mapstructure (download)
github.com/pelletier/go-toml (download)
github.com/spf13/afero (download)
Fetching https://golang.org/x/text/transform?go-get=1
Parsing meta tags from https://golang.org/x/text/transform?go-get=1 (status code 200)
get "golang.org/x/text/transform": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/transform?go-get=1
get "golang.org/x/text/transform": verifying non-authoritative meta tag
Fetching https://golang.org/x/text?go-get=1
Parsing meta tags from https://golang.org/x/text?go-get=1 (status code 200)
golang.org/x/text (download)
Fetching https://golang.org/x/text/unicode/norm?go-get=1
Parsing meta tags from https://golang.org/x/text/unicode/norm?go-get=1 (status code 200)
get "golang.org/x/text/unicode/norm": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/unicode/norm?go-get=1
get "golang.org/x/text/unicode/norm": verifying non-authoritative meta tag
github.com/spf13/cast (download)
github.com/spf13/jwalterweatherman (download)
Fetching https://gopkg.in/yaml.v2?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
gopkg.in/yaml.v2 (download)

$ cf-terraforming
cf-terraforming: command not found

$ go run cmd/cf-terraforming/main.go
stat cmd/cf-terraforming/main.go: no such file or directory

Second attempt - clone:

$ ls
total 84K
4.0K drwxrwxr-x  5 kiler129 kiler129 4.0K Apr  2 16:39 .
4.0K drwxr-xr-x 14 kiler129 kiler129 4.0K Apr  2 16:31 ..
4.0K drwxrwxr-x  3 kiler129 kiler129 4.0K Apr  2 16:39 cmd
//....

$ go get -v -u github.com/cloudflare/cf-terraforming/...
github.com/cloudflare/cf-terraforming (download)
//....

$ go run cmd/cf-terraforming/main.go
cmd/cf-terraforming/main.go:3:8: use of internal package not allowed

Can you maybe add more clear instruction how to install the tool for non-Go person? ;)

Error from make request: HTTP status code 400 (Invalid format for Authorization header)

Hello,

I had an issues when did tried to run a command to import:

$ source ../env.sh # source an environment variables
$ go run cmd/cf-terraforming/main.go --email $CLOUDFLARE_EMAIL --key $CLOUDFLARE_TOKEN --account $CLOUDFLARE_ACCOUNT_ID all

And it throw me this:

ERRO[0000] error from makeRequest: HTTP status 400: content "{\"success\":false,\"errors\":[{\"code\":6003,\"message\":\"Invalid request headers\",\"error_chain\":[{\"code\":6111,\"message\":\"Invalid format for Authorization header\"}]}],\"messages\":[],\"result\":null}"

Here my versions:

  • Terraform v0.12.7
  • Darwin donatas 18.7.0 Darwin Kernel Version 18.7.0: Thu Jun 20 18:42:21 PDT 2019; root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64
  • macOS Mojave 10.14.6
  • go version go1.12.9 darwin/amd64

Also I tried to run other command instead all but same happen.

Naming of resources in config state cause conflicts in terraform

For any zone that has the multiple of a DNS record with the same hostname, Terraform will complain about duplicate resources:

go run ${GOPATH}/src/github.com/cloudflare/cf-terraforming/cmd/cf-terraforming/main.go --email $CLOUDFLARE_EMAIL --key $CLOUDFLARE_KEY -z be9827cd1241f1f259c6af218827dc7b record
resource "cloudflare_record" "mitigateddos_net_a123_mitigateddos_net" {
    domain = "mitigateddos.net"

    name = "a123.mitigateddos.net"
    type = "NS"
    ttl = "1"
    proxied = "false"

    value = "rafe.ns.cloudflare.com"
}

resource "cloudflare_record" "mitigateddos_net_a123_mitigateddos_net" {
    domain = "mitigateddos.net"

    name = "a123.mitigateddos.net"
    type = "NS"
    ttl = "1"
    proxied = "false"

    value = "terin.ns.cloudflare.com"
}

From https://www.terraform.io/docs/configuration/resources.html

The resource block creates a resource of the given TYPE (first parameter) and NAME (second parameter). The combination of the type and name must be unique.

page_rule actions not working

Hi -- when fetching page_rule specifies actions are not fully shown. It gets properly listed actions but there are not values:

$ sh cf.get.sh page_rule

resource "cloudflare_page_rule" "XXX" {
    zone = "YYY"

    target = "*YYY/*"

    priority = 1
    actions {

    cache_level = 

    edge_cache_ttl = 

    explicit_cache_control = 

    }
}

cf-terraforming version is latest one (Latest commit bae9562 3 days ago, Feb 14).

While here; maybe to remove extra new line and to ident actions with tab.

error from makeRequest: HTTP status 403: insufficient permissions

Hi,

When I run the below command, I'm getting this error: ERRO[0000] error from makeRequest: HTTP status 403: insufficient permissions

command:
go run cmd/cf-terraforming/main.go --email $CLOUDFLARE_EMAIL --key $CLOUDFLARE_TOKEN --account $CLOUDFLARE_ACCOUNT_ID zone

Formating issue with access policy

Hello ,
I'm facing an issue when i try to import our access policy from CF.
The issue is as follow:
cf_terraforming generated a configuration which is formated :

resource "cloudflare_access_policy" "access_policy_id" {
    application_id = "exp"
    zone_id = "id"
    name = "test"
    precedence = "2"
    decision = "allow"
    include {		
            email_domain =   ["toto.com",]  
            email_domain =   ["tata.fr",]  

But Terraform provider expect something like this :

resource "cloudflare_access_policy" "access_policy_id" {
    application_id = "exp"
    zone_id = "id"
    name = "test"
    precedence = "2"
    decision = "allow"
    include {		
            email_domain =   ["toto.com", "tata.fr"]  

I have tried to fix it, but i'm not that ripe with go templating.
The issue comes from the file accss_policy.go line 23, how the loop is processed for Policy.include.
The format provided by CF is :
[map[email_domain:map[domain:toto.com]] map[email_domain:map[domain:tata.fr]]]
Is it possible to fix the issue ?
Thank you !

Some resources are read-only for free accounts

Hello,

I backported cloudflare_zone_settings_override using cf-terraforming.

When applying, I get this error:

Error: Error applying plan:

1 error(s) occurred:

* cloudflare_zone_settings_override.settings: 1 error(s) occurred:

* cloudflare_zone_settings_override.settings: invalid zone setting "sort_query_string_for_cache" (value: off) found - cannot be set as it is read only

So I suppose cf-terraforming should no retrieve "sort_query_string_for_cache", since it cannot be modified by Terraform.

Add cloudflare provider in terraformer

We published new project for generate terraform files from existing resources.
https://github.com/GoogleCloudPlatform/terraformer
It's work with terraform providers directly and get all data from providers. Generate tf + tfstate files with hcl/terraform libs.
For add new options in resources you need provider update only.
Today it's support GCP, AWS providers.
You can add support cloudflare provider in terraformer(need to implement interfaces) and you can to depricate this project.
No more templating, no more update/support in 2 places and no more other "terraform" bugs.
I can help you with this.

importing dns records to state file

I downloaded my DNS records with cf-terraforming, and now want to push/apply them to my existing state bucket in S3. Since the record(s) already exists, terraform errors with an HTTP 400. I'm not sure where this should go - possibly to the API maintainers? To the Terraform provider maintainers? To hashicorp? Ideally, an existing resource would issue a 200 and just move on, maybe?


$ tf apply -var-file=local.tfvars
...

`An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:

  • create

Terraform will perform the following actions:

  • cloudflare_record.<my_string>
    id:
    created_on:
    domain: "redacted.com"
    hostname:
    metadata.%:
    modified_on:
    name: "redacted.com"
    proxiable:
    proxied: "true"
    ttl: "1"
    type: "A"
    value: "x.x.x.x"
    zone_id:

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

cloudflare_record.<my_stirng>: Creating...
created_on: "" => ""
domain: "" => "redacted.com"
hostname: "" => ""
metadata.%: "" => ""
modified_on: "" => ""
name: "" => "redacted.com"
proxiable: "" => ""
proxied: "" => "true"
ttl: "" => "1"
type: "" => "A"
value: "" => "x.x.x.x"
zone_id: "" => ""

Error: Error applying plan:

1 error(s) occurred:

  • cloudflare_record.<my_string>: 1 error(s) occurred:

  • cloudflare_record.redacted: Failed to create record: error from makeRequest: HTTP status 400: content "{"success":false,"errors":[{"code":81057,"message":"The record already exists."}],"messages":[],"result":null}"`

Cannot install using go 1.13.1 on MacOS

12:59 $ brew upgrade go
Updating Homebrew...
==> Upgrading 1 outdated package:
go 1.13 -> 1.13.1
==> Upgrading go 
==> Downloading https://homebrew.bintray.com/bottles/go-1.13.1.mojave.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/77/77887193657770ff7ae524ddfef468a3063f4f6a6d38d80391c0c50d0d168b09?__gda__=exp=1570641168~hmac=99f5e38ff9f16e118522cddbb256415f35abc43e2e12b83ef519d13a
######################################################################## 100.0%
==> Pouring go-1.13.1.mojave.bottle.tar.gz
🍺  /usr/local/Cellar/go/1.13.1: 9,268 files, 414.0MB
Removing: /usr/local/Cellar/go/1.13... (9,268 files, 414.0MB)
Removing: /Users/shawn/Library/Caches/Homebrew/go--1.13.mojave.bottle.tar.gz... (148.0MB)

✔ ~/dev/go 
13:01 $ go get -u github.com/cloudflare/cf-terraforming/...
go: finding github.com/cloudflare/cf-terraforming latest
go: downloading github.com/cloudflare/cf-terraforming v0.0.0-20191007153855-83f27391d1b2
verifying github.com/cloudflare/[email protected]: github.com/cloudflare/[email protected]: reading https://sum.golang.org/lookup/github.com/cloudflare/[email protected]: 404 Not Found

Load balancing host is not quoted

In addition to #46, load balancer hosts are not quoted:

resource "cloudflare_load_balancer_monitor" "eccb3fcfbb1bbb46b8521e34bc121e46" {
    ...

    header {

        Host = [foo.bar]

    }

}

Access rules panic on insufficient credentials

DEBU[0000] Importing Access Rule data
DEBU[0000] Processing zone                               ID=abc Name=foo.org
DEBU[0000] error from makeRequest: HTTP status 401: invalid credentials
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x14459a8]

goroutine 1 [running]:
github.com/cloudflare/cf-terraforming/internal/app/cf-terraforming/cmd.glob..func3(0x196dda0, 0xc0000e6630, 0x0, 0xb)
	/Users/bobrik/cf-repos/cf-terraforming/internal/app/cf-terraforming/cmd/access_rule.go:64 +0x228
github.com/spf13/cobra.(*Command).execute(0x196dda0, 0xc0000e6420, 0xb, 0xb, 0x196dda0, 0xc0000e6420)
	/Users/bobrik/cf-repos/cf-terraforming/vendor/github.com/spf13/cobra/command.go:766 +0x2ae
github.com/spf13/cobra.(*Command).ExecuteC(0x196d420, 0xc00013df88, 0x10076bf, 0xc000088058)
	/Users/bobrik/cf-repos/cf-terraforming/vendor/github.com/spf13/cobra/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/bobrik/cf-repos/cf-terraforming/vendor/github.com/spf13/cobra/command.go:800
github.com/cloudflare/cf-terraforming/internal/app/cf-terraforming/cmd.Execute()
	/Users/bobrik/cf-repos/cf-terraforming/internal/app/cf-terraforming/cmd/root.go:38 +0x32
main.main()
	/Users/bobrik/cf-repos/cf-terraforming/cmd/cf-terraforming/main.go:6 +0x20

Compare to account_member, which silently swallows the error:

DEBU[0000] Importing Account Member data
DEBU[0000] error from makeRequest: HTTP status 403: insufficient permissions

I'm not sure which one is the preferred way, but it should be the same between them.

This is the line where it panics on dereference (accessRules == nil):

<no value> output illegal characters

It's possible for the zone_settings_override command to output a field with a value of <no value>. Terraform will consider these as illegal characters.

Page rule command writes invalid syntax

Output from cf-terraforming page_rule:

resource "cloudflare_page_rule" "page_rule_xxxxxxx" {
    zone = "foo.bar"

    target = "foo.bar/*"

    priority = 1
    status = "active"
    actions = {
        forwarding_url {
            status_code = 302
            url = "https://signup.foo.bar/"
        }

    }
}

Result of running $ terraform validate:

Error: Unsupported argument

  on cloudflare-page_rule.tf line 6, in resource "cloudflare_page_rule" "page_rule_xxxxxxx":
   6:   actions = {

An argument named "actions" is not expected here. Did you mean to define a
block of type "actions"?

Can't export DNS records from account

latest version of cf-terraforming

go version
go version go1.11.5 darwin/amd64

go run cmd/cf-terraforming/main.go --email $CLOUDFLARE_EMAIL --key $CLOUDFLARE_KEY -a cf158db8694a70604a9916976a58e22b record

# github.com/cloudflare/cf-terraforming/internal/app/cf-terraforming/cmd
internal/app/cf-terraforming/cmd/zone.go:53:23: impossible type assertion:
	*schema.Provider does not implement "github.com/terraform-providers/terraform-provider-cloudflare/vendor/github.com/hashicorp/terraform/terraform".ResourceProvider (wrong type for Apply method)
		have Apply(*"github.com/hashicorp/terraform/terraform".InstanceInfo, *"github.com/hashicorp/terraform/terraform".InstanceState, *"github.com/hashicorp/terraform/terraform".InstanceDiff) (*"github.com/hashicorp/terraform/terraform".InstanceState, error)
		want Apply(*"github.com/terraform-providers/terraform-provider-cloudflare/vendor/github.com/hashicorp/terraform/terraform".InstanceInfo, *"github.com/terraform-providers/terraform-provider-cloudflare/vendor/github.com/hashicorp/terraform/terraform".InstanceState, *"github.com/terraform-providers/terraform-provider-cloudflare/vendor/github.com/hashicorp/terraform/terraform".InstanceDiff) (*"github.com/terraform-providers/terraform-provider-cloudflare/vendor/github.com/hashicorp/terraform/terraform".InstanceState, error)
internal/app/cf-terraforming/cmd/zone.go:54:34: impossible type assertion:
	*schema.Provider does not implement "github.com/terraform-providers/terraform-provider-cloudflare/vendor/github.com/hashicorp/terraform/terraform".ResourceProvider (wrong type for Apply method)
		have Apply(*"github.com/hashicorp/terraform/terraform".InstanceInfo, *"github.com/hashicorp/terraform/terraform".InstanceState, *"github.com/hashicorp/terraform/terraform".InstanceDiff) (*"github.com/hashicorp/terraform/terraform".InstanceState, error)
		want Apply(*"github.com/terraform-providers/terraform-provider-cloudflare/vendor/github.com/hashicorp/terraform/terraform".InstanceInfo, *"github.com/terraform-providers/terraform-provider-cloudflare/vendor/github.com/hashicorp/terraform/terraform".InstanceState, *"github.com/terraform-providers/terraform-provider-cloudflare/vendor/github.com/hashicorp/terraform/terraform".InstanceDiff) (*"github.com/terraform-providers/terraform-provider-cloudflare/vendor/github.com/hashicorp/terraform/terraform".InstanceState, error)

SIGSEGV

Running cf-terraforming all --email "[email protected]" --key my-api-key --account my-accound-id all

results in the error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x805696]

goroutine 1 [running]:
github.com/cloudflare/cf-terraforming/internal/app/cf-terraforming/cmd.glob..func3(0xca7e00, 0xc0001142a0, 0x1, 0x7)
        /path/to/cf-terraforming/internal/app/cf-terraforming/cmd/access_rule.go:50 +0x186
github.com/cloudflare/cf-terraforming/internal/app/cf-terraforming/cmd.glob..func5(0xca7e00, 0xc0001142a0, 0x1, 0x7)
        /path/to/cf-terraforming/internal/app/cf-terraforming/cmd/all.go:21 +0x11b
github.com/spf13/cobra.(*Command).execute(0xca7e00, 0xc0001141c0, 0x7, 0x7, 0xca7e00, 0xc0001141c0)
        /path/to/go/src/github.com/spf13/cobra/command.go:766 +0x2cc
github.com/spf13/cobra.(*Command).ExecuteC(0xca8060, 0x1, 0xca7ba0, 0xc000113f78)
        /path/to/go/src/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/spf13/cobra.(*Command).Execute(0xca8060, 0x0, 0x12)
        /path/to/go/src/github.com/spf13/cobra/command.go:800 +0x2b
github.com/cloudflare/cf-terraforming/internal/app/cf-terraforming/cmd.Execute()
        /path/to/go/src/github.com/cloudflare/cf-terraforming/internal/app/cf-terraforming/cmd/root.go:33 +0x31
main.main()
        /path/to/go/src/github.com/cloudflare/cf-terraforming/cmd/cf-terraforming/main.go:6 +0x20

Use of internal package not allowed

Hello,
I keep getting the following error every time I try to run this.

cmd/cf-terraforming/main.go:3:8: use of internal package github.com/cloudflare/cf-terraforming/internal/app/cf-terraforming/cmd not allowed

I am not quite sure what I am supposed to do about this, do I need to change something?

Thanks,

Possibility to export everything?

I've tried using an all command, or nothing, or appending multiple commands after each other to no aval. Would it be possible to export everything in one go?

cloudflare_record issue with Error: Missing required argument for cloudflare_record

Run:

cf-terraforming --email $CLOUDFLARE_EMAIL --key $CLOUDFLARE_TOKEN -z $CLOUDFLARE_ZONE record > record.tf
terraform plan

Error:

Error: Missing required argument

  on importing-record.tf line 111, in resource "cloudflare_record" "TXT_tru_xxx_xxxx":
 111: resource "cloudflare_record" "xxxxxx" {

The argument "zone_id" is required, but no definition was found.


Error: Unsupported argument

  on importing-record.tf line 112, in resource "cloudflare_record" "xxxxxxxxx":
 112:     domain = "tru.xx"

An argument named "domain" is not expected here.

https://www.terraform.io/docs/providers/cloudflare/guides/version-2-upgrade.html

Terraform Cloudflare Provider Version 2 uses zone_id instead of domain

WAF rules aren't fully exported

Due to Pagination, when WAF rules are exported, currently only 20 items from each of the 2 packages get saved - this isn't useful.

Total rule counts
OWASP package currently: "total_count": 2499
Cloudflare package currently: "total_count": 366

I understand it may not be desirable to export all of the WAF rules as it's often the case that many will be left as default but at minimum, a means of exporting rules that have been changed from default would be very useful for Cloudflare (Enterprise) customers wanting to use TF to bootstrap many new zones.

Errors with installation after following README

cf-terraforming cannot be found after following the README instructions. See example output:

Here are the exact commands and output:

troy.s$ go get -u github.com/cloudflare/cf-terraforming/...
troy.s$ cf-terraforming
   bash: cf-terraforming: command not found
troy.s$ go cf-terraforming
   go cf-terraforming: unknown command
   Run 'go help' for usage.
troy.s$ go cmd/cf-terraforming
   go cmd/cf-terraforming: unknown command
   Run 'go help' for usage.
troy.s$ go run cmd/cf-terraforming
   package cmd/cf-terraforming: cannot find package "cmd/cf-terraforming" in any of:
        /usr/local/go/src/cmd/cf-terraforming (from $GOROOT)
        /Users/troy.s/go/src/cmd/cf-terraforming (from $GOPATH)

When I go and look for the package, it's being installed at:

/Users/troy.s/go/src/github.com/cloudflare/cf-terraforming

Is there a missing step here?

Missing value for CAA cloudflare_record

The cloudflare_record CAA records don't appear correct. There is no "value" set for CAA records.

When I ran go run cmd/cf-terraforming/main.go --email $CLOUDFLARE_EMAIL --key $CLOUDFLARE_TOKEN --account $CLOUDFLARE_ACCOUNT_ID all I get the following resource:

resource "cloudflare_record" "CAA_example_com_ee7c46060f3f700960bbc5b1c81d3deb" {
    domain = "example.com"

    name = "example.com"
    type = "CAA"
    ttl = "1"
    proxied = "false"

}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.