Terraform weekly tips (5)

Welcome everyone to another weekly Terraform tips blog post! In today's edition, we'll be discussing resource dependencies and how they are essential in cloud environments. The structure of cloud resources is often organized in a way that allows them to be logically divided and dynamically utilized across configurations and environments. So, without further ado, let's dive right in!

When creating resources in Terraform, you may encounter situations where one resource depends on another. In cloud environments, resources are typically divided into separate sub-components that collectively define a resource configuration. Let's take the example of an Azure Virtual Machine, where the following resources need to be defined in Terraform:

• Resource Group

• Virtual Network (VNet)

• Subnet

• Network Interface Card

• Public IP or VPN to access private IP

• Virtual Machine

As you start defining these sub-resources, you'll notice that everything after the resource group requires the name of the associated resource group as an attribute. The simplest but non-dynamic and non-scalable approach to handle this is by statically typing the resource group name in all the other resources. However, this method can lead to human errors. A better approach is to utilize the output from the creation of the resource group and dynamically populate the names in the other sub-resources.

By using the return output from the resource group creation, you can automate the process of filling in the required names for all the dependent sub-resources. This approach ensures dynamic and scalable configuration, reducing the potential for errors.

Not only does using return values help with limiting human-errors, it also helps Terraform as it defines its dependency graph at compile time. The term 'dependency graph' Comes from the technology inside the engine of Terraform which is used specifically to identify the full deployment order of any given resource deployment configuration. This technology is very special as it basically means that regardless of which order we define resources in, Terraform decides the correct order of deployment as it compiles, hence giving full freedom for us to structure main.tf files after what makes the most sense to us. This can simply come down to preference, readability or what else it might be.

Enough talking, lets play with some code ! The below code can either be manually copied or the codeterraform repo can be cloned at -> https://github.com/ChristofferWin/codeterraform.git (If cloned, go to folder 'week 5 05-06-2023')

We will test the behavior of Terraforms dependency graph and see how it is affected by us both using direct return values and static typed values for dependencies and see what happens.

First of, in the main.tf file, lets create all the required boilerplate code. Remember to have either a SPN setup for this or logging in using 'az cli'. If in any doubt about how to do this, visit the following blog post -> Terraform weekly tips (1) (codeterraform.com) and go to section 'Using azure cli to authenticate to Azure'

terraform {
  required_providers {
    azurerm = {
        source = "hashicorp/azurerm"
    }
  }
}

provider "azurerm" {
  features {
  }
}

Now, set the path to the root folder for the above main.tf file and run 'terraform init' To download the newest provider version of 'azurerm'

Run:

terraform init

Result:

Now, Terraform is usually pretty good at automatically understanding any dependencies that exist between resources, but there is a catch; We cannot control how long any call to the Azure Resource provider takes - One time it might be super fast, other times take multiple seconds. This will cause issues in our deployments, if we do not take control of the full deployment flow and help build all required dependencies. Issues typically arise because Terraform might understand an underlying dependency but not how important any given dependency actually is. E.g, lets construct an Azure Resource group and an Azure Virtual Network to be deployed at the same time. We will not define any 'extra' Help for Terraform to determine its dependency graph:

Define the following 2 resources to be deployed:

resource "azurerm_resource_group" "demo_rg_object" {
  name = "demo-rg"
  location = "West Europe"
}

resource "azurerm_virtual_network" "demo_vn_object" {
  name = "demo-vnet"
  location = "West Europe"
  resource_group_name = "demo-rg" //we define static values instead of using return
  address_space = ["192.168.0.0/24"]
}

Run terraform apply:

terraform apply --auto-approve=true

Result:

It partially failed; The resource group created successfully, but the Vnet deployment failed. Lets explain each of the points from the above picture:

1. Terraform realizes that 2 resources must be created - It even starts creating the resource group first, which it should as the Vnet must reside inside of that resource group

2. Terraform completes the creation of the resource group but immediately after, the Vnet deployment fails, simply because it cannot find set resource group. The Azure Resource Manager API returns HTTP 404

3. Terraform not only provides the HTTP error message from the provider, but also specifies the exact local resource provider definition that failed

But how can this be? The resource group was created first and successfully, so how is it not found ? The answer is simply timing. Due to the fact that we did not help Terraform understand how important the exact dependency was, Terraform expected to create both resources without taking into account, the time it takes Azure to actually update and be able to return the resource group that was just created. The more important a given dependency is, the more Terraform is willing to try / wait to deploy the next resource. Because of these facts, we always want to help Terraform as much as possible. Lets therefore instead use the return object of the resource group on the Vnet resource and see the result:

First, destroy the already created resource group with 'terraform destroy'

terraform destroy --auto-approve=true

Then on the line for the resource group attribute in the Vnet resource, change the static value to:

resource_group_name = azurerm_resource_group.demo_rg_object.name

Again, lets run 'terraform apply'

terraform apply --auto-approve=true

Result:

Suddenly, everything deploys successfully. Due to the return output provided by creating the resource group, Terraform now understands to keep trying to retrieve the object of the resource group BEFORE it creates the Vnet.

I will always advice to build dependencies whenever possible using this technique, but there will be situations, where that is not enough to make sure we do not receive dependency errors like we did earlier. This is typically caused by having a complex configuration using loops or nested modules - Even return values in specific calls might not be enough to make Terraform stay stable. When we hit walls like this, we can use the terraform built in meta argument called 'depends_on'

This argument forces Terraform to make certain dependencies directly based on this block defined on any resources. It has it drawbacks as well, e.g. Terraform will typically be way more 'conservative' When forced to deploy in a certain order which can lead to resources being re-deployed, even if they should not be. Never the less, its a tool we must use from time to time, in order to make robust main.tf scripts. Lets simply showcase it below using the same resources as above:

First, run 'terraform destroy'

terraform destroy --auto-approve=true

Now, first change the value for the resource group name back to the static version, simply because we know Terraform will most likely fail the deployment this way.

resource_group_name = "demo-rg"

Then inside the resource block for the Vnet, we define the meta argument 'depends_on' And set the resource group inside:

  depends_on = [ 
        azurerm_resource_group.demo_rg_object
   ]

Run 'terraform apply'

terraform apply --auto-approve=true

As it can be seen, either using return values or forcing the exact order of resource deployments successfully deployed our simple configuration. This topic is complicated and how to best help Terraform will always depend on the exact configuration at hand. We will dive into this topic of dependencies, dependency graph, etc. In later blog posts :)

That was all for now, thank you for reading this weeks edition of the 'Terraform weekly tips'. We will continue to advance ourselves as we move forward through more tips!

Want to learn more about Terraform? Click here -> terraform (codeterraform.com)

Want to learn more about other cool stuff like Automation or Powershell -> powershell (codeterraform.com) / automation (codeterraform.com)

#iac #terraform #terraform_weekly_tips #learning #code