During a recent engagement, our team worked alongside the customer to configure a project infrastructure in Azure using Terraform. While this blog post is not meant to be an introduction to Terraform itself, we’re excited to share our recommended guidelines on how to structure your Terraform configuration files.
We believe the recommendations presented in this blog represent a common best practice due to their proven efficiency, reusability and extensibility. By following these guidelines, you can ensure that your Terraform configurations are organized and easy to manage, allowing you to make the most out of this powerful infrastructure management tool.
Organizing your Terraform project
When creating an infrastructure configuration, it is important to follow a consistent and organized structure to ensure maintainability and scalability of the code. Based on our experience we’ve put together the set of guidelines presented below.
1. Place each component you want to configure in its own module folder
Analyze your infrastructure code and identify the logical components that can be separated into reusable modules. For example, consider a scenario where you need to provision a data science workspace. To accomplish this you can create a Terraform module that deploys an Azure Machine Learning workspace along with all the resources it depends on: resource group, Application Insights, and so on. For more details on modules and when to use them, see the Terraform guidance.
By structuring the code this way, you achieve a clear separation of concerns between different components, making it easier to manage and update them. It also facilitates reusability of the components in other parts of your infrastructure or in new infrastructure configurations.
2. Place the .tf module files at the root of each folder and make sure to include a README file
It is recommended to place the .tf files at the root of the folder, because this file structure will be automatically picked up by the Terraform Registry. The README file will make it easy for others to understand what the module does and how to use it, which is particularly important when using modules developed by others or when collaborating on a project. Moreover, this README file can be automatically generated based on the module code, which can help ensure that it is kept up-to-date.
3. Use a consistent module structure
Using a consistent module structure for all your components facilitates maintenance. By having the same structure in each folder, it also becomes easier to navigate and understand the different components of the configuration.
A good example of a set of configuration files for each module is: provider.tf, data.tf, main.tf, backend.tf, outputs.tf, variables.tf.
By dividing the configuration into this set of files you will have a clear separation of concerns at module level. It will be easy to navigate and understand what each file is used for.
4. Use subfolders for documentation, examples and tests
By using subfolders for documentation, examples and tests, you make it easier for others to understand how to use the module and how it works.
The documentation can include what the module is installing, what are the options, an example use case and so on. You can also add here any other relevant details you might have.
The example folder can include one or more examples of how to use the module, each example having the same set of configuration files. It is also recommended to include a README that provides a clear understanding of how it can be used in practice.
The test folder includes one or more files to test the example module along with a documentation file that has instructions on how these tests can be executed.
5. Place the root module in a separate folder called main
By placing the root module in a separate folder, you make it clear what the main entry point of the configuration is.
6. Include a description for outputs and variables, as well as marking the values as ‘default’ or ‘sensitive’
Make sure this information is captured in the generated documentation, as it will provide guidance for anyone using this component in the future.
7. Establish a naming convention for your resource names, variables, outputs and data source names
When naming Terraform variables, it’s essential to use clear and consistent naming conventions that are easy to understand and follow.
The general convention is to use lowercase letters and numbers, with underscores instead of dashes, for example: “azurerm_resource_group”.
When naming resources, start with the provider’s name, followed by the target resource, separated by underscores. For instance, “azurerm_postgresql_server” is an appropriate name for an Azure provider resource. When it comes to data sources, use a similar naming convention, but make sure to use plural names for lists of items. For example, “azurerm_resourcegroups” is a good name for a data source that represents a list of resource groups.
Variable and output names should be descriptive and reflect the purpose or use of the variable. It’s also helpful to group related items together using a common prefix. For example, all variables related to storage accounts could start with “storage“. Keep in mind that outputs should be understandable outside of their scope. A useful naming pattern to follow is “{name}_{attribute}”, where “name” represents a resource or data source name, and “attribute” is the attribute returned by the output. For example, “storage_primary_connection_string” could be a valid output name.
Generating the documentation
The documentation can be automatically generated based on the configuration code in your modules with the help of terraform-docs. To generate the Terraform module documentation, go to the module folder and enter this command:
terraform-docs markdown table --output-file README.md --output-mode inject .Then, the documentation will be generated inside the component root directory.
Testing the Terraform configuration
As it is well known, one of the main advantages of using Terraform to manage your infrastructure is that its configuration files can be easily tested.
Our tests were made using Terratest, written in Go language and contain 3 steps: setup, validation and teardown. Make use of the example folder to validate the outputs of the configuration or the integration between a resource and the rest of its dependencies.
For example, if we’re deploying a Container Registry to Azure and we’re interested in receiving the resource url and id as output we can test it in the following manner:
func TestEndToEndDeploymentScenario(t *testing.T) {
t.Parallel()
fixtureFolder := "../example"
test_structure.RunTestStage(t, "setup", func() {
terraformOptions := &terraform.Options{
TerraformDir: fixtureFolder,
}
test_structure.SaveTerraformOptions(t, fixtureFolder, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
})
test_structure.RunTestStage(t, "validate", func() {
terraformOptions := test_structure.LoadTerraformOptions(t, fixtureFolder)
acrUrl := terraform.Output(t, terraformOptions, "acr_url")
acrResourceId := terraform.Output(t, terraformOptions, "acr_resource_id")
// Check that output is not empty
assert.NotEmpty(t, acrUrl, "acrUrl is empty")
assert.NotEmpty(t, acrResourceId, "acrResourceId is empty")
})
test_structure.RunTestStage(t, "teardown", func() {
terraformOptions := test_structure.LoadTerraformOptions(t, fixtureFolder)
terraform.Destroy(t, terraformOptions)
})
}To execute you have to go to the test directory. Before building, you need to add any missing module requirements necessary to build the current module’s packages and dependencies, using the command:
go mod tidyAfter executing this, we can run the tests using the command:
go test -timeout 5m end2end_test.goThe
-timeout 5mparameter will define the timeout to 5 minutes instead of the default 30 seconds.
Executing the configuration from your local machine
To execute the code from your local machine you first need to make sure that you’re using the right Azure subscription by running the commands:
az login
az account showIf the correct subscription id is displayed, you’re all set. If not, you have to manually set the subscription by running:
az account set --subscription <SUBSCRIPTION_ID>You can test the individual modules by executing the Terraform commands inside each example folder, or the complete configuration by running the commands inside the main module. When executing the complete configuration you can choose to store the Terraform state in Azure, but first you need to create a Terraform Storage Account and execute the commands in the main module by providing the Terraform Resource Group name, Storage Account name, Storage Container name and Storage Key as input parameters.
Conclusion
In this post, we aim to share our learnings and tips on how to configure and manage infrastructure using Terraform. Our approach is designed to be flexible and easy to use, making it straight forward for you to add new resources or update existing ones. The separation of concerns also makes it easy to reuse existing components in other projects, with all the information (module, examples, documentation and tests) located in one place.
Finally, we invite you to explore our recommendations and leverage the structure presented here to jump-start your infrastructure development using an efficient approach.
