Making Calls to the Azure ARM API with CURL and Manipulating Results with JQ

My work frequently involves using remote, ephemeral, containers to run scripts/code/whatever for DevOps type automation. Typically these containers are running some flavor of Linux. My work also frequently involves designing these automations for Azure and using Terraform for IaC. On occasion though, I find the need to drop out of “declarative” mode and utilize scripted imperative actions to make dynamic changes at run time. I frequently lean on either using the remote-exec provisioner with a null resource -or- if I really need my script to run during the Terraform plan phase, I will use the special external data source.

All that said, this article isn’t really focused on Terraform per se’ or the usage of those methods. Perhaps at some point in the future I will have an opportunity to do some write-ups on the topic of using run-time scripts. Rather, this is much more narrowly focused on how to interact and affect changes in Azure using the ARM API using commonly available tools on most runners like BASH and CURL.

Why Not Just Use Azure CLI?
The short answer is that I want my solutions to work on a variety of platforms, consistently, without having to worry about special tooling. Granted you could probably argue that curl and jq are special tooling but I do feel they are more ubiquitous than Azure CLI. Also, as of this writing, my general experience with getting Azure CLI installed on my Linux distributions is frequently cumbersome and platform specific. All that said, if you don’t care quite as much about portability and are in an environment where you have control over the container on which your code will be executed – Azure CLI is probably a better way to go.

Why Avoid Calling the ARM API?
I think it’s worthwhile to recognize that this approach may not be the most sane. Making an API call directly (vs. depending on an intervening layer like Azure CLI, ARM Template, or Terraform) is more complex in my opinion and I don’t have enough experience to know if this is going to require more maintenance over the long term.

Today’s Use Case
One of the recent headaches I have dealt with is that I need to add my runner’s IP address to an Azure storage account that is being used as a remote Terraform backend. The runner’s public IP is dynamic so it isn’t something I can guarantee will be the same every time my automation is going to run. If the storage account NACL doesn’t contain this IP, this will lead to a critical failure during Terraform initialization. The solution presented is a simple bash script that will make ARM API calls to get the current NACL from the storage account and append the current IP address of the runner to the NACL if it isn’t already on the list.

With all that prologue now given, I think it’s good to go ahead and dive into the code. I am just going to walk through the bash script which could be used as a stage in something like Gitlab or Jenkins -or- (if it wasn’t needed for initialization) called by Terraform via one of the aforementioned methods.

The first part of the script is the input parameters followed by a curl to get the public IP of the runner and save that within another variable. Next, we need to get an authentication token for the API calls which can be done with a POST to https://login.microsoftonline.com/ with our tenant id and the oauth2/token endpoint. We are also going to be supplying a valid client_id and client_secret at this point as well. All together it looks like this:

Typically when I am working through all of this I am running these commands outright (vs. having them being fed into a variable) in bash so I can see what all I am getting back and get better error output.

Once you have a valid access token, the next thing to do for our example use case is to query Azure for the current state of our specific storage account. This is done via a GET request to management.azure.com – targeting the storage account’s ARM API endpoint. That looks like this:

So we can pause right here because this next bit is somewhat odd… This particular API endpoint doesn’t take POST – it only takes PATCH (which Microsoft refers to as an update operation). PATCH lets you update/replace a subset of data – in our case the networkAcls.ipRules part of the json that defines the configuration of the storage account.

API documentation (which is your best, albeit sometimes annoying, friend) is here: https://docs.microsoft.com/en-us/rest/api/azure/. For example, this operation was found here:
https://docs.microsoft.com/en-us/rest/api/storagerp/storage-accounts/update.

The jq program has a modality where you can search a json array (in this case the value of the $sa variable is the json array we retrieved in the previous step) and then run a conditional test on it and return a boolean. In effect we are searching our current storage account NACL and checking if our current IP is already on the list.

If our IP is not on the list then we want to take the current list and append our IP to it. This is being stored in the $combined variable (which exists in the scope of the if/then statement). Because PATCH requires that I apply the fully qualified location of the change in the json array we are then building that json structure out in the $patch variable. We then take that value and use the API call with PATCH against the api endpoint of the storage account. Lastly, we echo a message back to the cli explaining what happened.

If, rather, the current IP is already on the list then we will leave well enough alone and echo a message back saying as much.

That all looks like this:

Conclusion

The manipulation of the data structures with jq was a lot of fun. The resulting code isn’t the easiest to read but the construction isn’t actually all that difficult as you play around with jq. The focus of this article for me however was more so to document some examples of calling the Azure ARM api. It’s one of the things I have had a consistently hard time finding information and examples for. Hopefully this is helpful. Cheers!