Getting Organised to Quickly Prototype Boto3 Automation Tools in Jupyter Notebooks

A little bit of organisation goes a long way. I would like to share with you some organisational practices I have evolved that in my experience make it quicker to move from a python/boto3 prototype to a production automation tool.

To demonstrate them I am going to walk through the creation of a python class that uses the AWS Cost Explorer API via Boto3 to return the total cost of an account. Maybe this particular use case is useful to you, maybe not in any event, it provides a good way to share some effective working practices.

Prototype Using Multiple Jupyter Notebooks

I am sure many of you are already using Jupyter Notebooks for prototyping automation tools. The way the output is shared dynamically in the notebook alongside your code makes it great for exploring functionality you may not be fully familiar with.

Something that may be new to you is you can run one notebook within another. The code to do this is:

%run another-notebook.ipynb

Now I am not suggesting you get carried away with this functionality but I am suggesting when developing a new automation tool you use a structure of three notebooks, namely:

• A notebook for your functions and/or classes
• A notebook to test your program with %run to include your classes/ function notebook
• A notebook to run you program with %run to include your classes/ functions notebook

You could call the notebooks: “run”, “test” and “my-class”

Defining Our a Demo Tool (Application)

I consider myself a “Toolmaker” hence the use of the word tool rather than application but I am sure you get what I mean.

Our demonstration tool will accept:

• A session (you could be using access keys or assuming a role to access a target account)
• An end date
• The number of days to go back from the end date

The tool will provide the total cost for the account identified in the profile for the time period defined by the inputs.

For this article, I have left it there. I have not extended the design to send the answer somewhere else. You could let your imagination run wild, providing your answer to the world through an API or SNS’ing it to a more select crowd.

Write a Failing Test

First, we need a use case to test against. I suggest using one month's costs. You can get the total cost for the target account from Cost Explorer via the console. You will hardcode the date of the first day of the next month, the number of days in the month, and the result i.e. the cost you get from Cost Explorer. Let's say you choose the month of October (with 31 days), your code in the “test” notebook will look like this:

%run my-class.ipynb
ce_test_value=123.00
assert accountCost(mysession,'2020-11-01',-31).cost \
                          == ce_test_value,'FAILED total cost test'

As intended it will fail (in many ways). The value for ce_test_value is the cost you looked up via the console in cost explorer.

Put a message in to make it clear it’s the assertion that failed and which one it is in case you end up with more than one.

There are two main reasons that the current test is failing:

• The “my-class.ipynb” notebook containing the definition of the class “accountCost” does not exist
• You have not define any way for the program to connect to a target account

Connecting to the Target AWS Account

If you are running locally I am assuming everyone is familiar with using access keys that are defined in a local ~.aws/credentials file to access accounts. My suggestion when doing this is don’t create the keys for the user you normally use to login to the console. Rather create a user specifically for the tool you are creating, create access keys for it and use those access key to create a named profile in your ~.aws/credentials file along the lines of the following:

[default]
aws_access_key_id=XXXXXXXXXXXXXXXX
aws_secret_access_key=aaaaaaAAAAAAbbbbbbBBBBBB
[my-named-profile]
aws_access_key_id=YYYYYYYYYYYYYY
aws_secret_access_key=cccccccCCCCCCddddddDDDDDD

This will mean you can define the profile you want to use explicitly in your program rather than relying on the default. You will be able to switch between accounts within your program and if you later want to assume a role moving to this will be less disruptive to your code.

For the new user you have created you are going to have to define an access policy. I think sometimes “least privilege” seems like a lot of hard work to figure out exactly what actions you need to allow. It is easier than you maybe think to find out what action you must allow. Simply create a blank policy, don’t allow anything, run your program, and generally the error message will tell you exactly what action you need to allow.

In this case, I will put you out of your misery by giving you the policy you require which you can attach as an inline policy. Here it is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ce:GetCostAndUsage"
            ],
            "Resource": "*"
        }
    ]
}

Generally, I would prefer using an AWS-managed policy but in the case of using the Cost and Usage API, none of the obvious candidates do the job and are in fact overly permissive for this narrow use case.

So now we have a way to connect to the target AWS account we can amend the “test” code to accommodate this as follows:

import boto3
# set the profile to be used
profile_name='my-named-profile'
my_session = boto3.Session(profile_name=profile_name)
%run my-class.ipynb
ce_test_value=123.09
assert accountCost(my_session,'2020-11-01', 31).cost \
                          == ce_test_value,'FAILED total cost test'

The only thing left to do is to create the class notebook and define the class within it.

Class to Get Your Account Costs from Cost Explorer Via Boto3

The class assumes input of:

• session
• date until
• days back

You can create the notebook “my-class” and add the following code:

import boto3
from dateutil import parser as p
from dateutil.relativedelta import relativedelta as rd
class accountCost:
    def __init__(self,session,mydate,mydays):
        
        # cost explorer requires region to us-east-1
        client = session.client('ce', region_name='us-east-1')
            
        #convert input end date from string
        e_date=p.parse(mydate)
        end_string= \
        f'{e_date.year}-{e_date.month:02}-{e_date.day:02}'
        
        #calculate start date
        s_date = (e_date + rd(days=mydays))
    
        #convert start date to string
        start_string= \
        f'{s_date.year}-{s_date.month:02}-{s_date.day:02}'
       
        #get cost from cost explorer api
        response = client.get_cost_and_usage(  
                TimePeriod={
                    'Start': start_string,
                    'End': end_string
                },
                Granularity="DAILY",
                Metrics=[
                    "UnblendedCost",
                ]
            )
        
        #total up days between start and end date
        total_cost=0.0
            
        for day in response['ResultsByTime']:
            total_cost=total_cost+float(day['Total']
                             ['UnblendedCost']['Amount'])
        
        #return result as string with two decimal points
        self.cost=f'{total_cost:.2f}'

Parsing the date makes the solution a little more robust in that it will accept a variety of date formats, standardising them in the program.

Putting it all Together

Having saved the “my-class” notebook you can go back to the “test” notebook and now run you run it you should get no errors. Note you will need to take into account that in order accommodate the technique used to fix the float to two decimals your test value needs to be converted to a string ( f’{ce_test_value}’ ). If in fact, you get nothing this means the assertion passed with the result of instantiating your object equals that of the value you retrieved using the console from Cost Explorer.

Now you can create the “run” notebook and yes at this stage it will look suspiciously similar to the “test” notebook. Here is my version:

# set the profile to be used
profile_name='my-named-profile'
my_session = boto3.Session(profile_name=profile_name)
%run my-class.ipynb
print(accountCost(my_session,'2020-11-01', 31).cost)

The benefit you have gained is that not only have you tested your class but you can also try things out in the “test” notebook as you develop your prototype further before incorporating them into your “run” notebook.

I hope you found this approach useful in stimulating your own thoughts on how to approach your next tool building exercise. Any thoughts or suggestions please feel free to reach out to me.