Menu Close
Menu Close

Want to know more about this blog?

Tom de Brouwer will happily answer all your questions.

Get in touch

Jul 07, 2020

In a series of blog posts we will focus on some of the best practices we use within Merapar to evolve our DevOps practices we have built around the AWS platform. Why? Because we think it’s fun to share knowledge and to learn from others in the industry.

Situation

In this blog post we will focus on how we set up monitoring using a single AWS CloudWatch dashboard for a project where we have multiple AWS CloudFront Distributions.

Situation-In-this-blog-post-we-will-focus-on-how-we-set-up-monitoring-using-a-single-AWS-CloudWatch-dashboard-for-a-project-where-we-have-multiple-AWS-CloudFront-Distributions-1024x598

The metrics that need to be monitored will be the same for all of the Distributions as the type of traffic is very similar. As the number of Distributions will change often we need a flexible solution to prevent repetitive tasks of adding / removing references to the Distribution in our CloudFormation templates. We also have a strong need to have the same code base between all our DTAP environments.

Requirements:

  • Creation of AWS CloudWatch dashboard must be fully automated
  • Multiple CloudFront Distributions within a single dashboard
  • No code changes required on CloudFormation dashboard to add new Distributions

Solution

The solution we have chosen uses a CloudFormation template and the macro functionality to perform the required transformations to the template.

What are CloudFormation macros

CloudFormation macros are simply AWS Lambda functions that you create and can be used to transform your input CloudFormation template. This gives you the ability to do search and replace operations or to transform the template completely.

Macros are run after uploading the template and before applying it. It is possible to see both the input and the processed template within the AWS CloudFormation console to make it easier to debug.

It is possible to have a macro process the complete template using the Transform section

Full-template-scope-from-AWS-documentation--1024x659

Or to limit the scope to a smaller part of the template using Fn::Transform

Limited-template-scope-from-AWS-documentation--1024x642

For more specific information about AWS CloudFormation macros take a look at this user guide from AWS.

CloudFormation template

For our solution we have chosen to use the smaller scoped solution and we created to following template:

AWSTemplateFormatVersion: "2010-09-09"
Description: 'AWS CloudFront dashboard'

Parameters:
  DistributionConfiguration:
    Type: String
    Description: "JSON object like: [ { \"distributionId\" : \"A1234\", \"domain\" : \"cloudfront.example.com\"}, 
                { \"distributionId\" : \"A4123\", \"domain\" : \"secondcloudfront.example.com\"}
              ]"
Resources:
  CloudFrontDashboard:
    Type: AWS::CloudWatch::Dashboard
    Fn::Transform: 
        Name: ExpandDashboard
        Parameters: 
          Distribution: !Ref DistributionConfiguration
    Properties:
      DashboardName: "CloudFrontDashboard"
      # The region here is fixed, since the metrics are stored in us-east-1.
   DashboardBody: !Sub
        - |
          {
            "widgets": [
                {
                    "type": "metric",
                    "x": 12,
                    "y": 0,
                    "width": 12,
                    "height": 6,
                    "properties": {
                        "metrics": ###5XXERRORRATE###,
                        "view": "timeSeries",
                        "stacked": false,
                        "region": "${AWS::Region}",
                        "title": "Per distribution 5xx error rates %",
                        "stat": "Maximum",
                        "period": 300
                    }
                },
                {
                    "type": "metric",
                    "x": 12,
                    "y": 6,
                    "width": 12,
                    "height": 6,
                    "properties": {
                        "metrics": ###4XXERRORRATE###,
                        "view": "timeSeries",
                        "stacked": false,
                        "region": "${AWS::Region}",
                        "title": "Per distribution 4xx error rate %",
                        "stat": "Maximum",
                        "period": 300
                    }
                },
                {
                    "type": "metric",
                    "x": 0,
                    "y": 0,
                    "width": 12,
                    "height": 6,
                    "properties": {
                        "metrics": ###REQUESTS###,
                        "view": "timeSeries",
                        "stacked": false,
                        "region": "${AWS::Region}",
                        "stat": "Sum",
                        "period": 300,
                        "title": "Per distribution request count (Sum)"
                    }
                }
           }

The Fn::Transform is used to scope just the dashboard and the Distribution variable is passed to the transform. We have used the ###xxx### markers to easily find and replace the values we need to transform.

In our setup there is a deployment script that will first deploy all required CloudFront Distributions based on a configuration file and once completed, it will deploy the dashboard with that same list. This list of Distributions is passed as the DistributionConfiguration parameter with a JSON object as input for this template.

Macro Lambda code

The ExpandDashboard macro is a Python3 AWS Lambda that is deployed separately from the CloudFront Distributions and the CloudWatch dashboard.

import json
import traceback
import dataclasses
@dataclasses.dataclass
class CloudFrontMetric:
    kind: str
    distribution_id: str
    label: str
    index: int
    def getMetric(self):
        return [
                "AWS/CloudFront", self.kind,
                "Region", "Global",
                "DistributionId", self.distribution_id,
                { 
                    "region": "us-east-1", #us-east-1 because of CloudFront
                    "yAxis": "right", 
                    "id": f"m{self.index}", 
                    "visible": False, 
                    "label": self.label 
                }
            ]
def replaceMarker(original, marker, json_input):
    return original.replace(marker, json.dumps(json_input).replace("\"", "\\\""))
def handler(event, _):
    print(event)
    region = event["region"]
    # Add the visible normalized values
    fivexx_error_rate = [ [{ "expression": "FILL(METRICS(), 0)", "label": "[avg: ${!AVG}]", "id": "e1", "region": region } ]]
    fourxx_error_rate = [ [{ "expression": "FILL(METRICS(), 0)", "label": "[avg: ${!AVG}]", "id": "e1", "region": region }]]
    requests = [[{ "expression": "FILL(METRICS(), 0)", "label": "[avg: ${!AVG}]", "id": "e1", "region": region } ]]
    # Add the invisible raw metrics for each distribution
    distributions = json.loads(event["params"].get('Distribution'))
    index = 0
    while index < len(distributions):
        distribution = distributions[index]
        print(distribution)
        distribution_id = distribution.get('distributionId')
        domain = distribution.get('domain')
        if distribution_id and domain:
            fivexx_error_rate.append(CloudFrontMetric("5xxErrorRate", distribution_id, domain, index).getMetric())
            fourxx_error_rate.append(CloudFrontMetric("4xxErrorRate", distribution_id, domain, index).getMetric())
            requests.append(CloudFrontMetric("Requests", distribution_id, domain, index).getMetric())
        index += 1
    final_fragment = event["fragment"]
    # First convert the input to text, this makes it easier to do replacements.
    jsonText = json.dumps(final_fragment)
    print(jsonText)
    # Replace 
    jsonText = replaceMarker(jsonText, "###5XXERRORRATE###", fivexx_error_rate)
    jsonText = replaceMarker(jsonText, "###4XXERRORRATE###", fourxx_error_rate)
    jsonText = replaceMarker(jsonText, "###REQUESTS###", requests)
    print(jsonText)
    # Convert back to JSON
    final_fragment = json.loads(jsonText)
    response = {
        "requestId": event["requestId"],
        "status": "success",
        "fragment": final_fragment
    }
    return response

This Lambda function will loop through the input JSON object and add the configuration required for the 4xx, 5xx errors and request widgets for all distributions.

Dashboard

This is what the dashboard will look like:

Dashboard-600x1024