For the complete documentation index, see llms.txt. This page is also available as Markdown.

AWS ECS (Terraform)

Complete guide for deploying All ID on AWS ECS Fargate using Terraform.

Overview

This guide explains how to deploy All ID using Terraform Infrastructure as Code.

The deployment requires:

  • All ID Application Terraform (src/terraform/allid/) - Application services (ECS, RDS, ALB)

  • Existing VPC - You must provide a VPC with the required subnet structure

Network Requirements: The All ID Terraform modules require an existing VPC with public, private, and isolated subnets. If you don't have a VPC yet, Certta provides a reference Terraform project (src/terraform/shared/) that creates a production-ready network infrastructure.

Prerequisites

Before deploying, ensure you have:

Tools:

  • Terraform >= 1.5

  • AWS CLI configured with credentials

AWS Account:

  • AWS account with appropriate permissions

  • Permissions to create: ECS, RDS, ALB, CloudFormation, IAM roles, Security Groups

  • AWS credentials configured (aws configure)

Network Infrastructure - You have three options:

  1. Option A (Recommended): Deploy new VPC using the Certta reference Terraform project (src/terraform/shared/)

  2. Option B: Use existing VPC by manually creating SSM Parameters (see step 2B)

  3. Already have SSM Parameters: Skip directly to step 3 if parameters are already configured

Required VPC structure (regardless of option):

  • Public subnets (minimum 2 AZs) for Application Load Balancer

  • Private subnets with NAT (minimum 2 AZs) for ECS services

  • Isolated subnets (minimum 2 AZs) for RDS database

  • Internet Gateway and NAT Gateway configured

Container Images:

  • Access to Certta's private container registry

  • ECR repository URIs for Peer and Facematch images

  • Image versions/tags to deploy

Makefile Automation

This project includes Makefiles that simplify Terraform operations. Instead of running terraform commands directly, you can use make commands that automatically handle environment-specific configurations.

Benefits:

  • Consistent execution: Same commands work across all environments

  • Automatic navigation: No need to cd to environment directories

  • Built-in safety: Confirmation prompts for destructive operations

  • Easy multi-environment: Switch environments with ENV parameter

  • Self-documenting: Run make help to see all available commands

  • Error prevention: Validates environment names and required parameters

Throughout this guide, you'll see make commands instead of direct terraform commands. This is the recommended approach for managing this infrastructure.

Alternative: If you prefer using Terraform directly, you can still do so by navigating to the environment directory (e.g., cd environments/dev) and running standard terraform commands.

Quick example:

Safety example:

Project structure

Certta will provide you with Terraform project files. The package structure:

Shared infrastructure is optional: Only deploy if you need to create a new VPC. If you already have a VPC, skip the src/terraform/shared/ project and configure the VPC information via SSM Parameters.

Main Files per Environment

Each environment directory contains:

  • main.tf: Main configuration with provider settings and module invocations

  • variables.tf: All variable definitions with types and defaults

  • outputs.tf: Outputs of created resources (ALB DNS, database endpoint, etc.)

  • terraform.tfvars.example: Template with example values and comments

  • terraform.tfvars: Real values (not versioned in git, contains secrets)

  • ssm.tf (shared only): SSM Parameters for cross-project references

Step 1: Extract Terraform project

Extract the Terraform project files provided by Certta:

Step 2: Network infrastructure (Option A)

Choose your option:

  • This step (Option A): Deploy a new VPC using the Certta reference Terraform project

  • Step 2B (Option B): Use your existing VPC by creating SSM Parameters manually

Option A: Deploy new VPC with Terraform

If you need to create a new VPC, deploy the reference network infrastructure:

What gets deployed:

  • VPC with environment-specific CIDR blocks

  • 2 Public subnets (across 2 availability zones)

  • 2 Private subnets with NAT (across 2 availability zones)

  • 2 Isolated subnets (across 2 availability zones) for databases

  • Internet Gateway for public internet access

  • NAT Gateway for private subnet outbound access

  • SSM Parameters for cross-project references

Deployment time: ~5-10 minutes

After deployment, the VPC information is automatically stored in SSM Parameters:

The following SSM Parameters are created:

  • /shared/dev/vpc-id - VPC ID

  • /shared/dev/vpc-cidr - VPC CIDR block

  • /shared/dev/public-subnet-ids - Public subnet IDs (comma-separated)

  • /shared/dev/private-egress-subnet-ids - Private subnet IDs with NAT (comma-separated)

  • /shared/dev/private-isolated-subnet-ids - Isolated subnet IDs (comma-separated)

The All ID Terraform project automatically reads these SSM Parameters - no manual configuration needed!

Step 2B: Using Existing VPC (Option B)

Skip this step if you chose Option A (deploying new VPC with terraform-shared).

If you already have a VPC and don't want to use terraform-shared, you can create the SSM Parameters manually so the All ID Terraform project can find your VPC.

Prerequisites: Your VPC must have:

  • Public subnets (with route to Internet Gateway)

  • Private subnets with egress (with route to NAT Gateway)

  • Private isolated subnets (no internet route, for databases)

Get your VPC and subnet IDs

Use AWS CLI to identify your VPC and subnets:

Identify each subnet type

  • Public Subnets: Has route 0.0.0.0/0 → igw-xxxxx (Internet Gateway)

  • Private Egress Subnets: Has route 0.0.0.0/0 → nat-xxxxx (NAT Gateway)

  • Private Isolated Subnets: NO route to 0.0.0.0/0

Create SSM Parameters

Once you've identified your VPC and subnets, create the SSM Parameters:

Verify parameters were created

Delete parameters (if you need to redo)

Step 3: Configure application variables

Navigate to the application directory:

Copy the example variables file and configure your environment-specific values:

Edit environments/dev/terraform.tfvars and configure the following values:

Configuration values to update:

Field
Description
How to obtain

peer_ecr_repository_uri

Peer Service container registry

Provided by Certta

peer_version

Peer image tag to deploy

Provided by Certta (e.g., latest, commit hash)

facematch_ecr_repository_uri

Facematch container registry

Provided by Certta

facematch_version

Facematch image tag to deploy

Provided by Certta (e.g., latest, commit hash)

desired_count

Number of tasks per peer service

Set to 0 for dev (cost saving), 1+ for stg/prd

router_rest_url

Certta Router endpoint for your region/env

Provided by Certta

About desired_count: This variable controls the number of tasks for all three Peer services simultaneously (default, client-a, client-b). This ensures consistency across all peers.

Recommended values:

  • Development: 0 (no tasks running - saves costs when not in use)

  • Staging: 1 (one task per peer for testing)

  • Production: 2+ (multiple tasks per peer for high availability)

Note: Facematch service is hardcoded to 2 tasks. To change, edit the facematch module in environments/<env>/main.tf.

Router mTLS Certificates: The certificates are NOT configured in terraform.tfvars. Terraform creates empty secrets in AWS Secrets Manager, and you will populate them with actual certificates after deployment in Step 5.

Alternative approach: You can include certificates in terraform.tfvars if preferred, but be aware:

  • Values will be stored in Terraform state files (even if marked as sensitive)

  • State files must be encrypted and access-controlled

  • This approach is optional and not recommended for security reasons

Multiple environments: Create separate terraform.tfvars files for each environment (dev, stg, prd) with environment-specific values.

Step 4: Deploy All ID application

Deploy the All ID application services:

Type yes when prompted to confirm the deployment.

What gets deployed:

  1. ECS Cluster + Service Discovery (ecs-cluster module)

    • ECS Fargate cluster

    • Cloud Map private namespace (allid.local)

    • Container Insights enabled

  2. Database (database module)

    • Aurora MySQL Serverless v2 cluster

    • Database security group

    • Secrets Manager for credentials (auto-generated)

    • CloudWatch Logs for error, general, and slow query logs

    • Performance Insights enabled

  3. Load Balancer (load-balancer module)

    • Application Load Balancer (internet-facing)

    • HTTP listener (port 80)

    • Security group with IP whitelisting

    • Default action: 403 Access Denied

  4. Facematch Service (facematch-service module)

    • ECS Task Definition (CPU: 1024, Memory: 2048 MB)

    • Fargate Service with 2 instances

    • Cloud Map service registration (facematch.allid.local)

    • Security group for internal communication

    • IAM roles for execution and task

  5. Peer Services (peer-service module - instantiated 3 times)

    • Creates 3 peer instances (default, client-a, client-b)

    • Each peer has:

      • ECS Task Definition (CPU: 256, Memory: 512 MB)

      • Fargate Service with configurable instance count

      • ALB target group with path-based routing

      • Cloud Map service registration

      • Database connection to RDS

      • Router mTLS certificate stored in Secrets Manager

      • IAM roles for execution and task

      • CloudWatch Log Group

Deployment time: ~15-20 minutes

Terraform automatically handles resource dependencies and deploys them in the correct order. You can monitor the progress in the terminal output.

Step 5: Configure Router mTLS certificates

After deploying the infrastructure, you need to populate the Router mTLS certificates in AWS Secrets Manager. Terraform creates empty secrets that must be filled with actual certificate content.

Why certificates are not in Terraform:

  • Keeps sensitive credentials out of Terraform state files

  • Allows certificate rotation without Terraform changes

  • Follows security best practices for secrets management

Alternative: You can include certificates in terraform.tfvars if your security policies allow it, but this will store them in the Terraform state file.

Configure certificates for each peer

Option 1: AWS Console (recommended)

  1. Navigate to Secrets Manager Console → Secrets

  2. Find and click on each secret:

    • {prefix}-peer-default-router-certificate

    • {prefix}-peer-client-a-router-certificate

    • {prefix}-peer-client-b-router-certificate

  3. Click "Retrieve secret value" → "Edit"

  4. Update the JSON with your actual certificates:

  1. Save changes

Option 2: AWS CLI

Step 6: Initialize database

The Peer Service requires a database schema to be initialized. Certta will provide a SQL dump file that must be restored to the Aurora MySQL database.

Multi-tenant setup: Each peer instance uses its own database. The database names follow the pattern peer-{name}:

  • peer-default (for default peer)

  • peer-client-a (for client-a peer)

  • peer-client-b (for client-b peer)

Database access methods

The Aurora MySQL database is deployed in isolated subnets with no direct internet access. You need to establish a secure connection to access it.

Option 1: Bastion Host (Recommended for production)

Deploy a bastion host (EC2 instance) in a public subnet to act as a jump server:

  1. Launch an EC2 instance in a public subnet of your VPC

  2. Configure security groups to allow:

    • SSH access from your IP to the bastion host

    • MySQL access from bastion host to the RDS security group

  3. Connect to the database through SSH tunnel:

Option 2: VPN Connection

If you have a VPN connection configured to your VPC:

  1. Connect to your VPN

  2. Access the database directly using its private endpoint

Option 3: AWS Systems Manager Session Manager

Use Session Manager for secure access without exposing SSH ports:

  1. Ensure your bastion host has SSM agent installed

  2. Grant necessary IAM permissions

  3. Create port forwarding session:

Get database endpoint and credentials

Option 1: Terraform outputs

Option 2: AWS Console

  1. Navigate to RDS Console → Databases

  2. Find the Aurora cluster (look for name with allid-database)

  3. Copy the Writer endpoint (e.g., allid-database-cluster.cluster-xxx.us-east-1.rds.amazonaws.com)

  4. Navigate to Secrets Manager Console → Secrets

  5. Find the database secret (look for name with allid-database)

  6. Click "Retrieve secret value" to see username and password

Option 3: AWS CLI

Create and initialize databases

You can use any of the following tools to create databases and restore the dump:

Option A: MySQL Command Line Client

Best for automation and CI/CD pipelines:

Option B: DBeaver (GUI Tool)

Recommended for visual database management:

  1. Create a new connection:

    • Database: MySQL

    • Host: {DB_ENDPOINT} (from above)

    • Port: 3306

    • Username: {DB_USERNAME} (from Secrets Manager)

    • Password: {DB_PASSWORD} (from Secrets Manager)

  2. Create databases:

    • Right-click on connection → SQL Editor → New SQL Script

    • Execute:

  3. Restore dump:

    • Right-click on each database → Tools → Execute Script

    • Select the allid-dump.sql file provided by Certta

    • Click "Start" to execute

Option C: MySQL Workbench (GUI Tool)

Alternative GUI tool for MySQL management:

  1. Create a new connection:

    • Connection Name: All ID Database

    • Hostname: {DB_ENDPOINT}

    • Port: 3306

    • Username: {DB_USERNAME}

    • Password: Store in Keychain/Vault

  2. Create databases:

    • Open connection → Query tab

    • Execute:

  3. Import dump:

    • Server → Data Import

    • Select "Import from Self-Contained File"

    • Choose allid-dump.sql file

    • Select target database (repeat for each: peer-default, peer-client-a, peer-client-b)

    • Click "Start Import"

Connection troubleshooting: If you cannot connect to the database, ensure:

  • You have established proper access (bastion host, VPN, or Session Manager)

  • The RDS security group allows connections from your source

  • The database endpoint and credentials are correct

Step 7: Retrieve endpoints

After deployment completes, get the application endpoint:

Important outputs:

Output
Description
Example

alb_dns_name

Load Balancer DNS name

allid-load-balancer-123456789.us-east-1.elb.amazonaws.com

database_endpoint

Aurora MySQL cluster endpoint

allid-database-cluster.cluster-xxx.us-east-1.rds.amazonaws.com

database_secret_arn

Secrets Manager ARN for DB credentials

arn:aws:secretsmanager:...

ecs_cluster_name

ECS cluster name

allid-cluster

peer_default_service_name

Peer default service name

allid-peer-default

facematch_service_name

Facematch service name

allid-facematch

The deployment creates 3 peer instances with path-based routing for multi-tenant support. Each peer has its own isolated endpoint with path prefix.

API endpoints (3 peer instances by default):

Health check endpoints:

Step 8: Validate deployment

Test that all services are healthy:

Test peer endpoints (using ALB DNS from Step 7):

All endpoints should return HTTP 200 with status information.

Check ECS services status:

AWS Console:

  1. Navigate to ECS Console → Clusters → allid-cluster

  2. Click on "Services" tab

  3. Verify all services show "Running" status and desired count matches running count

AWS CLI:

All services should show runningCount matching desiredCount and deployment status PRIMARY.

Check CloudWatch Logs (if services fail to start):

AWS Console:

  1. Navigate to CloudWatch Console → Log groups

  2. Find log groups: /ecs/allid/peer-default, /ecs/allid/facematch, etc.

  3. Check recent log streams for errors

AWS CLI:

Makefile Commands

Both src/terraform/allid/ and src/terraform/shared/ include a Makefile with automation commands for easier infrastructure management.

Getting Started: Run make help in either project directory to see all available commands and their descriptions.

Available Commands

Environment Selection

All commands support the ENV parameter to specify the target environment:

Default environment: If you don't specify ENV, the Makefile defaults to dev.

Shared Infrastructure Commands

The shared infrastructure Makefile includes an additional command to view SSM parameters:

Workflow Example

Here's a typical workflow using Makefile commands:

Quick Reference

Most Common Commands

Environment Management

Common Tasks

Deploy to new environment:

Update container images:

Scale services:

View logs:

Architecture overview

Network layers

The deployment creates a three-tier network architecture:

Service communication

Path-based routing (multi-tenant)

The ALB uses path-based routing to support multiple peer instances:

Each peer instance:

  • Has its own ECS service

  • Has its own isolated database (peer-default, peer-client-a, peer-client-b)

  • Operates independently from other peers

  • Shares the same Facematch service pool

Configuration management

The Terraform modules automatically configure environment variables and secrets for all services.

Environment variables are defined in the peer-service module and include:

  • Database connection (host, port, database name)

  • Facematch service endpoint

  • Router Service URL

  • Feature flags (RabbitMQ, Redis, Router communication)

Secrets are injected via ECS task secrets and include:

  • Database credentials (auto-generated by Terraform)

  • Router mTLS certificates (provided via terraform.tfvars)

See Configuration for complete list of environment variables, required values, and configuration details.

Secrets management

Terraform automatically creates secrets in AWS Secrets Manager:

Secret naming convention:

  • Database credentials: {prefix}-database-secret (e.g., allid-database-secret)

    • Auto-generated by Terraform during deployment

  • Router certificates: {prefix}-peer-{name}-router-certificate (e.g., allid-peer-default-router-certificate)

    • Created empty by Terraform, populated manually after deployment (see Step 5)

Router certificates security approach:

By default, Router certificate secrets are created empty and populated manually after deployment. This approach:

  • ✅ Keeps sensitive credentials out of Terraform state files

  • ✅ Allows certificate rotation without Terraform changes

  • ✅ Follows security best practices for secrets management

Alternative: You can include certificates in terraform.tfvars if needed, but be aware that they will be stored in Terraform state files (even if marked as sensitive). If you choose this approach:

  • Store state files in encrypted S3 buckets with restricted access

  • Use state locking with DynamoDB

  • Never commit state files to version control

  • Limit access to state files to authorized personnel only

Security

Terraform automatically configures security groups and IP whitelisting following least-privilege principles.

Security groups:

  • ALB accepts traffic only from whitelisted IPs

  • Peer Services accept traffic only from ALB and internal VPC

  • Facematch accepts traffic only from Peer Services

  • Database accepts traffic only from Peer Services

IP whitelisting:

  • ALB is configured to accept traffic only from Certta Router IP addresses

  • Configure additional IPs in terraform.tfvars via the alb_allowed_cidr_blocks variable

  • Complete list of Certta Router IP addresses

  • Network segmentation recommendations

  • Secrets management best practices

  • Additional security hardening options

Updating the deployment

Update container images

  1. Update image versions in terraform.tfvars:

  1. Apply the changes:

Terraform will update the ECS task definitions and trigger a rolling deployment.

Update environment variables

  1. Edit the peer-service module in src/terraform/allid/modules/peer-service/main.tf

  2. Modify the environment variables in the task definition

  3. Apply the changes:

This will update all peer instances (default, client-a, client-b) with the new configuration.

Rotate Router mTLS certificates

If you need to rotate the Router mTLS certificates (certificate expiration, security incident, etc.), follow the same process as the initial configuration in Step 5.

When to rotate certificates:

  • Certificate expiration/renewal

  • Security incident requiring certificate replacement

  • Moving between Certta environments (dev/stg/prd)

  • Compliance requirements for periodic rotation

The application will automatically pick up the new certificates from Secrets Manager without requiring a service restart.

Scale services

Scale Peer Services: Edit environments/dev/terraform.tfvars and change the desired_count variable:

Then apply:

Scale Facematch Service: Edit the facematch module in environments/dev/main.tf:

Then apply:

Add/remove peer instances

To add a new peer instance:

  1. Edit the environment main.tf file (e.g., src/terraform/allid/environments/dev/main.tf):

  1. Add output for the new peer in outputs.tf:

  1. Apply the changes:

  1. Create and initialize the database for the new peer (see Step 6):

The new Peer Service will automatically have its Router certificate secret created by Terraform using the same certificate from terraform.tfvars. The service will become healthy after you complete database initialization.

Troubleshooting

Error: SSM Parameter not found

Cause: Network infrastructure (src/terraform/shared) has not been deployed, or SSM Parameters were not created.

Solution:

  1. Verify if parameters exist:

  1. If parameters don't exist, deploy the shared infrastructure:

  1. If you have an existing VPC, create SSM Parameters manually:

Peer Service won't start

Check CloudWatch Logs:

Common causes:

  1. Database not initialized: See Step 6 (database initialization and dump restoration)

  2. Database connection failed: Check security groups and RDS endpoint

  3. Facematch service unavailable: Check Facematch service status

Check task stopped reason:

AWS Console:

  1. ECS Console → Clusters → allid-cluster

  2. Click on service (e.g., allid-peer-default)

  3. Go to "Tasks" tab → Click on stopped tasks

  4. Check "Stopped reason" field

AWS CLI:

Health check failures

Check ECS service status:

Check ALB target health:

AWS CLI:

Common issues:

  • Security groups blocking traffic between ALB and Peer Services

  • Service not registered in Cloud Map (DNS resolution fails)

  • Database connection errors (check credentials secret)

  • Facematch service not responding

  • Health check grace period not long enough

No tasks running / Services showing 0/0

Cause: desired_count is set to 0 in terraform.tfvars.

Solution:

This is often intentional for development environments to save costs. To start tasks:

  1. Edit environments/dev/terraform.tfvars:

  1. Apply changes:

  1. Verify tasks are starting:

Cost saving tip: Set desired_count = 0 when not actively using development environments to stop all ECS tasks and reduce infrastructure costs.

Can't access ALB

Check if your IP is whitelisted:

If you get connection timeout or 403 Forbidden, your IP is not whitelisted.

Add your IP to the allowlist:

  1. Edit environments/dev/terraform.tfvars:

  1. Apply changes:

See Security Best Practices for the complete list of Certta Router IP addresses that should be whitelisted.

Cleanup

To remove all resources and stop incurring costs:

Targeted destruction: If you need to destroy specific resources in order, you can navigate to the environment directory and use targeted destroy:

Create database snapshot before cleanup (optional):

Environment Standardization

All environments (dev, stg, prd) follow the same base structure and configuration patterns. This standardization simplifies management and reduces errors when promoting changes across environments.

Standardized Components

  1. Module Structure: All environments use the same modules with the same parameters

  2. Tag Naming: Lowercase with hyphens (e.g., environment, managed-by, business-unity-id)

  3. Default Values: Consistent defaults across environments, customizable via terraform.tfvars

  4. Resource Naming: {prefix}-{resource-type} pattern

Environment-Specific Customization

Customize each environment by adjusting values in environments/<env>/terraform.tfvars:

Development (dev):

Staging (stg):

Production (prd):

Default Tags

All resources are automatically tagged via AWS provider default_tags:

Additional resource-specific tags can be added through module parameters.

Promoting Changes Across Environments

Best practice for promoting changes from dev → stg → prd:

  1. Test in Development:

  2. Promote to Staging:

  3. Deploy to Production:

Detailed File Structure

All ID Application (src/terraform/allid/)

Shared Infrastructure (src/terraform/shared/)

Key Files Explained

main.tf:

  • Terraform and AWS provider configuration

  • Module invocations with parameters

  • Data sources for SSM Parameters (allid project)

  • Identical structure across environments

variables.tf:

  • All variable definitions with types and defaults

  • Standardized defaults across environments

  • Inline documentation for each variable

outputs.tf:

  • Outputs of created resources

  • Used for retrieving values after deployment

  • Referenced by dependent modules or external systems

terraform.tfvars.example:

  • Template with all configurable values

  • Explanatory comments for each variable

  • Environment-specific example values

terraform.tfvars (not versioned):

  • Real environment values

  • Contains secrets (certificates, private keys)

  • Must be created from .example template

  • Never commit this file to git

ssm.tf (shared only):

  • Creates SSM Parameters for VPC and subnet IDs

  • Enables other projects to lookup resources

  • Format: /shared/{environment}/{parameter-name}

Next steps

Last updated