--- title: SAA-C03 summary: Digitizing physical notes updatedAt: 1732688161000 --- This is a digital copy of my physical notes taken while studying for AWS Certifications. Not all content is here since I've only taken notes on services which I haven't actively used during my time with AWS. I've given every service a heading but notes may be terse if I am already familiar with the service. - [EC2](#ec2---elastic-compute-cloud) - [Fundamentals](#fundamentals) - [Security Groups](#security-groups) - [AWS SDK Setup](#aws-sdk-setup) - [Purchasing Options](#purchasing-options) - [Spot Instances](#spot-instances) - [Placement Groups](#placement-groups) - [Elastic Network Interface](#elastic-network-interface) - [Hibernation](#hibernation) - [AMI (Amazon Machine Image)](#ami---amazon-machine-image) - [EBS (Elastic Block Store)](#ebs---elastic-block-store) - [Volume Types](#volume-types) - [EBS MultiAttach](#ebs-multiattach) - [EBS Encryption](#ebs-encryption) - [EFS (Elastic File System)](#efs---elastic-filesystem) - [ELB (Elastic Load Balancer)](#elb---elastic-load-balancer) - [CLB (Classic Load Balancer)](#clb---classic-load-balancer) - [ALB (Application Load Balancer)](#alb---application-load-balancer) - [NLB (Network Load Balancer)](#nlb---network-load-balancer) - [GWLB (Gateway Load Balancer)](#gwlb---gateway-load-balancer) - [Sticky Sessions](#sticky-sessions) - [Cross Zone Load Balancing](#cross-zone-load-balancing) - [SSL/TLS on ELB](#ssltls-on-elb) - [Connection Draining](#connection-draining) - [ASG (Auto Scaling Groups)](#asg---auto-scaling-groups) - [RDS (Relational Database Service)](#rds---relational-database-service) - [Read Replicas](#read-replicas) - [RDS Multi AZ](#rds-multi-az) - [RDS Custom](#rds-custom) - [RDS Backups](#rds-backups) - [Aurora](#aurora) - [Aurora Serverless](#aurora-serverless) - [Global Aurora](#global-aurora) - [Aurora ML](#aurora-ml) - [Aurora Backups](#aurora-backups) - [Aurora Clones](#aurora-clones) - [RDS Security](#rds-security) - [RDS Proxy](#rds-proxy) - [Elasticache](#elasticache) - [Elasticache Security](#elasticache-security) - [Route53](#route-53) - [Elastic Beanstalk](#elastic-beanstalk) - [S3 (Simple Storage Service)](#s3---simple-storage-service) - [S3 Replication](#s3-replication) - [Requester Pays](#requester-pays) - [S3 Transfer Acceleration](#s3-transfer-acceleration) - [S3 Byte Range Fetching](#s3-byte-range-fetching) - [S3 Encryption Options](#s3-encryption-options) - [S3 MFA Delete](#s3-mfa-delete) - [S3 Glacier Vault Lock](#s3-glacier-vault-lock) - [S3 Access Points](#s3-access-points) - [S3 Object Lambda](#s3-object-lambda) - [Cloudfront](#cloudfront) - [Cloudfront Pricing](#cloudfront-pricing) - [Global Accelerator](#global-accelerator) - [Amazon FSx](#amazon-fsx) - [Hybrid Cloud](#hybrid-cloud) - [AWS Storage Gateway](#aws-storage-gateway) - [AWS Transfer Family](#aws-transfer-family) - [AWS DataSync](#aws-datasync) - [SQS (Simple Queue Service)](#sqs---simple-queue-service) - [SQS Message visibility](#sqs-message-visibility) - [Long Poling](#long-poling) - [FIFO Queue](#fifo-queue) - [SQS and ASG](#sqs-and-asg) - [SNS (Simple Notification Service)](#sns---simple-notification-service) - [SNS Fan out pattern](#sns-fan-out-pattern) - [SNS FIFO](#sns-fifo) - [SNS Filters](#sns-filters) - [Kinesis](#kinesis) - [Kinesis Data Streams](#kinesis-data-streams) - [Kinesis Data Stream security](#kinesis-data-stream-security) - [Kinesis Data Firehose](#kinesis-data-firehose) - [Data Streams vs Firehose](#data-streams-vs-firehose) - [Ordering Kinesis data](#ordering-kinesis-data) - [Kinesis vs SQS - FIFO](#kinesis-vs-sqs---fifo) - [SQS vs SNS vs Kinesis](#sqs-vs-sns-vs-kinesis) - [Kinesis Data Analytics](#kinesis-data-analytics) - [Kinesis Data Analytics for Apache Flink](#kinesis-data-analytics-for-apache-flink) - [Amazon MQ](#amazon-mq) - [Containers](#containers) - [Amazon ECR](#amazon-ecr) - [Amazon ECS](#amazon-ecs) - [ECS IAM Roles](#ecs-iam-roles) - [ECS Load Balancing](#ecs-load-balancing) - [ECS Data Volumes](#ecs-data-volumes-efs) - [ECS Auto Scaling](#ecs-autoscaling) - [EC2 Launch Type Scaling](#ec2-launch-type-scaling) - [Amazon EKS](#amazon-eks) - [EKS Data Volumes](#eks-data-volumes) - [AWS App Runner](#aws-app-runner) - [Lambda](#lambda) - [Lambda Limits](#lambda-limits) - [Lambda Snap Start](#lambda-snap-start) - [Edge Functions](#edge-functions) - [Cloudfront Functions](#cloudfront-functions) - [Lambda@Edge](#lambdaedge) - [Cloudfront Functions vs Lambda@Edge](#cloudfront-functions-vs-lambdaedge) - [Lambda in a VPC](#lambda-in-a-vpc) - [Lambda with RDS Proxy](#lambda-with-rds-proxy) - [DynamoDB](#dynamodb) - [DynamoDB capacity modes](#dynamodb-capacity-modes) - [DynamoDB accelerator (DAX)](#dynamodb-accelerator-dax) - [DynamoDB Stream processing](#dynamodb-stream-processing) - [Global Tables](#global-tables) - [DynamoDB TTL](#dynamodb-ttl) - [DynamoDB Backups](#dynamodb-backups) - [API Gateway](#api-gateway) - [API Gateway endpoint types](#api-gateway-endpoint-types) - [API Gateway security](#api-gateway-security) - [Step Functions](#step-functions) - [Cognito](#cognito) - [Choosing the right Database](#choosing-the-right-database) - [Amazon Keyspaces](#amazon-keyspaces) - [Athena](#athena) - [Athena Federated Query](#athena-federated-query) - [Redshift](#redshift) - [Redshift Spectrum](#redshift-spectrum) - [Amazon Elastic Map Reduce (EMR)](#amazon-elastic-map-reduce-emr) - [Quicksight](#quicksight) - [AWS Glue](#aws-glue) - [Glue Crawler](#glue-crawler) - [Lake Formation](#lake-formation) - [Amazon Managed Streaming for Apache Kafka (MSK)](#amazon-managed-streaming-for-apache-kafka-msk) - [Kinesis vs MSK](#kinesis-vs-kafka) - [Amazon Rekognition](#amazon-rekognition) - [Rekognition moderation](#rekognition-moderation) - [AWS Transcribe](#aws-transcribe) - [AWS Polly](#aws-polly) - [Translate](#translate) - [Lex + Connect](#lex--connect) - [AWS Comprehend](#aws-comprehend) - [Comprehend Medical](#comprehend-medical) - [Sagemaker](#sagemaker) - [Forecast](#forecast) - [Amazon Personalize](#amazon-personalise) - [Cloudwatch](#cloudwatch) - [Cloudwatch Metrics](#cloudwatch-metrics) - [Cloudwatch Agent](#cloudwatch-agent) - [Cloudwatch Unified Agent](#cloudwatch-unified-agent) - [Cloudwatch Contributor Insights](#cloudwatch-contributor-insights) - [Event Bridge](#event-bridge) - [Cloudtrail](#cloudtrail) - [AWS Config](#aws-config) - [Cloudwatch vs Cloudtrail vs Config](#cloudwatch-vs-cloudtrail-vs-config) - [AWS Organizations](#aws-organizations) - [IAM Conditions](#iam-conditions) - [IAM Roles vs Resource based Policies](#iam-roles-vs-resource-based-policies) - [IAM Permission Boundaries](#iam-permission-boundaries) - [AWS Active Directory Services](#aws-active-directory-services) - [AWS Control Tower](#aws-control-tower) - [KMS](#kms) - [Multi-Region KMS](#multi-region-kms) - [S3 Replication Encryption](#s3-replication-encryption) - [Sharing Encrypted AMIs](#sharing-encrypted-amis) - [SSM Parameter Store](#ssm-parameter-store) - [Secrets Manager](#secrets-manager) - [Amazon Certificate Manager (ACM)](#amazon-certificate-manager-acm) - [Web Application Firewall (WAF)](#web-application-firewall-waf) - [AWS Shield](#aws-shield) - [AWS Firewall Manager](#aws-firewall-manager) - [Guard Duty](#guard-duty) - [Amazon Inspector](#amazon-inspector) - [Macie](#macie) - [VPCs](#vpcs) - [Default VPC](#default-vpc) - [Subnets](#subnets) - [Internet Gateway](#internet-gateway) - [Bastion Hosts](#bastion-hosts) - [NAT Instances](#nat-instances) - [NAT Gateway](#nat-gateway) - [NACL & Security Groups](#nacl--security-groups) - [Security Groups vs NACLs](#security-groups-vs-nacls) - [VPC Peering](#vpc-peering) - [VPC Endpoints (AWS PrivateLink)](#vpc-endpoints-aws-privatelink) - [VPC Flow Logs](#vpc-flow-logs) - [AWS VPN CloudHub](#aws-vpn-cloudhub) - [Direct Connect (DX)](#direct-connect-dx) - [Site to Site VPN as a backup](#site-to-site-vpn-as-a-backup) - [Transit Gateway](#transit-gateway) - [VPC Traffic Mirroring](#vpc-traffic-mirroring) - [IPv6 for VPCs](#ipv6-for-vpc) - [Egress only Internet Gateway](#egress-only-internet-gateway) - [Disaster Recovery](#disaster-recovery) - [Database Migration Service (DMS)](#database-migration-service-dms) - [AWS Backup](#aws-backup) - [Elastic Network Adapter (ENA)](#elastic-network-adapter-ena) - [Elastic Fabric Adapter (EFA)](#elastic-fabric-adapter-efa) - [AWS Parallel Cluster](#aws-parallel-cluster) - [Amazon Pinpoint](#amazon-pinpoint) # EC2 - Elastic Compute Cloud - Can run Mac, Linux or Windows - By Default, when terminated will delete all attached storage - Naming Convention consists of "Instance Class" -> "Instance Generation" -> "Size" ```mermaid flowchart LR start(Start Here) --> instanceClass instanceClass[M] --> generation[5] generation --> size[2xLarge] instanceLabel("Instance Class") --> instanceClass generationLabel("Instance Generation (Improves over time)") --> generation sizeLabel("Size within instance class") --> size ``` ## Fundamentals - Setting up a budget is extremely important! - By Default IAM users cannot see billing information & it must be enabled in root - EC2 is made up of many parts - EC2 - Virtual Machines - EBS - Storage - ELB - Load Balancing - ASG - Auto Scaling ## Security Groups - Only contains "Allow rules" - Is like a firewall as in - it specifies inbound/outbound rules - Can also specify access rules for other EC2 related services - Many to Many - Security groups are **per region** - Default inbound **_all_** is blocked - Default outbound **_all_** is allowed ## AWS SDK Setup - Never manually run `aws configure` from the EC2 - Instead, associate an IAM role with the instance - EC2 will inherit perms from the role associated with instance ## Purchasing Options - Multiple ways of purchasing an EC2 instance - On Demand - Reserved (1-3 years) - Convertable type available - Allows changing EC2 type - Savings (1-3 years) - Instead of reserving specific instances, you dedicate a set spend - Spot instances - These instances can die at any point but are extremely cheap. - It's possible to sell reserved instances on the AWS Marketplace ## Spot Instances - Using Amazon "Spot Requests", it can handle automatically launching spot instances if you've configured the request to be persistent - Spot Requests will automatically purchase and configure a spot instance if it falls within a set price range - Using "Spot Fleets" you can purchase groups of instances at the lowest price - Fleets is different to requests, both are automatic however fleets can pick and choose instance types to fit within your budget. ## Placement Groups - Placement groups can control how AWS will provision your groups of EC2s - Cluster - Same AZ / Will have low latency (Literally same rack) - Spread - Across different physical hardware (Same AZ) - **_Maximum 7_** instances per AZ - Partition - Spread Cluster/Spread groups within an AZ - **_Maximum 7_** per AZ, can be spread across regions ## Elastic Network Interface - Virtual Network card - 1x Mac Address - 1x Private IP Address - Infinite Secondary IP Addresses - Infinite Security Groups - Bound to a set AZ - ENIs can be moved between EC2s and will retain Mac & IPs, allows for manual failovers etc. ## Hibernation - Different ways of "turning off" EC2 instances - Terminate - Data is destroyed - Stop - Disk is preserved for next start - Hibernate - RAM & Disk is preserved for next start - RAM is written to disk, there must be enough free space on disk to hibernate ## AMI - Amazon Machine Image - Customised EC2 instances, running own OS/Software/Etc - Faster boots than default - Created from snapshots of existing EC2s - i.e. create an AMI of the EC2 instance that is fully configured running our application, we can then deploy that AMI to other instances, and they will also be fully configured. ## EC2 Instance Store - Real physical disks attached to EC2 - **_Ephemeral storage_** - Data is deleted on stop - Users responsibility to back-up ## EBS - Elastic Block Store - Similar to a Network Drive - can only be attached to **ONE** instance - can only exist in **ONE** AZ - Snapshot & Restore to move to different AZ - Is deleted by default when an EC2 is terminated - Snapshotting EBS can be done at any time for backup - It's possible to "archive" snapshots, this delays restoration by 24 hours but saves up to 75% on storage cost ### Volume Types - Can be used to boot: - GP2 / GP3 - General purpose SSD - IO1 / IO2 - Block Express High perf SSDs - Cannot be used to boot: - ST1 - High Throughput HDD - SC1 - Low cost HDD ### EBS MultiAttach - Only available to IO1 & IO2 - Allows for up to 16 instances to attach ### EBS Encryption - This is handled Transparently, EC2 does all the work itself - In Flight & At Rest encryption by default ## EFS - Elastic Filesystem - Managed service network filesystem - Can be mounted to **Many** EC2 instances across **Multiple AZ**s - Access is managed via [Security Groups](#security-groups) - Linux only - Scales storage automatically ## ELB - Elastic Load Balancer - Splits traffic between EC2 Instances - Exposes a single path into EC2 with DNS - Domain name -> ELB -> EC2s - Provides health checks for EC2 Instances - ELB can ping ports/paths on EC2 instances to know if the instance is working and can receive requests - ELB can be Public (External) or Private (Internal, only inter-VPC) - ELB can be the source on a security group - "Allow * From ELB" ```mermaid flowchart user[User] -- "Allow 80/443 from 0.0.0.0" --> ELB ELB -- "Allow 80/443 from ELB" --> EC2 user -- "Blocked" --> EC2 ``` ### CLB - Classic Load Balancer - HTTP/HTTPS/TCP/TLS - **_Deprecated_** ### ALB - Application Load Balancer - HTTP/HTTPS/Websocket - Can target Multiple EC2s **OR** Multiple containers _inside_ a single EC2 - Can route based on URL, Hostname, or Query Strings - Good for Docker and [ECS](#containers) - Possible to map to dynamic ports - Can target [Lambda](#lambda) - ALB gets given a fixed hostname URL - ALB forwards the clients real IP in the header `x-forwarded-for` ### NLB - Network Load Balancer - TCP/TLS/UDP - Can handle millions of requests per second with **_< 100ms Latency_** - One static IP per AZ, you can assign an Elastic IP - Good if your security requires a fixed IP address - Can be placed **_in front_** of an [ALB](#alb---application-load-balancer) - Health checks can be done using TCP/HTTP/HTTPs ### GWLB - Gateway Load Balancer - Layer 3 (IP Network Protocol) - For Fleets of 3rd party network virtual appliances - e.g. Filter traffic through a scalable fleet of firewalls - Uses the **GENEVE** protocol on port `6081` ```mermaid flowchart LR user -- "Request" --> GWLB GWLB <-- "First" --> Appliance[Firewalls ETC on EC2/ELB/ALB] GWLB -- "Second" --> EC2 EC2 --> user ``` ### Sticky Sessions - Possible to keep clients going to the same EC2 - "Sticky" - Works with CLB, ALB & NLB - Takes advantage of users cookies - "Application Based" - Generated by target EC2 (your application) - "Application" - Generated by load balancer - "Duration Based" - Generated by load balancer ### Cross Zone Load Balancing - With CZB - Each load balancer distributes traffic entirely across all registered instances in all AZ's. **_This is the default [ALB](#alb---application-load-balancer) behaviour_** - Without CZB - Requests are distributed in the instances at the node of the balancer. **_This is the default [NLB](#nlb---network-load-balancer) and [GWLB](#gwlb---gateway-load-balancer) behaviour_** ```mermaid flowchart User --> enabled[With CZB] User --> disabled[Without CZB] subgraph Zone 3 EC21[EC2] EC25[EC2] end subgraph Zone 2 EC22[EC2] EC26[EC2] end subgraph Zone 1 EC23[EC2] EC24[EC2] end enabled --> EC21 enabled --> EC22 disabled --> EC23 disabled --> EC24 ``` ### SSL/TLS on ELB - SNI only works with [ALB](#alb---application-load-balancer) & [NLB](#nlb---network-load-balancer) as well as [Cloudfront](#cloudfront) - This allows multiple certs on instances that apply based on domain ### Connection Draining - **_Specifically on Exam_** - Is called _"Registration Delay"_ on [ALB](#alb---application-load-balancer) and [NLB](#nlb---network-load-balancer) - If an EC2 enters the draining state, ELB will send new traffic elsewhere and allow the existing traffic to wrap up - default **5 Minutes** ## ASG - Auto Scaling Groups - Scales EC2 instances to meet demand - Can set Minimum required instances as well as the Maximum instances - Automatically replaces instances that fail health checks - Requires a _Launch Template_ which contains **all** required EC2 configuration - Multiple ways of handling scaling - Cloudwatch Alarms - "Simple/Step" scaling policy - i.e. Scale out by `x` steps if alarm hits 25%/50%/etc - "Target Tracking" scaling policy - i.e. Keep all instances under 40% CPU - "Scheduled" scaling policy - i.e. At 5pm double instance count - "Predictive" scaling policy - Based on historic load, automatically scale - Once a scaling action takes place there is a **5-Minute** delay on the next action # RDS - Relational Database Service - Managed service for SQL Databases - Postgres - MySQL - MariaDB - Oracle - Microsoft SQL Server - IBM DB2 - Aurora - Automatically handles Backups, Patches & Multi AZ deployment - Automatically scales - Auto-scales compute & storage ## Read Replicas - **_Specifically on Exam_** - Can create up to 15 *read only* replicas across multiple AZ or Regions - **Eventually consistent** - Replication is Async, data may be behind - Good for heavy tasks like reporting or analytics, avoid slowing main instance ## RDS Multi AZ - **Synchronous** replication - Single access point with automatic fail-over - _NOT_ for scaling, primarily for **disaster recovery** - Read Replicas can be configured as Multi-AZ for disaster recovery. - There is Zero Downtime on a Multi-AZ failover - **_This stat is specifically on Exam_** ## RDS Custom - Specifically for Oracle & Microsoft SQL Server - Gives access to the Operating System and underlying EC2 instances - More flexible lift & shift migrations ## RDS Backups - Automated every day during set backup window - Transaction logs are backed up every 5 minutes and can be restored at any time within 5 minute intervals - Automated backups have 1 to 35 days retention - Manual backups can be triggered by the user at any time - These are retained until deleted - It's cheaper to store snapshots than to keep RDS "Stopped". - **_This trick may specifically be on the Exam_** ## Aurora - Custom DB, Compatible with SQL & Postgres drivers - Claims **5x performance** over MySQL - Claims **3x performance** over Postgres - Underlying storage will automatically scale - Handles up to **15 replicas** with **< 10ms** latency - Instant fail-over - Creates 6 copies across 3 AZ - 4/6 are used for writing - 3/6 are used for reads - Data self-heals across instances in a P2P fashion - One instance takes writes (Master instance) - Offers "Reader endpoints" - URLs that target specifically Read instances & Read Replicas - **_This is specifically on Exam_** - If high read traffic Aurora will auto-scale the read endpoint by adding more instances - It's possible to add multiple Reader Endpoints which target different instances with different capabilities, i.e. Endpoint for Aurora instances running on High RAM EC2s ### Aurora Serverless - 100% Auto-scaling & hands off, Good for unpredictable workloads - Pay per second of uptime ### Global Aurora - Can set a primary region and up to 5 secondary regions - Data will replicate across all regions in **< 1 second**. - **_This stat is specifically on Exam_** - Up to 16 Read replicas can exist _per instance_ - Promoting a secondary region instance for disaster recover takes **< 1 minute** ### Aurora ML - Directly integrates with services like [Sagemaker](#sagemaker) or [Comprehend](#aws-comprehend) to provide ML predictions around your data - Allows for SQL requests that trigger ML services and returns standard data - i.e. Get me all purchases that may be fraudulent etc. ### Aurora Backups - Automatically backed up daily - Retention for 1 to 35 days - If instances are **not** encrypted, neither will the backup ### Aurora Clones - Ability to clone the database for Developer instances or testing - Faster than a snapshot - Copy-on-write ## RDS Security - At-Rest & In-Flight Encryption - Replicas copy encryption of primary instance - If primary is not encrypted, neither will the replicas - Possible to limit logins to IAM - Can limit logins with [Security Groups](#security-groups) ## RDS Proxy - Allows applications to pool and share database connections - Improves efficiency by reducing CPU & RAM usage on main instances and minimises open connections - Considered Serverless, Auto-scales & is Highly Available - Can reduce fail-over time by up to **66%** - Can **enforce** IAM logins & can limit access to only inside VPCs (Not Public!) - Good for [Lambda](#lambda) -> RDS connections since Lambda may leave open stale conections # Elasticache - Managed service for implementing Redis & Memcache - Redis is Multi-Az with automatic fail-over, Read Replicas & backups - Memcache is Multi-Node (sharding) with no backups. However, it's extremely fast ## Elasticache Security - IAM Authentication is available _only_ to Redis - Redis can also authenticate with a password/token to extra layer of encryption - Supports SSL for in-flight encryption - Memcache uses SASL-Based authentication # Route 53 - Private Zones can be created to specify DNS rules inside a VPC - Weighted Records can distribute traffic across different resources - i.e. send 10% of traffic to a new A/B testing site - Latency Based Records will direct traffic based on the ping. - 15 Health Checkers run in a Hosted Zone - _> 18%_ 2XX codes required to be considered "Healthy" # Elastic Beanstalk - Managed service that automatically deploys and manages EC2, ASG, ELB, RDS etc - Cloudformation under the hood[^1] # S3 - Simple Storage Service - Data is stored in "Buckets" ## S3 Replication - Versioning must be enabled to turn on replication - Multiple types: - CRR - Cross Region Replication - SRR - Same Region Replication - Async action, eventually consistent - Can be cross-account - By default, only new objects are replicated once enabled - You must enable replicating existing objects - S3 Analytics can suggest lifecycle rules for replicated objects ## Requester Pays - **_Specifically on Exam_** - By default, the bucket owner pays for storage and bandwidth - With Requester Pays enabled the requester pays for _bandwidth_ ## S3 Transfer Acceleration - Cross region transfer accelerator by allowing upload to an edge location which then uses AWS internal network to transfer to S3 ## S3 Byte Range Fetching - Receive less data & use less bandwidth by requesting a set byte range from a file. - You can also use this to split fetches into smaller chunks by running get requests in parallel. ## S3 Encryption Options - S3 Encrypts objects based on the headers when object is created - SSE: Enabled by default, uses AWS keys - SSE-KMS: Use a key stored in KMS - **_ON EXAM:_** KMS limits requests per second, this could cause issues with high throughput files. - SSE-C: Key is provided by the client - Client Side Encryption: Client encrypts file before uploading ## S3 MFA Delete - Can be enabled to require MFA when deleting versioned files or root files. - Only the root account can enable/disable this ## S3 Glacier Vault Lock - W.O.R.M (🪱) - Write once Read Many - Allows the creation of lock policies that prevent deletion or modification, it's possible to lock these policies to prevent changes - S3 Object lock can be per object - Per version of objects etc - Offers modes: - Compliance Mode: Default - Governance Mode: Allow _some_ IAM users to delete - Legal Hold: Completely lock object regardless of modes or permissions ## S3 Access Points - Attach Policies to prefixes - i.e. Allow the sales team to only access `/sales/*` in a bucket - Possible to use these to limit access to only VPCs ### S3 Object Lambda - Attach a lambda function to an access point - i.e. Run a file through a redaction function when it is requested. - If requested via the public endpoint it will be redacted since it runs through the lambda - If requested via another endpoint it will be the raw file # Cloudfront - CDN ## Cloudfront Pricing - Pricing is per region and can differ for edge locations, Amazon offers "Price classes" - All Regions - Price class 200 (Popular 200 regions) - Price class 100 (Popular 100 regions) # Global Accelerator - Uses any-cast IP to route to the closest Edge location. This can then route anywhere on the AWS network internally. Faster than hopping through the public net. # Amazon FSx - Allow 3rd party file systems to run on AWS - Windows Server FS - Supports SMB & NTFS - Active Directory support - _Can be mounted to linux_ - Supports Windows DFS - Lustre - For High Performance Computing - **L**inux Cl**uster** - Can directly read & write to an S3 bucket - Offers two types: - Persistent FS: Same AZ Replication - Scratch FS: No Replication, **6x faster**. Considered Temporary - NetApp ONTAP - For pulling over existing Netapp workloads - Supports NFS, SMB or iSCSI - Allows Point in Time recovery - OpenZFS - For pulling over existing ZFS workloads - Allows Point in Time recovery # Hybrid Cloud - Extend AWS Infrastructure to on premises & edge locations ## AWS Storage Gateway - Bridge storage from On-Prem to AWS - Gateway allows on-prem users to use NFS or SMB and then translates these requests to S3 actions - FSx Gateway: Possible to also target an FSx system instead of S3 - Volume Gateway: Allow iSCSI interfacing - Tape Gateway: Allow iSCSI interfacing for direct Tape backup - Storage Gateway Hardware Appliance is a physical server that can be run on prem as an interface for these services ## AWS Transfer Family - Managed Service for S3/EFS access using FTP/SFTP/FTPS - Pay per hour of connection + GBs - Can interface with Active Directory & LDAP ## AWS DataSync - **_Specifically on Exam_** - Move large data to and from AWS to on-prem - Can target S3, EFS of FSx - Metadata & POSIX file permissions **are preserved**. - **_Exam will likely ask about this_** - If there is not enough network capacity, it's possible to use DataSync with SnowCone. - **_Exam will likely ask about this_** - Can sync inter-service - S3 -> EFS for example - Available for S3, EFS, FSx # SQS - Simple Queue Service - Queue service for AWS - Unlimited Throughput with no max messages - 4 day message life - Messages must be < 256kb - **OUT OF ORDER** and sometimes messages may be processed twice ## SQS Message visibility - **_Specifically on Exam_** - When a message is accessed by another service, by default it is "hidden" for 30s to be processed. This time can be modified & the service can request more time from SQS. ## Long Poling - **_Specifically on Exam_** - Consumers of SQS messages can Long Poll a queue if there is nothing on it, they will instantly grab the message when it appears - Long poling can last 1 to 20 seconds ## FIFO queue - First in First out queue system, enforces order. ## SQS and ASG - SQS Can trigger Autoscaling Groups through Cloudwatch - `ApproximateNumberOfMessages` - SQS should be used to buffer Database writes - Combo with ASG to scale DB EC2's if large queue # SNS - Simple Notification Service - 1 to Many message sender (Pub/Sub) - Provider can send to a "Topic" and then Subscribers can read the messages - Can target Email, SMS, HTTP, SQS, Lambda or Kinesis ## SNS Fan Out Pattern - Client sends message to SNS & then multiple SQS queues queue messages. - i.e. Client sends order SNS, Shipping SQS & Inventory SQS then handle the action independently - Reduce errors if client "misses" a queue ## SNS FIFO - Same as [SQS FIFO](#fifo-queue) but for SNS ## SNS Filters - Subscribers can filter what messages they want to parse based on message content - i.e. Only read failure messages etc # Kinesis - **_Specifically on Exam_** - Service for easily processing real-time data from IOT, Logs, Metrics, etc. ## Kinesis Data Streams - Service for streaming big data - Uses "Shards" which are provisioned ahead of time and are your throughput limit - Producers push messages into shards - 1MB/s or 1000 messages/s per shard - Consumers pull messages from shards - Different types: - Shared - 2MB/s per shard across all consumers - Enhanced - 2MB/s per shard _per consumer_ - Retention is between 1 - 365 Days - Data can be replayed - Data is immutable & cannot be modified - Two capacity modes: - Provisioned - Manually set shard count and pay per shard per hour - 1MB/s in 2MB/s out - On-Demand - Automatically scales based on usage - Scales automatically based in 30 day usage - Pay per stream per hour ### Kinesis Data Stream security - Control access via IAM - In-Flight & At-Rest encryption - Can use VPC Endpoints to keep data off the public net ## Kinesis Data Firehose - Can pull data from many services and push it into S3, Redshift (S3 -> Redshift copy), Elasticsearch etc - Can modify data optionally with Lambda - Can optionally write all data (or only errors) to a separate S3 bucket - Only pay for data pushed through the service - Near real time, with no buffering - Can set up to a 900s buffer with the minimum size being 1MB ## Data Streams vs Firehose | Data Streams | Data Firehose | |-------------------------------------|--------------------------------------| | Streaming for ingest at scale | Load stream data into other services | | Write custom producer/consumer code | Fully Managed | | Real time (~200ms) | Near real time | | User manages sharding | Auto scaling | | 1 to 365 days data retention | No storage or data retention | | Supports data replay | No replay capability | ## Ordering Kinesis data - By using the same "Partition Key" you can ensure that data is processed in order - If you don't care about order, use a random key to distribute data evenly across shards - **_Exam question example:_** - If you have 100 trucks each sending GPS coordinates to a Kinesis stream how would you ensure that the data is processed in order? - Use the truck ID as the partition key, this will ensure that the data is processed in order per truck. Since each trucks data will always go to the same shard - data on shards are processes FIFO. ### Kinesis vs SQS - FIFO - SQS Standard has no ordering - SQS FIFO, without using GroupID will be first in first out - If you use GroupID, you can ensure that messages in groups are processed in order ## SQS vs SNS vs Kinesis | SQS | SNS | Kinesis | |---------------------------------------|-----------------------------------|--------------------------------------| | Consumers Pull data | Push data to many subscribers | Standard: 2MB per shard | | Delete after consume | Up to 12.5m subscribers | Enhanced: 2MB per shard per consumer | | Can have as many consumers as we want | Data lost if not delivered | Can replay data | | No need to provision | Up to 100k topics | Meant for Real Time big data | | Only Ordered if using FIFO | No need to provision | Shard level ordering | | Supports individual message delay | Integrates with SQS for "Fan out" | Retention for 1 - 365 days | | | FIFO capability with SQS FIFO | Provisioned _or_ on demand available | ## Kinesis Data Analytics - Sits between Kinesis streams & firehose, can apply SQL statements to data in real-time - Outputs back to a Kinesis stream or firehose - Can reference S3 Data - Pay for consumption rate ### Kinesis Data analytics for Apache Flink - Use Flink to process and analyse data - Run Flink on a managed cluster - Auto-scaling & backups provided - Can **_ONLY_** read data from Kinesis streams & [MSK](#amazon-managed-streaming-for-apache-kafka-msk) # Amazon MQ - SNS & SQS are "cloud native" and are AWS specific - Traditional on-prem solutions use open standards like MQTT, AMQP, STOMP etc - When migrating to the cloud, instead of porting everything to AWS services you can use Amazon MQ. - Does not scale well since it requires dedicated servers - Can run Multi-AZ for fail-over - Requires [EFS](#efs---elastic-filesystem) set up for shared storage - Offers its own version of Queues (SQS) and Topics (SNS) # Containers - Docker Containers - Can run on ECS, EKS, Fargate, EC2 ## Amazon ECR - Private Amazon docker repositories - Amazon also offer a public "Gallery" - Supports Vulnerability Scanning, versioning, tags, etc - Access to images controlled by IAM ```mermaid flowchart LR File[Docker File] -- Build --> Image[Docker Image] Image -- Run --> Container[Docker Container] Image -- Push --> Repo[ECR] Repo -- Pull --> Image subgraph Docker Repository Repo[ECR] Repo2[Docker Hub] end ``` ## Amazon ECS - When you launch containers on AWS it creates ECS tasks on an ECS cluster - If using the EC2 Launch type, an ECS cluster is a group of provisioned EC2 instances - Each EC2 instance must run the ECS Agent ```mermaid flowchart TD subgraph ECS - EC2 launch type New[New Request] subgraph EC2 Instance 1 ECSTask1[ECS Task] subgraph Docker One ECSAgent1[ECS Agent] end end subgraph EC2 Instance 2 ECSTask2[ECS Task] subgraph Docker Two ECSAgent2[ECS Agent] end end New -- Creates --> ECSTask1 New -- Creates --> ECSTask2 end ``` - Fargate is a Serverless launch type for ECS - No need to provision EC2 instances - AWS will automatically handle running tasks based on desired CPU & RAM - Sale by adding more tasks - not by adding more EC2 instances - **_EXAM Prefers fargate_** ### ECS IAM Roles - Two separate IAM Role types - ECS Instance Profiles (EC2 Launch types only) - Used by the _ECS Agent_ - Allows the agent to register with ECS & make ECS API calls - Allows sending logs to services like cloudwatch - Allows pulling images from ECR - ECS Task Roles - Used by the _Task itself_ - Allows the task to make API calls to other AWS services ```mermaid flowchart subgraph IAM - EC2 Instance Profile subgraph EC2 ECSAgent[ECS Agent] subgraph IAM - Task Role 1 Task1[Task 1] end subgraph IAM - Task Role 2 Task2[Task 2] end end end ECSAgent --> CloudWatch ECSAgent --> ECR ECSAgent --> ECS Task1 --> S3 Task2 --> DynamoDB ``` ### ECS Load Balancing - [ALB](#alb---application-load-balancer) can be run in front of ECS Clusters assuming the ECS tasks expose HTTP/s endpoints - [NWLB](#nlb---network-load-balancer) is only recommended for High Throughput, Low Latency applications or with AWS Private Link - [CLB](#clb---classic-load-balancer) is not recommended and does not work with Fargate ### ECS Data Volumes (EFS) - Can mount EFS instances to ECS Tasks - Tasks in teh same AZ can share data - Fargate + EC2 = Serverless[^2] ### ECS Autoscaling - Automatically increase/decrease the ECS Task count - ECS Auto-scales using AWS Application auto scaling - Average CPU, RAM, Request count per target, etc - Target Tracking: Scale based on Cloudwatch Metrics - Step Scaling: Scale based on Cloudwatch Alarms - Scheduled Scaling: Scale based on time of day - ECS Scaling != EC2 Scaling in SC2 Launch type, it just launches more ECS Tasks - Fargate Auto-scaling is handled by AWS, no need to set up ASG. It's much easier and the preferred method on the exam #### EC2 Launch Type Scaling - Accommodate ECS service scaling by adding more EC2 instances - Two Types - Auto-scaling group scaling - Scale ASG based on CPU utilisation - Add EC2 over time - ECS Cluster Capacity Provider - Automatically Scales - Paired with an [ASG](#asg---auto-scaling-groups) - Add EC2 instances whe missing CPU/RAM for required tasks - **_Preferred method_** ## Amazon EKS - Launch Kubernetes on services AWS - Fully Managed Service. - Alternate to ECS, Open Source - Can deploy EC2 or Fargate - Use cases are mainly if you already have Kubernetes experience or need to run Kubernetes - Cloud Agnostic, works on all cloud providers - EKS Nodes run _EKS Pods_ - These are just EKS Tasks but named differently - Node Types: - Managed Node Groups - Manages nodes (EC2) for you - Nodes in ASG managed by EKS - Supports on-demand and spot instances - Self-managed nodes - Nodes created by you then registered with EKS - Can use pre-made AMI or roll your own - Supports on-demand and spot instances ## EKS Data Volumes - Need to specify storage class manifest on EKS - Leverages a Container Storage Interface (**CSI**) driver - **_This acronym is on the exam!_** - Supports multiple filesystems: - EBS - EFS - Fargate _only_ supports EFS - FSx Lustre - FSX NETAPP ONTAP ## AWS App Runner - Managed service for deploying web apps on AWS - Provide either a container or source code as well as basic system requirements - AWS will automatically deploy ASG/ALB/ECS etc. and provide you with a public endpoint URL - Possible to also deploy inside a VPC # Lambda - Virtual Functions - serverless compute service - 15 minutes max runtime - Only runs when invoked - Pay per request + compute time - Easy to give access to more RAM - Up to 10GB of RAM - Increasing RAM also increases Compute and Networking power - Large language support - Node.js, Python, Java, C#, GoLand, .NET, Ruby - Custom Runtime API - Rust (Community managed) - etc... - Lambda Container Images - Must implement the Lambda Runtime API - ECS/Fargate preferred for arbirtrary containers - Lambda for quick functions - Unless specifically called for **_EXAM_** prefers [ECS](#amazon-ecs)/Fargate ## Lambda Limits - Per Region - Execution is maximum 15 minutes and 10GB of RAM - 4KB ENV space - 512MB - 10GB storage exists at `/tmp` - Default 10k lambdas concurrently but can be increased via support - Deployment limits - 50MB Compressed - 250MB Uncompressed - Lambda Layers can be used to include larger dependencies ## Lambda Snap Start - For Lambdas running Java 11 or higher - Increases startup speed at 10x no cost - Init phase in java takes a long time so AWS will pre-start the function, snapshot RAM and DISK and then on the next invoke it will restore the snapshot. ## Edge Functions - Attach to Cloudfront - Deployed Globally - Runs on the closest edge location to a user ### Cloudfront Functions - Lightweight JavaScript functions - High Scale low latency CDN customisations - Only on Viewer Request & Viewer Response - No Access to body ### Lambda@Edge - Viewer & Origin Request / Viewer & Origin Response access - Node.js & Python support - Can only be deployed in us-east-1 - 1000k requests/sec ### Cloudfront Functions vs Lambda@Edge | Cloudfront Functions | Lambda@Edge | |-------------------------|--------------------------------------| | Cache Key Normalisation | Server ms execution time available | | Header Manipulation | Adjustable CPU/Memory | | URL Rewriting | Allows 3rd party libraries | | | Network access for external services | | | Access to request body | ## Lambda in a VPC - By default, Lambda functions are deployed in the generic AWS VPC ("Not in a VPC") - Possible to attach a VPC to Lambda using an [ENI](#elastic-network-interface). - Lambda will create the ENI inside your selected subnet ### Lambda with RDS Proxy - Lambda may open many connections to RDS which may not close. - Highly recommend conneccting to RDS via RDS Proxy - RDS Proxy is **ONLY** accessible from inside a VPC ## Invoking Lambda from Aurora & RDS - Possible to invoke functions from _within_ the Database instance - Allows processing data from within the Database - Support by **_only_** RDS for Postgres & Aurora SQL - Ensure lambda has the correct IAM permissions and allows inbound/outbound traffic rules set - This is **external** from AWS events, AWS events do not include database data # DynamoDB - Fully manage multi-AZ NoSQL Database - Scales for millions of requests per second, infinite rows, 100TBs of Data - Extremely low cost - Two table types available - similar to [S3](#s3---simple-storage-service) storage classes - Standard - Infrequent access - Max 400KB object size - Values can be `null` ## DynamoDB capacity modes - Provisioned (Default) - Specific read/write capacity ahead of time - Pay per provisioned unit - Possible to auto-scale - On demand mode - Capacity autoscaling - Good for unpredictable and sudden workloads - Only pay for what you use ## DynamoDB accelerator (DAX) - In-memory cache for DynamoDB - Solves read congestion - Microsecond latency on cached data - requires no code changes ## DynamoDB Stream processing - Possible to run Lambdas whan a change is made to a DynamoDB table - Ordered - Two Options available - DynamoDB Streams - 24 Hour Data Retention - Limited consumers - Runs on Lambda - Kinesis Data Streams - Up to 1 year retention - High # of consumers - Connects to more AWS services ## Global tables - Two-way replication across regions ## DynamoDB TTL - Column contains a timestamp that when reached will delete the row - **_EXAM_** will likely ask about this, usually around storing user web sessions for 24 hours ## DynamoDB Backups - Continuous PITR - Optionally enabled for the last 35 days - PITR at any point within the window - Recovery generates a new table - On-demand - Manual backups, retained until deleted - Can be configured in the AWS Backup service - Recovery generates a new table # API Gateway - Proxy HTTP to lambda - Offers caching, throttling, logging, monitoring, API Keys, etc - Supports Websockets - Handles versioning APIs - Integrates with: - Lambda - Any AWS API - Any HTTP endpoint (external/internal) ## API Gateway endpoint types - Edge Optimise (Default) - Requests routed via Cloudfront Edge locations - Regional - Requests routed to the closest region - Manually combine with Cloudfront for more Cache control - Private - Only accessible from inside a VPC via an ENI ## API Gateway Security - Can authenticate users with IAM Roles, Cognito or a Custom Authorizer # Step Functions - Build serverless visual workflows to orchestrate AWS services - often Lambda - Possible to implement human intervention # Cognito - Gives users identities to interact with AWS apps - Two types - User Pools - Sign in functionality - Integrates with API Gateway & ALB - Identity Pools (Federated Identities) - Access to AWS services # Choosing the right Database **_EXAM_** will likely ask about all of these from a high level - **RDBMS (=SQL/OLTP)** - Great for joins - RDS - Aurora - **NOSQL** - DynamoDB - ElastiCache - Neptune (Graph) - DocumentDB (Mongo-esque) - Keyspaces (Apache Cassandra) - **Object Storage** - S3 - Glacier - **Data Warehousing (SQL Analytics/BI)** - Redshift (OLAP) - Athena - EMR - **Search** - OpenSearch (Elasticsearch) - **Graph** - Neptune - **Ledger** - Quantum Ledger DB (QLDB) - **Time Series** - Amazon TimeStream # Amazon KeySpaces - For Apache Cassandra - Analyse S3 data with SQL (Based on Presto) - $5 per TB parsed - **SERVERLESS** - **_EXAM_** will ask about improving performance - Use columnar data for cost savings - Less scanning - Recommend Parquet or ORC - AWS Glue can handle conversion - Compress data - Partition datasets in S3 - Virtual columns - `s3://data/year=2001/month=03/day=14` - Use larger files (> 128GB preferred) # Athena - Serverless SQL Query service - Query S3 data with SQL ## Athena Federated Query - Run queries on relational, non-relational dataset both on and off AWS - Uses data source connectors which run on Lambdas # Redshift - Based on Postgres (But not used for OLTP) - OLAP - 10x better performance than other data warehouses - SQL interfaces - BI tools like Quicksight integrate with Redshift - VS Athena: Faster Queries, joins, aggregations due to indexing - Leader Nodes are used for query planning & Compute nodes are used for performing the queries - These must be provisioned ahead of time - Snapshots to S3 - Only stores the changes - Loading data with large inserts is preffered - Via S3 or Firehose - Can use JDBC on EC2s ### Redshift Spectrum - Query S3 data without having to load data into Redshift - Must have a cluster already configured and running - Query is submitted to thousands of nodes for processing # Amazon Elastic Map Reduce (EMR) - Create Hadoop clusters (Big Data) - Hundreds of EC2 instances make up clusters - Bundled with Apache Spark, HBase, Presto, & Flink - Autoscaling & Integrated with Spot instances - Use cases include data processing, ML, web indexing and any big data operation - Node Types: - Master node manages cluster & health - must be long-running - Core node manages tasks & stores data - must be long-running - Task nodes (optional) only run tasks and can be spot-nodes - Purchasing options include - On Demand - Reserved instances - Spot Nodes - Can be "Long Running" or "Transient" - Transient nodes are only used for a single job and then terminated automatically # QuickSight - Serverless BI application that creates dashboards - Imports from most AWS services - **_EXAM_** - In memory computation is possible using a SPICE engine if data is imported directly into quicksight - Possible to set up column level security (CLS) # AWS Glue - Managed ETC Service - Serverless - Transform data between formats and between services - **Exam:** Convert CSV from an S3 Bucket into Parquet and store in another S3 bucket ## Glue Crawler - Crawls DB services and saves metadata that Glue can then use to perform jobs/actions - Used by Glue Jobs, Athena, Redshift, EMR - Glue job Bookmarks can prevent reprocessing data - **_EXAM:_** Will likely ask about this - Glue elastic views allow combining & replicating data across multiple sources using SQL - No Custom code is required, auto monitors sources, serverless - Glue Studio is a GUI form monitoring jobs, creating crawlers, etc # Lake Formation - Managed Service for making data lakes - Discover, cleanse, transform, ingest data from S3, RDS, Relational & NoSQL DBs - Can parse structured and unstructured data - Offers "Blueprints" for ingesting data from AWS services automatically - Offers fine-grain access control per column & row - **_Exam_** -> If a company is using athena and quicksight to analyze data, users should only access their assigned data. Lake formation can sit between data sources & athena to allow for extremely fine grain permissions # Amazon Managed Streaming for Apache Kafka (MSK) - Kafka is an alternative to Kinesis - Fully managed - Offers manual cluster provisioning - Serverless option available ## Kinesis vs Kafka | Kinesis Data Streams | MSK | |-----------------------------|-----------------------------------------| | 1MB Size limit on messages | 1mb default but up to 10MB | | Data streams with shards | Kafka topics use partitions | | Shard splitting and merging | Can only add patterns to topics | | TLS inflight & KMS at rest | Plaintext or TLS inflight & KMS at rest | # Amazon Rekognition - Find Objects, peoples, text in images using ML - Facial analysis & search - Can create a database of "Familiar faces" - User cases involve: - Labeling - Moderation - Text detecting - user verification - Pathing (Sports analysis etc.) ## Rekognition moderation - Examples in exam include - Detecting inappropriate content - Use on user generated content to creat a safe environment - Requires a minimum confidence threshold to trigger - Can trigger manual reviews using A2I - Helps comply with regulations # AWS Transcribe - Auto convert speech to text using ML - Can automatically redact PII - Can automatically detect languages # AWS Polly - Convert text to speech - Offers pronunciation lexicons to allow for user defined words & pronunciations - Can generate audio using SSML (Speech Synthesis Markup Language) - Breaths, pauses, etc # Translate - Language translation, allowing automated localisation - Works well with extremely large swaths of data # Lex + Connect - Lex is what drives Alexa - Automatic speech recognition & Natural Language Understanding - Good for chatbots - Connect is a virtual call centre that can integrate with Lex - Receive calls, route to agents, connect to CRMs, etc - No upfront payments required # AWS Comprehend - Natural language processing - Can use ML to find insights in text such as sentiment, entities, key phrases, language, etc ## Comprehend medical - Can detect medical data specifically - Converts plain text medical notes into structured data # Sagemaker - Fully managed service to build and train ML models - Will provision infrastructure for you - Labeling, training, tuning, deploying, monitoring all handled by Sagemaker # Forecast - Fully managed service for forecasting data - Pulls historical data from S3 & many third party services - Offers document search and allows indexing data and searching with natural language # Amazon Personalise - ML driven data personalisation's from data passed via S3 or an API - eg. recommended products, content, etc - Can output to API, SES etc # Cloudwatch - Logging across many services ## Cloudwatch Metrics - Cloudwatch provides metrics for every AWS service - **Metric** is a variable that can be monitored - They belong to namespaces (service etc.) - Dimensions are attributes _on_ a metric - Instance ID, Environment, etc - Possible to create custom metrics - Can stream metric data to Firehose or other 3rd parties ## Cloudwatch Agent - By default EC2 data does **_not_** go to CloudWatch - Running the Cloudwatch Agent can send logs to Cloudwatch - Can run on on-prem servers ### Cloudwatch unified agent - Collects additional **Metrics** - Upgraded version of the basic agent - Centralised config via SSM parameter store ## Cloudwatch Alarms - Alarms can be trigger by any **Metric** - Forward alarm state to SNS, Lambda, etc ## Cloudwatch container insights - Summarise metrics and logs from a container - ECS, EKS, EC2 Kubernetes, Fargate ## Cloudwatch Contributor Insights - Analyse logs & metrics to find anomalies - Helps identify bad hosts / heavy network users - E.g. VPC Flow Logs -> Cloudwatch Logs -> Contributor Insights -> Top 10 IPs by traffic # Event Bridge - Offers a serverless event bus - Scheduled Cron jobs - can trigger Lambda, ECS, Fargate, etc - Can respond/forward almost any AWS action/event # CloudTrail - Provides governance, compliance and auditing for your AWS Account and is enabled by default - Can view all events & API calls - Multiple Event types - Management events: Events that are performed on resources in your account - By Default are analysed and logged - Data events: S3 Bucket data events / Lambda execution events - Not analysed and logged by default due to high volume - Insight events: Detect unusual activity, will constantly analyse management events to create a baseline - Will then alert on deviations from the baseline - Events are stored for 90 days - Can export to s3 to analyse later using Athena - Possible to send events to Event Bridge to then trigger SNS or Lambda # AWS Config - Helps by recording compliance & configuration changes over time - Can answer basic questions like "Is there an S3 bucket open to the public?", "is SSH enabled on this EC2?" etc. - Can create custom rules to check for compliance like "Are any EC2s running that are not t2.micro". - Can trigger AWS events -> SNS etc - No free tier, costs $0.003 per config per region then $0.001 per rule evaluation per region - Cannot explicit change or fix any items that break rules but can trigger events to do so # CloudWatch vs Cloudtrail vs Config - Cloudwatch - Perform monitoring and create dashboards - Events & alerting - Log aggregation & analysis - Cloudtrail - Logs all API calls made to account - Can define trails for specific resources - Global service - Config - Record configuration changes - Evaluate resources against compliance rules - Can show a timeline of changes and compliance # AWS organizations - Global services - Allows managing multiple AWS accounts - "Main" account manages all rules - Consolidates billing across accounts - Share reserved instances & price benefits - API available to automate account creation - Allows central cloudtrail aggregation # IAM Conditions - Allows for finer grain permissions - I.e. Block * except from from IP range - Can limit based on tags, time, etc - Block user access to EC2 start/stop unless tagged with "Analysis" etc. # IAM Roles vs Resource based policies - Cross account access: - Attach a resource based policy to the resource (S3 Policy etc.) - **OR** use a role as a proxy ```mermaid flowchart subgraph Account A User1[User] User2[User] end subgraph Account B User1 -- Assume Role --> IAM User2 --> Policy IAM[IAM Role] --> S3 Policy[S3 Policy] --> S3 end ``` - When assuming a role you give up your current permissions and take on the permissions of the role - When using a resource based policy you keep your current permissions and the resource policy is added on top - SNS, SQS, Cloudwatch, Lambda etc use resource based policies - Other services use IAM roles # IAM Permission Boundaries - Can set the maximum permissions a user can have - I.e. Boundary only allows `S3:*` but policy allows `EC2:*`, final permission will _block_ the `EC2:*` permissions. # AWS IAM Identity Center - Successor ot AWS Single Sign-on - One log in for multiple AWS Accounts - Also works with Windows EC2, Salesforce ETC as long as they support SAML2.0 Sign-in - Can use a built-in identity provider or a 3rd party - Active Directory, Onelogin, Okta etc. - Org users assume IAM roles in accounts - Can do attribute based control with IAM - I.e. only allow users with a specific attribute to assume a role # AWS Active Directory services - Managed windows active directory - Exam requires an extremely high level understanding of different offerings: - Managed AD: New AD on AWS with an established trust between on-prem AD - AD Connector: Proxy to on-prem AD - Simple: AD Compatible managed directory on AWS - Cannot be joined to on-prem AD # AWS Control tower - Easily govern & secure multiple accounts - organizations under the hood - Automates account/env setup & policy management - Provides a dashboard for compliance & security - Allows configuring Guardrails to enforce policies - Preventive: Uses SCPs (e.g. Prevent `us-east-1` resource provisioning) - Detective: Uses AWS Config (e.g. Find untagged resources) ```mermaid flowchart TD subgraph Control Tower Guardrail[Detective Guard Rail] <--> Acc1[Account 1] Guardrail <-- Detect issue --> Acc2[Account 2] Guardrail <-- Detect issue --> Acc3[Account 3] end Guardrail -- Trigger --> SNS SNS -- Trigger --> Lambda Lambda -- Remediate --> Acc3 Lambda -- Remediate --> Acc2 ``` # KMS - Managed service for creating & managing encryption keys - Most AWS services integrate for encryption - Has API to allow for encryption in custom applications - Backing key is rotated yearly - Customer master key (CMK) can be used to encrypt data - Must be rotated manually - Two KMS key types - Symmetric: Single key for encryption & decryption - AWS Services use these - Never leaves KMS - Asymmetric: Public/Private key pair - Used for encrypting/decrypting as well as signing/verifying - Public key can leave KMS but private key never does - good for external non-aws encryption - Client --> AWS data ## Multi-region KMS - Can replicate keys across regions - Good for cross-region encryption, exact key id & contents replicated - Not recommend unless explicit required - DynamoDB Global tables & Global Aurora require replicated keys for data encryption/decryption across regions ## S3 Replication Encryption - KMS Encrypted objects are not replicated by default - You must specify a new key (or the same key) in the new region to re-encrypt the data - Data is decrypted & then re-encrypted on replication - You may get throttled by KMS if you have a large amount of data to re-encrypt, request a limit increase ## Sharing encrypted AMIs - If an AMI is encrypted in "Account 1" you must modify the AMI launch permission to specify that it can launch in "Account 2" - You must also share the KMS key with "Account 2" to allow decryption of the AMI snapshot - "Account 2" must also have the correct IAM permissions to use the KMS key # SSM Parameter Store - Secure storage for configuration data & secrets - Integrates with Cloudformation - Can store plaintext, encrypted or secure strings - "Parameter Policies" such as TTL incur an extra cost # Secrets Manager - Different to parameter store since it can **rotate keys** - Can connect a lambda function to generate new keys - Secrets are encrypted using KMS - Primarily used for RDS - Multi-region secrets available - Replication for disaster recovery # Amazon Certificate Manager (ACM) - Easily Provision TLS certificates - Does not work with EC2 instances - Works with Cloudfront, ALB, API Gateway, etc # Web Application Firewall (WAF) - Protect against layer 7 (HTTP) exploits - Deploys in-front of ALB, API Gateway, Cloudfront, Cognito etc - Allows defining ACLs for limits, headers, body etc - Regional except for Cloudfront which is at edge # AWS Shield - DDOS Protection - Offers an advanced service for $3,000/month which can mitigate more complex attacks and provides 24/7 live support # AWS Firewall Manager - Org wide firewall rules at a regional level - Can manage WAF, Shield, Security Groups, NACLs etc across multiple AWS accounts # Guard Duty - ML Threat detection through log parsing - One click enable with a 30-day free trial - Can ping AWS event bridge to trigger events # Amazon Inspector - Automatic security assessments - EC2 using SSM Agent - ECR Images inspected on push - Lambda inspected on deploy - Will look for common CVEs on any installed packages # Macie - Using ML on S3 data to detect PII - Can ping Event Bridge to trigger events # VPCs - Maximum CIDR per VPC is 5 - Soft limit of 5 VPCs per account ## Default VPC - New EC2 Instances are placed in the default VPC if there is no subnet specified - By default, they have Internet access and IPv4 DNS names ## Subnets - AWS Reserves the first 4 and last IP in each subnet. The following addresses cannot be used: - `*.*.*.0` - Network Address - `*.*.*.1` - VPC Router - `*.*.*.2` - Amazon Provided DNS - `*.*.*.3` - Future use - `*.*.*.255` - Network Broadcast Address **(Not Supported)** - By default, EC2 instances are _not_ assigned an IPv4 ## Internet Gateway - Allows EC2 instances to access the internet - Only one may exist in a VPC - Does not magically enable internet access. You must first attach it to a VPC and then update the route table. ## Bastion Hosts - Allows external users to connect to private EC2 Instances ```mermaid flowchart LR User -- SSH --> Bastion subgraph VPC subgraph Public Subnet Bastion[Bastion Host] end subgraph Private Subnet Private[Private EC2] end Bastion -- SSH --> Private end ``` ## NAT Instances - Outdated but potentially still in exam - Allows EC2 Instances in Private Subnet to access the internet - Nat instance exists in a public subnet and has an Elastic IP - Route tables need to be configured to route traffic from the private subnet to the NAT instance - AMI is available in the marketplace - Not recommended for high traffic - Not Highly available ## NAT Gateway - AWS Managed NAT, High Bandwidth, Highly Available - Pay per hour & per GB - NATGW is created in a specific AZ - Must have an Elastic IP - Cannot be used by EC2s in the same subnet - Highly resilient within on AZ but must create multiple across zones for high availability ## NACL & Security Groups - NACL intercepts all traffic into a subnet - NACL is stateless so all traffic is monitored - Security groups are stateful, so they allow through traffic that is a response to an outbound request - NACLs are evaluated before security groups - Default NACL allows all traffic - NACL rules are evaluated in order & are weighted - Recommend created a new NACL and associating it with the subnet instead of modifying the default ### Security Groups vs NACLs | Security Groups | NACL | |---------------------------|----------------------------| | Instance level | Subnet level | | Stateful (Return Allowed) | Stateless (Always checked) | | All rules evaluated | Weighted Rules | ## VPC Peering - Private connection between two VPCs via the AWS network - Must _not_ have overlapping CIDR blocks - Can be cross account & cross region - Route tables must be updated ## VPC Endpoints (AWS PrivateLink) - Allow instances in the VPC to access AWS services without needing to go over the internet - Types: - Gateway Endpoints - S3 & DynamoDB - Free - Interface Endpoints - Connect to services like SNS, SQS, etc - Pay per-hour & per GB ## VPC Flow Logs - Capture IP traffic logs across VPC, Subnet, ENI - Logs can be sent to S3, Cloudwatch, or Kinesis - Possible to capture traffic from AWS managed services like ELB, RDS, Redshift etc ## AWS Site to Site VPN - Connect a datacenter to VPC - Create a virtual private gateway (VPG) in your VPC - Set up a customer gateway in your datacenter - Configure the VPG with customer gateway IP - **OR** use the public IP of the NAT device it is behind - Route propagation must be enabled in your VPC route table for subnets to access the VPN ## AWS VPN Cloudhub - Allows access from _multiple_ on premises connections ## Direct Connect (DX) - Provides a dedicated private connection from on-prem to your VPC - Use-cases include: - Increase bandwidth at a lower cost - Consistent network performance - Direct Connect Gateway allows connecting multiple VPCs across regions into a single DX connection - Dedicated - 1, 10, 100 Gbps - Physical ethernet port for customer - Must be requested _from AWS_ - Hosted - 50Mbps to 1Gbps - Virtual interface - Can be requested _from a partner_ - Capacity can be increased on the fly - Lead times for DX connections can often be long (1+ month) - Residency is built in with multiple connections - High - 2 DX connections across 2 different locations - Maximum - 4 DX connections across 2 different locations ### Site to Site VPN as a backup - In the case that direct connect fails you can set up a site to site VPN as a backup connection, this is usually cheaper than having a second direct connect connection ## Transit Gateway - For transitive routing between VPCs & on-prem (Hub & Spoke model) - Cross region/account support & peer-able across regions - Only service which supports multicast ## VPC Traffic Mirroring - Allows capture & inspection of VPC traffic - Will not affect original traffic since it mirrors it to another ENI or NLB ## IPv6 for VPC - All IPv6 addresses are public & internet accessible ## Egress Only Internet Gateway - Similar to NAT but only for IPv6 - Allows IPv6 traffic to go out to the internet but not in ```mermaid flowchart LR subgraph VPC IG[Internet Gateway] EOIG[Egress Only Internet Gateway] subgraph Public Subnet NAT[Nat Gateway] end subgraph Private Subnet EC2 <-- IPv4 --> NAT end NAT <--> IG EC2[Private EC2] -- IPv6 --> EOIG end IG <--> Internet EOIG --> Internet ``` # Disaster Recovery - Traditional: On-premises -> On-premises fail-over - Hybrid: On-premises -> AWS fail-over - AWS: AWS -> AWS fail-over - Key-Words - RPO: Recovery Point Objective - Data Between RPO & Disaster is lost - RTO: Recovery Time Objective - Time between Disaster & RTO is downtime - Different strategies - Backup & Restore (High RPO) - Backing up snapshots to s3 or using a snowball, may take time to restore & may lose lots of data depending on backup periods - Pilot Light - Critical systems are always running & are replicated - Database & _core_ systems - Nothing extra - Warm Standby - Full system is running cloned but is at the smallest scale possible - Can be scaled up quickly - Multi-Site (Low RPO) - Full production clone running at all times # Database Migration Service (DMS) - Quickly migrate databases to AWS - Source remains operational during migration - Supports Homogenous & Heterogeneous migrations - Homogenous: Oracle -> Oracle - Heterogeneous: Oracle -> Aurora - Replication service runs from an EC2 instance # AWS Backup - Fully managed service for managing backups across almost all AWS services # Elastic Network Adapter (ENA) - Extremely low latency EC2 Networking - 100 GBPs - Used for high-performance computing # Elastic Fabric Adapter (EFA) - Improved ENA but only works on Linux - Great for inter-node tightly coupled workloads # AWS Parallel Cluster - Open source cluster management on AWS - Ability to auto-create VPCs, Subnets, etc - EFA can be enabled on the cluster as a while # Amazon Pinpoint - 2-way SMS, Email, Push notifications for marketing [^1]: This is an assumption [^2]: Technically yes, but it pains me so much to see containers & this be described as Serverless, some of these diagrams that are given as examples are terrifying. So much infrastructure and layers of complexity for what could be a lambda function... I get we're not up to it yet in the course but man.