Migrate EC2 Instances from one account to another (same region)

There are a lot of cases, which I encounter every now and then and do not find solutions easily and I have to refer to books that are available with me or on the Internet. One such scenario that I faced recently was that I had a development instance (in my aws account) and I had to create an instance for client and install/deploy all required packages/app etc (in his aws account). At that point I was wondering if there is some way to simply transfer the instance to other account (I was using my personal AWS account for development purposes and had to create/configure instance in client’s account).  Being lazy to the core I just didn't wanted to create an instance and do the rework. 

Lot of times I was regretting not asking client to provide infra for development purposes as well. A lot was going through my mind like “Why I didn’t used docker”, “ Why I didn’t note down all configuration steps”, “Should I write Ansible script to make it easier”. While I was thinking all this and searching web at the same time I came across several answers (on stackoverflow) where it was mentioned how snapshots can be used to create new instances etc.

At that point I decided to explore AWS Console a bit more deeply and decided to play around with it while referring to documentation as well and very soon I was able to migrate instance across accounts in same region.

I am sharing steps below in hope of helping others :

Pre-requisite :

-       You should be able to access the support section of destination account (more on this below).

Step 1: Log into source account and go to EC2.

Step 2: Select the instance that you want to migrate.

Step 3: Create an AMI and give it a name or description that its easier to understand. I named my AMI as devmigrate20may17.

Step 4: Click on Create Image and wait for snapshot to appear in Snapshots section in the same region.

Step 5: Select the snapshot and create image from it.

Step 6: Keep all the values as default and give a descriptive name.

Step 7: Once the Image is created, go to AMI section and select the newly created image.  In lower half note “Permissions” tab, click on that.

Step 8: You will notice that AMI’s are by default private. (Most will get excited at this point). Yes , we can edit and make it public.

Step 9: At this point we need account number of destination account, you can find account number in support section in top right corner copy that account number and paste it in the box. There are two options here to make AMI public i.e it will be visible to everybody and to make it visible to the account number we specify. We will go with the second option for obvious reasons.

Step 10: Log into destination account Go to Images -> AMI and select “Private Images” from drop down filter. Here you will see the AMI and from this AMI you can simply launch an instance.

I hope this post help people like me who needed to migrate instance from one account to another.

Do share your feedback and if you think its helpful, please share it with others as well.

Preparation/Unofficial Guide for AWS Solutions Architect Exam - Associate

DAY 1

Objective : Study the Exam Guide and create a working study plan.

I am experienced AWS user (administrator) and I plan to get a certificate in first attempt but as confusing as it was getting as to from where and how to start. I finally decided to write a study plan and my day to day progress and share it with others as well with the help of this blog. 

One thing that I realised is that AWS is a plethora of services and we may or may not be using all of them in our day to day operations hence we are left out of that practical knowledge. I started reading Exam Guide and also referred to AWS In Action but somewhere I was not satisfied with the way I was progressing.

So, rather than getting lost in documentation and practicing services to get hold of them I decided its better to decode the Exam Guide and understand what AWS really want me to prepare for.

Here is the download link for the Exam Guide (PDF).

As per exam guide one of the expectations from examinee is that they should be well aware of the architecture principals of the AWS. So where do I find this information? Answer is obvious.

I should read the AWS White Papers focusing on architecture best practices. Problem is all white papers are scattered all over the documentation and its really hard to get to them. As we need to prepare a study plan we should be able to gather all the study material before we start. To save time I have included links to most of the white papers at the end of the article.

Second expectation mentioned is that as solution architect I should be able to guide a team of developers or system administrators to build a project, which follows the best practices of AWS.

How Guide has bifurcated hands-on experience and theory?

The section named AWS Knowledge clearly mentions that examinee’s should have hand-on experience on compute, networking, storage and database services.

This means that examinees should be familiar with the following services:

-                EC2 (compute).

-                VPC (networking)

-                S3, Glacier, EBS, Storage Gateway (storage)

-                Aurora, RDS, DynamoDB, ElastiCache, AWS Database Migration Services (Database)

Lets segregate these and other components/services in buckets of highest % in exam coverage.

60% Bucket :

-                Planning and Designing Cloud services

-                Monitoring and Logging.

-                AWS Architecture

-                Price/Cost Evaluation, Budgeting

-                Trade-offs evaluation i.e Cost vs High Availability

-                Hybrid IT Architectures (Direct Connect, Storage Gateway, VPC, Directory Services etc)

-                Elasticity and Scalability (Auto-Scaling, SQS, ELB, CloudFront etc)

20% Bucket :

-                AWS Shared responsibility Model

-                Compliances

-                IAM

-                VPC

-                KMS

-                CloudTrail

-                Firewall, VPN

-                Design Patterns

-                DoS Mitigation

-                Security Groups, ACL’s

-                CloudWatch

-                Trusted Advisor

-                CloudWatch Logs

-                Disaster Recovery

o   Recovery Time Objective

o   Recovery Point Objective

o   Amazon Elastic Block Store

-                AWS Import/Export

-                Storage Gateway

-                Route53

-                Validation of data recovery methods

10% Bucket :

-                BeanStalk

-                CloudFormation

-                OpsWorks

-                VPC, IAM

-                AMI

-                How to create/operate hybrid IT Architectures

-                High Availablity by installing instances acrsoss regions.

-                IAM best practices

-                 

10% Bucket:

-                Troubleshooting of all above mentioned services

After understanding the study guide now I am in a position to prepare my study plan. I plan to take 60% bucket head-on and cover theory and practical as much as possible. Moving on to 20% and 10% buckets. As, I went through the content I found VPC, IAM and Logs to be appearing repeatedly in most of the buckets so let make these as priority and not to forget that VPC is an integral part of EC2 so I’ll prioritize that too.

I think now the study path is pretty clear and I know what to take up first and where should I focus more.

Links to all whitepapers and important documentations :

Build highly-scalable and reliable web or mobile-web applications

PDF  AWS_ac_ra_web_01.pdf

Build highly reliable systems that serve massive amounts of content and media

PDF AWS_ac_ra_media_02.pdf

Build auto-scalable batch processing systems like video processing pipelines

PDF  AWS_ac_ra_batch_03.pdf

Build systems that quickly failover to new instances in an event of failure

PDF  AWS_ac_ra_ftha_04.pdf

Large Scale Processing and Huge Data sets

Build high-performance computing systems that involve Big Data

PDF   AWS_ac_ra_largescale_05.pdf

Ad Serving

Build highly-scalable online ad serving solutions

PDF  AWS_ac_ra_adserving_06.pdf

Disaster Recovery for Local Applications

Build cost-effective Disaster Recovery solutions for on-premises applications

PDF   AWS_ac_ra_disasterrecovery_07.pdf

File Synchronization

Build simple file synchronization service

PDF   AWS_ac_ra_filesync_08.pdf

Media Sharing

Cloud-powered Media Sharing Framework

PDF    AWS_ac_ra_mediasharing_09.pdf

Online Games

Build powerful online games 

PDF   AWS_ac_ra_games_10.pdf

Log Analysis

Analyze massive volumes of log data in the cloud

PDF    AWS_ac_ra_loganalysis_11.pdf

Financial Services Grid Computing

Build highly scalable and elastic grids for the Financial Services Sector

PDF   AWS_ac_ra_financialgrid_12.pdf

E-Commerce Website Part 1: Web Frontend

Build elastic Web Front-ends for an e-Commerce website 

PDF   AWS_ac_ra_ecommerce_webfrontend_14.pdf

E-Commerce Website Part 2: Checkout Pipeline

Build highly scalable checkout pipeline for an e-Commerce website

PDF    AWS_ac_ra_ecommerce_checkout_13.pdf

E-Commerce Website Part 3: Marketing and Recommendations

Build highly scalable recommendation engine for an e-Commerce website

PDF   AWS_ac_ra_ecommerce_marketing_15.pdf

Time Series Processing

Build elastic systems that process time series data

PDF   AWS_ac_ra_timeseriesprocessing_16.pdf

WHITEPAPERS

AWS Well-Architected Framework

Download Whitepaper (PDF)   AWS_Well-Architected_Framework.pdf

AWS Cloud Architecture Best Practices Whitepaper

Download Whitepaper (PDF)   AWS_Cloud_Best_Practices.pdf

Building Fault-Tolerant Applications on AWS Whitepaper

Download Whitepaper (PDF)    aws-building-fault-tolerant-applications.pdf

Using AWS for Disaster Recovery Whitepaper

Download Whitepaper (PDF)    aws-disaster-recovery.pdf

Operational Checklists for AWS

Download Whitepaper (PDF)     aws-operational-checklists.pdf

Web Hosting Best Practices Whitepaper

Download Whitepaper (PDF)    aws-web-hosting-best-practices.pdf

Leveraging Different Storage Options in the AWS Cloud Whitepaper

Download Whitepaper (PDF)    aws-storage-options.pdf

AWS Security Best Practices Whitepaper

Download Whitepaper (PDF)    aws-security-best-practices.pdf

NIST WEBSECURITY FRAMEWORK

NIST_Cybersecurity_Framework_CSF.pdf

AWS Well-Architected

 AWS Well-Architected Framework whitepaper    

Download the Security Pillar whitepaper »

Download the Performance Efficiency whitepaper »

Download the Cost Optimization whitepaper »

Instaling Ansible on Linux and Mac OS X

Ansible today is the new buzzword in world of DevOps because it allows sysadmins to easily and quickly transform their manual tasks into scripts for automated environment. The best thing about Ansible is that it is easy to learn, is agent less and doesn't require a lot of configuration to make the tool run. 

Apart from being a tool for sysadmins Ansible can be easily learn and put into practice by any developer with the same ease because of its clear syntax, structure and very wide support from open source community. As per Ansible website there are around 2200 contributors out there creating plugins for Ansible, so most probably there's already a solution to the problem or task. 

In this article we will look into installation of Ansible on different OS's and Docker.

Lets begin:

Latest release via YUM:

$ yum install ansible

Latest release via APT(ubuntu)

$ sudo apt-get install software-properties-common
$ sudo apt-add-repository ppa:ansible/ansible
$ sudo apt-get update
$ sudo apt-get install ansible

Latest release via APT (debian)

$ deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main
$ apt-get install ansible

Latest release on MAC OS X
$ sudo easy_install pip
$ pip install ansible

Docker Cheatsheet - Handy Docker commands for everyday use.

Docker is a container management system that helps easily manage Linux Containers. It lets us create images in virtual environments on our laptop or development environments. The actions that we do on containers and its corresponding behaviour remains same when we run them in production environment.


Docker Commands :

$docker --help : It gives full list of all Docker commands.

$docker <COMMAND> --help : To get additional help pertaining for the given command.

$docker version : gives the information about docker installation.

Search Docker Images :

$docker search <search-term> for e.g docker search nginx

Pulling a Docker Image:

$docker pull tutum/ubuntu

List Docker Images :

$docker images

       

Remove Docker Image:

$docker rmi ubuntu:trusty

Run a Docker Image:

$docker run -i -t image_name:tag /bin/bash 

-i gives an interactive shell

-t will assign a pseudo-tty

Run Docker Image as Daemon

<pre>

$docker run -d image_name:tag for e.g. $docker run -d ubuntu:trusty

</pre>
View the running container/s:
$docker ps

Expose the Docker ports in Daemon mode
$docker run -d -p 8080:80 ubuntu:trusty (port 8080 of container is mapped to port 80 of host)

Check the logs of Docker Container
$docker logs container_id or name

Kill a Docker Container:
$docker kill container_id or name

Stop a Docker Container:
$docker stop container_id or name

Get the stats of Docker Container:
$docker stats container_name

$docker top container_name

Remove the container
$docker rm container_name

BUILDING A DOCKER IMAGE
$docker build --help 

$docker build -f path_to_Dockerfile -t REPOSITORY:TAG
REPOSITORY mostly username is used for Docker Hub and <TAG> is the container name.

Building with multiple config files

Create a new directory to hold config files and CD into that directory before executing the build command.

$docker build -t REPOSITORY:TAG


DATA VOLUMES

Mounting a single volume
$docker run -it -v /user/home ubuntu /bin/bash

Mounting multiple volumes 
$docker run -it -v /user/home -v /tmp ubuntu /bin/bash

Mounting local directory inside the Docker Container
$docker run -it -v /user/home:/data ubuntu /bin/bash

Mounting in Read Only mode
$docker run -it -v /user/home:/data:ro ubuntu /bin/bash

We can verify the mounts by issuing :
$docker inspect container_id

DATA VOLUMES CONTAINER