Packer – How packer has simplified Cloud provisioning

By Arun Bagul | August 27, 2021 | Comments 0 Comment

What is Packer –

Packer (HashiCorp) is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer can use tools like Chef or Puppet to install software onto the image.

A machine image is a single static unit that contains a per-configured operating system and installed software which is used to quickly create new running machines. Machine image formats change for each platform. Some examples include AMIs for EC2, VMDK/VMX files for VMware, OVF exports for VirtualBox, etc.

Packer and Continuous Delivery (CD) –

Packer is lightweight, portable, and command-line driven. This makes it the perfect tool to put in the middle of your continuous delivery pipeline. As part of this pipeline, the newly created images can then be launched and tested, verifying the infrastructure changes work. If the tests pass, you can be confident that the image will work when deployed. This brings a new level of stability and testability to infrastructure changes.

Packer template –

A Packer template is a configuration file that defines the image you want to build and how to build it. Packer templates use the Hashicorp Configuration Language (HCL). Packer configuration has different block in configuration like mentioned below…

Packer Block – The packer {} block contains Packer settings, including specifying a required Packer version.

Source Block – Thesource {} block configures a specific builder plugin, which is then invoked by a build block. Source blocks use builders and communicators to define what kind of virtualization to use, how to launch the image you want to provision, and how to connect to it.

Build block – The build { } block defines what Packer should do with the Virtual Machine or Docker container after it launches.

Builders are responsible for creating machines and generating images from them for various platforms. For example, there are separate builders for EC2, VMware, VirtualBox, etc.

Please find the supported builder list here – https://www.packer.io/docs/builders

- virtualbox-iso

- vmware-iso

- hyperv-iso

- docker

Provisioner block – The provisioner { } block defines what Packer should do with the image build or Docker container after it launches.

Provisioners use builtin and third-party software to install and configure the machine image after booting. Provisioners prepare the system for use, so common use cases for provisioners include:

installing packages
patching the kernel
creating users
downloading application code
Post-Processors – Post-processors run after the image is built by the builder and provisioned by the provisioner(s). Post-processors are optional, and they can be used to upload artifacts, re-package, or more.
Communicators – are the mechanism Packer uses to upload files, execute scripts, etc. with the machine being created.

Communicators are configured within the builder section. Packer currently supports three kinds of communicators:

none – No communicator will be used. If this is set, most provisioners also can’t be used.
ssh – An SSH connection will be established to the machine. This is usually the default.
winrm – A WinRM connection will be established.

Packer Workflow –

The Packer workflow takes a defined source (usually an operating system installation media in ISO format or an existing machine), creates a machine on the builder, executes the Provisioner’s defined tasks and generates an output. Doing this on Hyper-V, VMWare workstation and Docker may look like what is shown below.

How to use Packer –

Initialize your Packer configuration and check and validate packer template…

$ packer init .

$ packer fmt .

$ packer validate .

Build the image with the packer build command. Packer will print output like what is shown below.

$ packer build new-os-image-template.hcl

Reference –

Packer software – https://www.packer.io

Thank you,

Arun Bagul

(Personal Blog and This is based on my knowledge)

IaC ~ Terraform and Pulumi

By Arun Bagul | August 6, 2021 | Comments 0 Comment

Terraform and Pulumi

Before reading this blog I would recommend to read this blog – https://www.indiangnu.org/2017/top-5-infrastructure-as-code-iac-software/

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.

Key feature of Terraform and Pulumi are..

IaC (Infrastructure as Code) for better management of
Infrastructure and state management
Supports all Cloud providers like Azure, GCP, AWS and virtualization infrastructure like VMware and HyperV
Change Automation
Help you to adapt IaaS, SaaS and PaaS

How to user Terraform –

Terraform has 3 steps . First is “terraform init” initialization step in which Terraform will check and download required providers and plugins. Second is “terraform plan” in this step Terraform will create plan of change in your infrastructure and user has choice to verify and confirm then plan and then start third step ie “terraform apply” this in step Terraform will create resources in providers infrastructure like azure, aws or gcp. Terraform will also save all objects created in Terraform stafe “.tfstate” file and we can use same configuration to add or delete or update resource in same workflow….

To Debug Terraform error we can define debug and log fails mentioned below-

export TF_LOG_PROVIDER=TRACE

export TF_LOG_CORE=TRACE

export TF_LOG_PATH=logs.txt

terraform init

terraform plan

terraform apply

For formatting and configuration validation use below command –

terraform validate

terraform fmt

terraform refresh

HashiCorp Configuration Language (HCL) –

In Terraform we can define infrastructure required to deploy as code using HCL.

HCL is a toolkit for creating structured configuration languages that are both human- and machine-friendly, for use with command-line tools. Although intended to be generally useful, it is primarily targeted towards devops tools, servers, etc. HCL has both a native syntax, intended to be pleasant to read and write for humans, and a JSON-based variant that is easier for machines to generate and parse. HCL syntax is designed to be easily read and written by humans, and allows declarative logic to permit its use in more complex applications

What is Pulumi ?

Pulumi is a modern infrastructure as code platform that allows you to use familiar programming languages and tools to build, deploy, and manage cloud infrastructure.

Pulumi vs. Terraform

Here is a summary of the key differences between Pulumi and Terraform:

Component	Pulumi	Terraform
Language Support	Python, TypeScript, JavaScript, Go, C#, F#	Hashicorp Configuration Language (HCL)
State Management	Managed through Pulumi Service by default, self-managed options available	Self-managed by default, managed SaaS offering available
Provider Support	Native cloud providers with 100% same-day resource coverage plus Terraform-based providers for additional coverage	Support across multiple IaaS, SaaS, and PaaS providers
OSS License	Apache License 2.0	Mozilla Public License 2.0

Ref –

Pulumi – https://www.pulumi.com

Terraform – https://www.terraform.io

HCL – https://github.com/hashicorp/hcl

Terraform Error – https://learn.hashicorp.com/tutorials/terraform/troubleshooting-workflow

Thank you,

Arun Bagul

(Personal Blog and This is based on my knowledge)

Top 5 Infrastructure as Code (IaC) software

By Arun Bagul | November 22, 2017 | Comments 0 Comment

Introduction –

World is moving toward hybrid/multi-Cloud solutions and it is important for every Enterprise/Organizations to use different Cloud providers effectively!. Multi-Cloud strategy will help companies to save cost, make infrastructure highly available and businness continuity plan (disaster recovery) etc.

Infrastructure as Code (IaC) is a type of IT infrastructure that operations teams can automatically manage and provision through code, rather than using a manual process. Infrastructure as Code is sometimes referred to as programmable infrastructure. IaC is useful as it supports and make provisioning, deployment and maintenance of It infrastructure easy and simple in multi-Cloud scenario!

Why IaC?

* Manage infrastructure via source control, thus providing a detailed audit trail for changes.
* Apply testing to infrastructure in the form of unit testing, functional testing, and integration testing.
* Automate Your Deployment and Recovery Processes
* Rollback With the Same Tested Processes
* Don’t Repair, Redeploy
* Focus on Mean Time to Recovery
* Use Testing Tools to Verify Your Infrastructure and Hook Your Tests Into Your Monitoring System
* Documentation, since the code itself will document the state of the machine. This is particularly powerful because it means, for the first time, that infrastructure documentation is always up to date
* Enable collaboration around infrastructure configuration and provisioning, most notably between dev and ops.

Tops 5 Infrastructure as code (IaC) Software –

1) Terraform (https://www.terraform.io)
Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. Terraform provides a flexible abstraction of resources and providers. Terraform is used to create, manage, and manipulate infrastructure resources. Providers generally are an IaaS (e.g. AWS, Google Cloud, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services (e.g. Terraform Enterprise, DNSimple, CloudFlare).
NOTE – Vagrant is another tool from HashiCorp. Refer article for more information – https://www.vagrantup.com/intro/vs/terraform.html

2) Spinnaker (https://www.spinnaker.io)
Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence. Deploy across multiple cloud providers including AWS EC2, Kubernetes, Google Compute Engine, Google Kubernetes Engine, Google App Engine, Microsoft Azure, and Openstack.

3) AWS CloudFormation (https://aws.amazon.com/cloudformation)
AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion. You can use AWS CloudFormation’s sample templates or create your own templates to describe the AWS resources, and any associated dependencies or runtime parameters, required to run your application.

4) Google’s Cloud Deployment Manager (https://cloud.google.com/deployment-manager)
Google Cloud Deployment Manager allows you to specify all the resources needed for your application in a declarative format using yaml. You can also use Python or Jinja2 templates to parameterize the configuration and allow reuse of common deployment paradigms such as a load balanced, auto-scaled instance group. Treat your configuration as code and perform repeatable deployments.

5) Azure Automation and Azure Resource Manager(ARM)
Microsoft Azure Automation provides a way for users to automate the manual, long-running, error-prone, and frequently repeated tasks that are commonly performed in a cloud and enterprise environment. It saves time and increases the reliability of regular administrative tasks and even schedules them to be automatically performed at regular intervals. You can automate processes using runbooks or automate configuration management using Desired State Configuration. ARM Templates provides an easy way to create and manage one or more Azure resources consistently and repeatedly in an orderly and predictable manner in a resource group.

* Docker Compose (https://docs.docker.com/compose/overview)
NOTE- Docker Compose is mainly for Container technology and is different from above tools.

* Orchestrate containers with docker-compose
The powerful concept of microservices is gradually changing the industry. Large monolithic services are slowly giving way to swarms of small and autonomous microservices that work together. The process is accompanied by another market trend: containerization. Together, they help us build systems of unprecedented resilience. Containerization changes not only the architecture of services, but also the structure of environments used to create them. Now, when software is distributed in containers, developers have full freedom to decide what applications they need.

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. Compose preserves all volumes used by your services. Compose caches the configuration used to create a container. When you restart a service that has not changed, Compose re-uses the existing containers. Re-using containers means that you can make changes to your environment very quickly.

* IaC Tools and DevOps –

When we speak of the DevOps and continuous delivery/integration (CI/CD) toolchain, we’re referring to a superset of tools—many with overlapping capabilities—for helping organizations achieve faster and safer deployment velocity. This encompasses a broad range of solutions: provisioning tools, orchestration tools, testing frameworks, configuration management (CM) and automation platforms, and more. Please refer DevOps – Comparison of different Configuration Management Software for Comparisons between CM. Here we’ll compare different orchestration and management tools for provisioning infrastructures: Terraform and Spinnaker/CloudFormation.

CloudFormation is specific to AWS cloud resources, while Terraform/Spinnaker supports all cloud vendors.
Terraform allows you to define and manage your infrastructure, but Spinnaker allows you to manage your infrastructure from the perspective of code releases and deployment workflows
Infrastructure Lifecycle Management is easy using visualizations such as Terraform graph give developers and operators any easy way to comprehend dependent ordering
Docker Compose mainly for containers technology like Docker (https://www.docker.com)
Azure Automation is for Azure Cloud using Power-shell scripting

Thank you,
Arun Bagul

Cloud: Monitoring as a Service (MaaS)

By Arun Bagul | August 18, 2015 | Comments 0 Comment

Introduction-

Cloud computing has been evolving into different technology areas such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS). These technology areas are finding increasing adoption in the marketplace.

A recent report published by International Data Corporation (IDC) has indicated that the Cloud Computing market is expected to cross $70 billion in 2015. Ref- https://www.idc.com/getdoc.jsp?containerId=prUS25797415

The economic drivers for adoption of Cloud Technologies have been well understood in the market.

Lower Investments: Cloud offerings typically have very low upfront costs. All SaaS product offerings are charged on a pay per use monthly subscription basis, which implies lower cost of ownership.
No Infrastructure Costs: Cloud service offerings use cloud based infrastructures such as Amazon or Rackspace or a private cloud. Hence the customer does not have to bear the cost of the infrastructure setup with cloud services.
Outsourced Technology Expertise: With cloud offerings, customers get a ready to use product that can be immediately consumed by the end user. Customer does not need to invest in an in-house IT team having that particular technology expertise.
Simplified Management: With offerings in the cloud, the service provider deals with on-going management, maintenance and upgrades of technology. The customer can focus on his core business needs.

So Why I’m telling this story to you? As infrastructure engineer I believe that this is going to change how we do things in Traditional vs Could Infrastructure. Let us take example of “Monitoring”. Here in this blog, I’m going to talk about monitoring of Infrastructure and applications in Traditional vs Cloud. Adoption of “Cloud computing” is going to change they way we do monitoring of our servers and applications.

Monitoring as a Service (MaaS) in the Cloud! –

What does Monitoring-as-a-Service (MaaS) mean?
Monitoring-as-a-service (MaaS) is one of many cloud delivery models under anything as a service (XaaS). It is a framework that facilitates the deployment of monitoring functionalities for various other services and applications within the cloud. The most common application for MaaS is online state monitoring, which continuously tracks certain states of applications, networks, systems, instances or any element that may be deployable within the cloud. MaaS offerings consist of multiple tools and applications meant to monitor a certain aspect of an application, server, system or any other IT component. There is a need for proper data collection, especially of the performance and real-time statistics of IT components, in order to make proper and informed management possible.

IT infrastructure monitoring should an essential part of the IT Management Policy for an organization that is reliant on IT infrastructure. Proactive monitoring provides business continuity, quicker disaster recovery and easier capacity planning for all mission critical applications.

Monitoring as a Service (MaaS) in the Cloud is a concept that combines the benefits of cloud computing technology and traditional on-premise IT infrastructure monitoring solutions. MaaS is a new delivery model that is suited for organizations looking to adopt a monitoring framework quickly with minimal investments.

Traditional On-Premise Monitoring Framework-

On premise monitoring is the traditional deployment model for monitoring private networks (internal IT infrastructure). This has been a very effective model over the years and works well for organization that can afford to implement this monitoring framework. On-premise monitoring involves purchase of software tools and investing in monitoring infrastructure and skilled IT personnel.

On-Premise monitoring provides the following benefits:

In-House Monitoring Infrastructure:Customers can own the in-house infrastructure for monitoring. This implies more control over the infrastructure with regards to upgrades, maintenance and management
Higher Levels of Security:Since the monitoring infrastructure is located in-house, customer gets better security where the monitoring tool does not need to cross firewall domains and connect over the open internet.
Inherent Connectivity to Internal Assets:Monitoring infrastructure is already a part of the internal network (LAN and MPLS). Hence connecting to all the infrastructure assets is easy.
Real Time Monitoring Data:On-premise monitoring provides real-time data where alerts are generated and shown to the customer immediately. The monitoring dashboard and email servers are all within the customer premise and hence there are no delays.
Customization and Extensions:On-premise monitoring solutions can be heavily customized to meet the exact needs of a specific customer environment. This could be in the form of monitoring of custom applications or personalized dashboards and escalation matrices.
The Adoption of Cloud and Software as a Service Delivery Models

Monitoring as a Service Offering –

The monitoring as a service (MaaS) offering provides a monitoring solution based on a monitoring infrastructure in the cloud. The MaaS vendor invests in the monitoring framework including the hardware, monitoring software and specialized IT personnel on behalf of the customer. The customer just needs to pay for the service he wants to use – on a subscription model similar to any SaaS product offering.

Benefits of Monitoring as a Service (MaaS)

The following are the benefits of a monitoring as a service (MaaS) product:

    Ready to Use Monitoring Tool Login: The vendor takes care of setting up the hardware infrastructure, monitoring tool, configuration and alert settings on behalf of the customer. The customer gets a ready to use login to the monitoring dashboard that is accessible using an internet browser. A mobile client is also available for the MaaS dashboard for IT administrators.
    Inherently Available 24x7x365: Since MaaS is deployed in the cloud, the monitoring dashboard itself is available 24x7x365 that can be accessed anytime from anywhere. There are no downtimes associated with the monitoring tool.
Easy Integration with Business Processes: MaaS can generate alert based on specific business conditions. MaaS also supports multiple levels of escalation so that different user groups can get different levels of alerts.
Cloud Aware and Cloud Ready: Since MaaS is already in the cloud, MaaS works well with other cloud based products such as PaaS and SaaS. MaaS can monitor Amazon and Rackspace cloud infrastructure. MaaS can monitor any private cloud deployments that a customer might have.
Zero Maintenance Overheads: As a MaaS, customer, you don’t need to invest in a network operations centre. Neither do you need to invest an in-house team of qualified IT engineers to run the monitoring desk since the MaaS vendor is doing that on behalf of the customer.
   Price Sensitive Customers: For small and medium enterprises, MaaS provides cost effective pay per use pricing model. Customers don’t need to make any heavy investments neither in capital expenditures (capex) nor in operating expenditures (opex).

Monitoring as a service (MaaS) is an attractive choice for the following scenarios:

Cloud Based SaaS and PaaS offering Add-On: MaaS provides a better technology fit for monitoring cloud based SaaS and PaaS offerings. MaaS can be provided as an add-on product offering along with SaaS and PaaS.
Distributed Infrastructure Assets: In scenarios where the IT infrastructure assets are distributed across different locations and branch offices, MaaS is a good option since the monitoring infrastructure is centralized in the cloud and can easily monitor all distributed infrastructure assets.
Mixture of Cloud and On-Premise Infrastructure: MaaS is already in the cloud. Hence in deployments where customer has a mix of on-premise and cloud infrastructure, MaaS provides good monitoring options for the hybrid environment.
Multitenant Monitoring Requirements: For vendors offering multi-tenant functionality on their hosted services, MaaS provides a strong backend framework for monitoring the multi-tenant services and their availability.

Thank you,

Arun Bagul