What is DevOps Automation?
DevOps automation uses tools to automate various software development and operations aspects, such as testing, deployment, and infrastructure management. DevOps automation aims to increase efficiency, speed up development cycles, and improve collaboration between development and operations teams. Some standard tools used for DevOps automation include Jenkins, Ansible, and Docker. If you're interested in exploring a career in DevOps, we've answered the question of "What does a DevOps Engineer Do?" in another post.
What is an Example of DevOps Automation?
In this example, we'll explore three examples of DevOps automation and its impact on the industry.
DevOps Automation - Continuous Monitoring
In the world of microservices and complex infrastructure supporting software applications, it's essential to know the state of every component. As complexity increases, a failure in a sub-system can result in downtime of the entire operation. In other words, in the best-case scenario, users aren't able to access some of the information they need. In the worst case, your infrastructure is compromised, isn't able to process payments, and is costing your business a lot of money.
How does DevOps automation continuous monitoring help?
Before any application is deployed, the code can be instrumented to provide metrics that communicate the status of said application. For example, a payment processing service may be instrumented to communicate the number of transactions, their status (Ex: in process, completed, failed), and the error codes encountered by the users. Depending on the severity of an outage, the continuous monitoring solution may choose to simply display the data, notify an engineer via email, or shut down the affected payment processing service.
Needless to say, DevOps teams rely heavily on continuous monitoring as it is impossible to observe and analyze each service manually.
Let's look at another example.
You're running a business that sells merchandise via an e-commerce solution. In addition, a business lead indicates that your company is running a marketing campaign that will bring higher traffic than during the typical holiday season. Since your system is deployed on the cloud, you assume that the cloud provider's auto balancing / autoscaling features will handle the load. Nevertheless, your team decided that DevOps automation in the form of continuous monitoring needs to be implemented. In this case, you're asked to monitor a number of metrics - Inbound traffic to key landing pages, load on the payment processing applications, number and nature of the errors thrown by the process, and lastly, the number of finalized checkouts.
By monitoring the data above, you're looking at the vitality metrics of the services, and you're looking at key business metrics that help your team identify potential bottlenecks and address them as they come up.
Teams that haven't implemented such best practices will undoubtedly take longer to find the root cause of the failure and thus lose valuable customers during the holiday season.
DevOps Automation - Continuous Integration (CI) / Continuous Deployment (CD)
CI / CD is a methodology and a set of tools that streamline the process of updating an application with minimal impact on the user. In other words, software engineering teams can seamlessly push code to the production environment without interrupting current operations.
Why are CI / CD best practices important?
Let's look at two examples - A bank and an e-commerce store.
A typical bank infrastructure resides on a number of self-managed servers, and the entire operation is run as a monolith. There's no segmentation of microservices or smaller applications. A bank will have little to no benefit from CI / CD methodologies and tools. It's nearly impossible to upgrade any part of the application without a complete shutdown. You'll often receive a notice that bank services will be down for a short duration - the team is performing an upgrade of the system / architecture.
A modern e-commerce site will rely on the team's agility to deliver updates regularly. In some of our conversations, it was clear that a team that is capable of reacting to the smallest changes in the environment would capture value from the buyers. What does that mean exactly? Let's take a clothing vendor as an example. As fall sets in, based on past data, jackets are put on sale and are the entire focus of the website. The team constantly monitors the weather conditions and pushes the update to the winter season as soon as they notice an interest in winter clothing. They're able to push small updates at a much more frequent rate, thus capturing a portion of the market they'd otherwise lose. This translates to higher sales for the business.
How does DevOps Automate CI / CD?
Jenkins is one of the most popular tools for CI/CD automation. The general process that the DevOps team would use Jenkins for can be broken down into the following steps:
Step 1 - An individual contributor (software dev / engineer) commits a piece of source code into a repository (Ex: GitHub, GitLab).
Step 2 - Jenkins will automatically detect the change and initiate the build process. At this time, the code will be checked for errors, compiled, validated against automated tests if they've been defined, and created as an executable or package ready for deployment.
Step 3 - Depending on the configuration and infrastructure of the company, Jenkins will deploy the executable created in the previous step into a staging environment. This step is optional but is in place in most software organizations. Again, you'd want to perform manual tests and validate the applications before exposing them to end users.
Step 4 - A variety of tests are executed in the staging environment. Once the tests are completed, teams will typically schedule the code deployment of their new changes. Jenkins will orchestrate the deployment to the production environment.
Jenkins is a flexible tool that saves time and money. The core value is delivered through faster deployment and automation of tests along the way. However, it's much more flexible than that. Through a series of plugins and customizations, Jenkins can be tailored to specific applications and team needs. For example, one can easily add DevOps automation to the pipeline by monitoring every stage. A team lead can receive specific metrics and notifications based on how the code is deployed, the tests that pass / fail, and the team's productivity based on code deployments. Another team may choose to manually monitor the Jenkins interface that provides similar metrics out of the box.
Multiple other tools are used for CI / CD DevOps automation. Depending on the conversation, you'll include various software management tools which include GitHub, GitLab, Travis, CircleCI, AWS CodePipeline, Azure DevOps, Drone.io, and more. They all provide various features within the software development world; most software engineering teams rely on multiple tools to accomplish the process we've covered above.
DevOps Automation - Infrastructure as Code
You may have heard of virtual machines or virtualization - tools such as Docker, Kubernetes, and Terraform may come to mind. But what exactly are they used for? How do they fit into the overall application development and deployment process?
A virtual machine is an environment that creates a software replica of an operating system. The process is complex, but the basics of it are such that you can run multiple operating systems on the same machine. The number of virtual machines will vary based on the computer's specifications - memory, CPU, hard drive storage, etc.
How is Docker different from a virtual machine?
Docker is an environment that is built on top of the host kernel. The advantage is that Docker containers are very small since they re-utilize the non-varying code that supports an operating system. In practice, this translates to deploying numerous Docker containers, saving on space and memory of the underlying machine. In other words, you benefit from software segmentation and preserve as much space as possible within the container.
Where do Kubernetes and Terraform Come in?
You can easily create and deploy a Docker container locally. It's often used to develop software in a stable environment that is isolated from updates affecting the OS and the underlying software packages.
Kubernetes and Terraform are technologies that allow software engineers to automate the process of creating and deploying containerized (Dockerized) software. In other words, after the creation of Docker, Kubernetes and Terraform can be used to programmatically deploy the Docker containers onto servers - on-prem or in the cloud.
What’s the advantage of deploying infrastructure as code?
The first advantage of deploying through Kubernetes or Terraform is the ability to define infrastructure that is independent of cloud providers. In other words, the definition of a compute engine is irrelevant if you’re going to deploy on AWS, Google Cloud (GCP), or Azure. This allows a team of engineers to be agnostic of the provider while leveraging the benefits of each one. An example would be to deploy critical infrastructure across multiple providers while using specific services released earlier by one or the other provider.
The second advantage is the elimination of the traditional DevOps deployment process through the cloud provide. In earlier stages of cloud deployment, engineers needed to manually choose and deploy components that would fit together perfectly. Nowadays, they will specify what is needed on the software side and Kubernetes / Terraform will handle the selection of the underlying infrastructure components. This is important as it saves engineers time and standardizes what is deployed based on requirements rather than a manual process.
The last advantage we’ve seen is the ability to scale applications and microservices in ways cloud providers couldn’t facilitate before. AWS could always add an additional computer (EC2) instance for example. However, by deploying through code, it’s possible to allocate additional resources by managing how the software is executed on the same instance. It’s possible that the application itself is scaled rather than adding a new compute instance.
How are Software Engineering Teams Benefiting from DevOps Automation?
As covered in the examples above, there are many advantages to DevOps automation. In this section, let’s briefly discuss some of the advantages and their impact on the business operations.
Reliability
While it is possible to manually deploy code onto servers, mistakes are often made at this stage of the process. In the earlier stages of DevOps, numerous checklists were used to make sure that no errors were made and that the code was running as expected. DevOps Automation tools have streamlined the process, take care of the validation step, and remove the human touch from the equation, thus making it much must reliable.
Scalability
It is now possible to build applications and services that respond to specific needs of the users. In other words, DevOps tools monitor demand in real-time and adjust the process accordingly. Once again, it’s possible to eliminate the human touch from the equation and to use code to handle all scalability challenges. It’s important to note that although cloud providers had ways to scale their services (Ex: compute instances, load balancer), the process was typically confined to the service. In other words, it was always possible to add more EC2 instances as the load increases and to spread the load across many instance. With current DevOps best practices, the emphasis is on the entire solution and stack of applications rather than infrastructure elements.
Responsiveness
As discussed in the e-commerce example above, one of the advantages for the online business is speed. By having the ability to quickly deploy code changes, they’re able to react to the end-user and thus capture a significant share unlike their competitors who may take longer to react. DevOps automation is directly responsible for bringing this advantage to numerous companies and for streamlining this process.
Which Processes can be Automated through DevOps?
As you can see, DevOps touches many areas of the code development process. As you make the decision to automate certain components for your specific use case, it’s critical to understand various stages of DevOps and the ways they can be automated.
Strategy Phase
Before developers can build the application, the business team must discuss the requirements and prioritize certain features that are part of the release plan for this version. During this phase, the team is looking to figure out the answers to the following questions:
- What exactly are we building?
- Which systems will this addition impact?
- What are the deployment procedures and risks?
- What are the requirements / features of this release?
- What are the security parameters?
At this stage, the teams will have a conversation about the tools they will use to accomplish the task at hand. At this time, we recommend looking at the following project management tools - Jira, GitHub, GitLab, Asana, etc.
Technical Breakdown Phase
At this point, the project has been set by the business team and the dev team can establish an architecture, artifacts, microservices, and deliverables of each team member. In these conversations, the primary goal for the team is to delegate deliverables and to ensure that every engineer is clear of their task and none of their code / deliverables overlap.
The tools we've seen most used at this stage are GitHub, GitLab, Subversion, Cloudforce, Bitbucket, etc.
Build / Development Phase
At this time, it's clear what every developer needs to accomplish. Led by a senior developer, every member of the team creates services to meet their own specifications. The team leverages streamlined code repository functions to merge their code into the main build. DevOps automation can be used at this stage to automate the validation process - unit tests are executed once code is merged to ensure compliance with the project and other services that have been identified in the previous phases.
Test Phase
We've briefly discussed testing in the example we covered above. This step aims to ensure that the code we've built is sound for deployment. At this time, manual code reviews are performed by developers as the initial check. Typically, senior engineers manually review the deliverables and skim through the code build.
Some automation can be implemented at this stage. For example, running tests to ensure cybersecurity compliance with a registration portal is standard practice. Before code is deployed in production, these tests are aimed at picking up any apparent vulnerabilities - Ex: the user can inject code via text / password fields, the user can overload the functions via extensive input variables, etc. In addition, it's possible to create custom tests that apply in unique situations. For example, a HIPPA-compliant application used in the healthcare industry may need to comply with strict data policies. Therefore, the data entered into this application must be tested differently than an application for a social media service.
Release / Deployment Phase
Once the code has been validated, it must be deployed onto a server - on-prem or in the cloud. At this stage, various tools are used to facilitate code deployment. We've covered the process extensively above.
Standard Operation & Debugging Phase
At this point, the code is running on the server. The DevOps teams are using tools to help them establish a baseline for operation and a means to debug / troubleshoot as issues arise.
Based on our conversations and industry knowledge, there's a distinction between engineering and operations. DevOps typically sits in the middle. So what's the key difference between the two? Engineering aims to deliver a product that drives the business. Operations are the "customer" of the product and are responsible for ensuring ongoing support of the application, microservices, etc. In other words, once the code is deployed onto the server, the operations department takes ownership. However, engineering may get involved in certain circumstances - 1. In smaller teams, the line between teams is blurred; an engineer may be responsible for the coding, deploying, and maintaining the application. 2. In instances where the operations team isn't able to find the root cause and solve the issue. In other words, operational teams are the first line of defense in case of an outage. If they can't figure it out, DevOps is involved.
Conclusion
DevOps Automation is a key ingredient in every modern software development department. It provides a set of methodologies and tools that allow engineers to build, validate, and deploy software automatically. Although numerous solutions are available on the market, they address different stages of development - Strategy, Technical Breakdown, Build, Test, Release, and Standard Operation. The need for DevOps automation arises from a constant push on the business side to make code changes in a reliable and fast manner. By doing so, a business can react to the changes in user / environment behaviors and thus gain an advantage over the competitors.
.jpg)
