Assignment 3 – Scripting for System Automation COMP9053
James Coughlan
R00169682
1) Identify key Infrastructure Automation tools.
Configuration Provisioning
Chef
Cobbler
Vagrant
Docker
Terraform
Ansible
Configuration management
Salt
Ansible
Puppet
CFEngine
Cobbler
Chef
Orchestration
Cloudify
Juju
Ansible
Cobbler
Heat
Continuous Integration
Jenkins
TeamCity
TravisCI
GitLabCI
2) For each category, pick 3 of the tools listed & give a high-level description/definition of each tool. The description should consist of a single paragraph
Configuration Provisioning
Terraform is an infrastructure provisioning tool designed with an emphasis on reusability and shareability. It allows for representation of physical hardware, virtual machines, containers, DNS providers and more. Through the use of configuration files written in GO, Terraform interprets, by generating an ‘execution plan’, what is required to provide the required infrastructure to the user.
Cobbler is a Linux installation server that allows for rapid setup of network installation environments. It automates many associated Linux tasks so you do not have to switch between many various commands and applications when deploying new systems and changing existing ones. Cobbler can help with provisioning, managing DNS and DHCP.
Chef provisioning is a collection of resources that enable the creation of machines and machine infrastructures using the chef-client. It has a plugin model that allows bootstrap operations to be done against any infrastructure, such as VirtualBox, DigitalOcean, Amazon EC2, LXC, bare metal, and more.
Configuration management
Puppet is an open-source, model driven management system, where the user defines required system resources in a Puppet ‘manifest’ file, written in a non-standard declarative language. These manifest files will often be hosted on a server and provided to the client as configuration instructions.
Ansible covers many aspects of Infrastructure as Code. It utilises YAML to construct ‘Ansible Playbooks’, allowing the user to manage configurations of web servers. It favours using ssh over the client/server model of Puppet.
CFEngine is an open-source confutation management system that has been in development since 1993 and has spawned several iterations since. Developed in the C programming language, it promotes lower memory usage, fewer dependencies and faster speeds than similar configuration management tools such as Puppet.
Orchestration
Juju uses ‘charms’ to to deploy, manage and scale software and interconnected services across one or more Ubuntu servers and cloud platforms.
Cloudify is an open-source orchestration tool that promotes it’s usability across all technologies. For architectures that run across multiple clouds, Cloudify can perform network automation at scale using TOSCA declarative blueprint files.
Heat is an orchestration tool for managing applications deployed on the cloud. Heat can orchestrate functionalities such as ports, routers, instances, and private networks.
Continuous Integration
Jenkins is an open-source tool to automate deployment of software through continuous building and testing. Through plugins it can be expanded to integrate with third party applications to handle all steps in the Continuous Integration and Continuous Deployment pipeline.
TravisCI builds are configured using a YML file which contains the build tasks that will be executed upon running the build.
GitLabCI is a hosted Continuous Integration solution fully integrated with GitLab that process builds written in the Go language. These builds can run on Windows, Linux, OSX, FreeBSD, and Docker
3) For the 3 chosen tools in a given category, highlight the key similarities/difference between them & from this identify environments/users that might be suitable for each tool
Configuration Provisioning
Configuration Management
Orchestration
Continuous Integration
4) For each category pick one of the tools & give a detailed description of it.
Chef
Chef is an automation tool that provides a way to define infrastructure as code, using a Ruby, domain-specific language for writing system configurations. In Chef, Nodes are dynamically updated with the configurations in the server. There is therefore no need to execute a command on the Chef server to push the configuration on the nodes, nodes will automatically update themselves with the configurations present in the Server.
https://www.edureka.co/blog/what-is-chef/
Chef supports several platforms such as AIX, RHEL/CentOS, FreeBSD, OS X, Solaris, Microsoft Windows and Ubuntu as well as client platforms including Arch Linux, Debian and Fedora. Chef can also be integrated with cloud-based platforms such as Internap, Amazon EC2, Google Cloud Platform, OpenStack, SoftLayer, Microsoft Azure and Rackspace to automatically provision and configure new machines.
https://www.edureka.co/blog/what-is-chef/
As outlined in the above diagram, there are three major Chef components:
• Workstation
• Server
• Nodes
The Workstation is the location from which all of Chef configurations are managed. This machine holds all the configuration data that can later be pushed to the central Chef Server. These configurations are tested in the workstation before pushing it into the Chef Server. A workstation consists of a command-line tool called Knife (replaced in the 12.0 release of Chef, with a config.rb file), that is used to interact with the Chef Server. There can be multiple Workstations that together manage the central Chef Server
The Chef Server acts as a hub for configuration data. The Chef Server stores Cookbooks, the policies that are applied to Nodes, and metadata that describes each registered Node that is being managed by the Chef-Client.
Nodes use the Chef-Client to ask the Chef Server for configuration details, such as Recipes, Templates, and file distributions. The Chef-Client then does as much of the configuration work as possible on the Nodes themselves (and not on the Chef Server). Each Node has a Chef Client software installed, which will pull down the configuration from the central Chef Server that are applicable to that Node. This scalable approach distributes the configuration effort throughout the organization.
Nodes can be a cloud based virtual server or a physical server in your own data center, that is managed using central Chef Server. The main component that needs to be present on the Node is an agent that will establish communication with the central Chef Server. This is called Chef Client.
Chef Client performs the following functions:
• It is responsible for interacting with the central Chef Server.
• It manages the initial registration of the Node to the central Chef Server.
• It pulls down Cookbooks, and applies them on the Node, to configure it.
• Periodic polling of the central Chef Server to fetch new configuration items, if any
Sample config.rb file
Sample metadata.rb file
https://www.digitalocean.com/community/tutorials/how-to-create-simple-chef-cookbooks-to-manage-infrastructure-on-ubuntu
Puppet
Puppet configuration consists of a language, client-server processes and the Resource Abstraction Layer. The language allows the description of a server configuration with an abstraction of the resources that an administrator already thinks in: users, groups, packages, files, cron, mount and services.
The relationships between the resources are also specified. For example, a service depends on a configuration file, and that file depends on a package being installed. The relationships provide order as the policy is applied and allow Puppet to restart dependent services when their configurations change.
The resources can also be composed into logical collections. To reuse the previous example, a package, a configuration file and a service can be grouped together. The group can then be reused and treated as a single logical entity in other Puppet code. The client-server setup provides a secure mechanism for transporting the specific configurations from the central description to the individual hosts over HTTP with SSL authentication and encryption — the same SSL used to secure online banking and e-commerce. Each host only receives its specifically compiled configuration to apply.
https://www.edureka.co/blog/puppet-tutorial/
The following functions are performed in the above image:
• The Puppet Agent sends the Facts to the Puppet Master. Facts are basically key/value data pair that represents some aspect of Slave state, such as its IP address, up-time, operating system, or whether it’s a virtual machine. I will explain Facts in detail later in the blog.
• Puppet Master uses the facts to compile a Catalog that defines how the Slave should be configured. Catalog is a document that describes the desired state for each resource that Puppet Master manages on a Slave. I will explain catalogs and resources in detail later.
• Puppet Slave reports back to Master indicating that Configuration is complete, which is visible in the Puppet dashboard.
• Puppet Slave asks for Puppet Master certificate.
• After receiving Puppet Master certificate, Master requests for Slave certificate.
• Once Master has signed the Slave certificate, Slave requests for configuration/data.
• Finally, Puppet Master will send the configuration to Puppet Slave.
Every Slave has got its configuration details in Puppet Master, written in the native Puppet language. These details are written in the language which Puppet can understand and are termed as Manifests. They are composed of Puppet code and their filenames use the .pp extension. These are basically Puppet programs.
Puppet manifest: https://www.digitalocean.com/community/tutorials/getting-started-with-puppet-code-manifests-and-modules
Ansible
Ansible is a minimalist IT automation tool that has a low learning curve, using YAML for it;s provisioning scripts. It has a great number of built-in modules that can be used to abstract tasks such as installing packages and working with templates. Modules are Ansible’s way of abstracting certain system management or configuration tasks. In many ways, this is where the real power in Ansible lies. By abstracting commands and state into modules, Ansible is able to make system management idempotent. This is an important concept that makes configuration management tools like Ansible much more powerful and safe than something like a typical shell script
Playbooks allow you to organize your configuration and management tasks in simple, human-readable files. Each playbook contains a list of tasks (‘plays’ in Ansible parlance) and are defined in a YAML file. Playbooks can be combined with other playbooks and organized into Roles which allow you to define sophisticated infrastructures and then easily provision and manage them
Playbook.yml: https://www.digitalocean.com/community/tutorials/configuration-management-101-writing-ansible-playbooks
Jenkins
Jenkins coordinates a wide variety of activities, such as checking out and building new versions of code, running tests, and deploying software
Jobs are the runnable tasks that are controlled and monitored by Jenkins. Examples of jobs include compiling source code, running tests, provisioning a test environment, deploying, archiving, posting build jobs such as reporting, and executing arbitrary scripts. Jenkins jobs can be scheduled to run continuously, often on a nightly basis. The setup of a Jenkins job is straightforward and can integrate with third party extensions in order to deploy to external applications. Webhooks can integrate with triggers to initiate builds when certain criteria is met, for example when a branch updates on Github. Jenkins monitors the execution of the steps and allows to stop the process, if one of the steps fails. Jenkins can also send out notifications in case of a build success or failure.
Jenkins stores all the settings, logs and build artefacts in its home directory. Users can view previous builds, console output and workspaces.
Jobs are defined in their config.xml file.
Sample config.xml file: https://github.com/cyberswat/jenkins-example-configs/blob/master/jobs/devops%20test%20boulder/config.xml