NetDevOps CI/CD with Ansible, Jenkins and Cisco VIRL (Part 2 - Components & Tools)

Welcome back. Now that we have an understanding of what NetDevOps is, let's dive into the components and tools that we will use to build our NetDevOps CI/CD pipeline.

Let’s go …

Components

Our CI/CD pipeline will be built using a number of components. Below shows an overview:

Figure 1 - Components.

Let’s look at each one in more detail:

Github

Github is a code hosting platform for version control and collaboration, upon which we will host our Infrastructure as Code Network as Code. Git provides the concept of branches. Feature/Dev branches are created from your master branch. Changes are then made within this feature/dev branch and then merged back into the master branch, as shown below.

Figure 2 - Git Branching. ^[1]

Jenkins

Jenkins is described by its creators as,

an open-source automation server which enables developers around the world to reliably build, test, and deploy their software.^[2]

However, I like to think of Jenkins as Cronjob on steroids! For example, Jenkins allows to perform actions, run scripts etc based on events that are triggered within your central source control, such as Github.

Jenkins has the concept of a Job and a Build - shown below:

• Job - a configuration for running autonomous tasks.
• Build - each run of the job is called a build.

Furthermore, Jenkins provides various Job types (also known as Project types), the main ones being Freestyle and Pipeline (shown below). Our example will be based upon Freestyle.

• Freestyle - The legacy way of configuring multiple tasks in a sequential manner.
• Pipeline - Allows you to configure your pipeline using Groovy-based scripting.

Figure 3 - Jenkin Project Types.

Therefore, to summarize, we will configure multiple Freestyle Projects, one for each of our pipelines.

Ansible

Ansible is an agentless configuration management tool. YAML based playbooks containing tasks are configured to describe your intent, these are then run against inventory files in order to define which hosts are configured. Further details around Ansible can be found here.

We will use Ansible to configure SNMPv3 on our hosts and also use it to validate that the configuration has been applied and is working correctly. Validation will consist of polling our devices using the new SNMPv3 user and checking the response.

Our network configuration will be defined via the group_var file all.yaml. This file (as shown below) includes the various variables that we will consume within our Ansible playbooks.

---
snmp_users:
  - user: snmp-user1
    auth: sha
    encrypt: yes
    privacy: PRIV-PASSWORD1
    pwd: AUTH-PASSWORD1
  - user: snmp-user2
    auth: sha
    encrypt: yes
    privacy: PRIV-PASSWORD2
    pwd: AUTH-PASSWORD2

VIRL

VIRL is a network virtualization simulator by Cisco. Similar to the likes of GNS3 and EVE-NG. VIRL will be used to build a test network (shown below), so that we can validate our changes prior to deploying them to production.

Figure 4 - VIRL Test Network

VIRL is typically managed via UI using via an application called VM Maestro. Therefore, we will use virlutils, this python package will interact with the VIRL API and provide us with a VIRL CLI (shown below) that we can consume within our pipelines.

root@desktop:~/cicd-demo-1# virl --help
Usage: virl [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  console   console for node
  down      stop a virl simulation
  flavors   Manage VIRL Flavors Attributes
  generate  generate inv file for various tools
  id        gets sim id for local environment
  logs      Retrieves log information for the provided simulation
  ls        lists running simulations in the current project
  nodes     get nodes for sim_name
  pull      pull topology.virl from repo
  save      save simulation to local virl file
  search    lists virl topologies available via github
  ssh       ssh to a node
  start     start a node
  stop      stop a node
  swagger   manage local swagger ui server
  telnet    telnet to a node
  up        start a virl simulation
  use       use virl simulation launched elsewhere
  uwm       opens UWM for the sim
  version   version information
  viz       opens live visualization for the sim

Tools

Make

Make is a utility used by developers when compiling/building software binaries. automatically determines which pieces of a large program need to be recompiled, and issues commands to recompile them.^[3] Simplifying the logic and improving build times.

Make uses a special file known as Makefile. A Makefile contains targets, dependencies and commands. Let's look at a Makefile example:

test: test.o anotherTest.o
    gcc -Wall test.c anotherTest.c -o test -I.

test.o: test.c
    gcc -Wall -c test.c -I.

anotherTest.o: anothertest.c
    gcc -Wall -c anotherTest.c -I.

Based on the example above this Makefile allows us to compile our code by running make test this would run the test: target. This has 2 dependencies test.o and anotherTest.o. Therefore, Make would run these targets first, so that the required dependencies are created before running test.

You may be asking, Why on earth are you telling me all of this? Well, outside of compiling software Make also provides a great method for running sets of shell commands.

We will make (excuse the pun) use of Make to create a Makefile that contains the various steps required within our pipelines. Though, in essence you could write a shell script to do the same thing, I find Make quicker and it provides greater simplicity.

Let's review our Makefile, the actual file can be seen here.

root@desktop:~/cicd-demo-1# make
  add-venv                  Install virtualenv, create virtualenv, install requirements
  configure-prod-network    Configure prod network
  configure-test-network    Configure test network
  distroy-test-network      Distroy test network in VIRL
  format                    Remove end of line spaces from yaml files
  install-deps              Install pip
  lint                      Perform linting against ansible yaml files
  remove-venv               Remove virtualenv
  run-tests-prod-network    Run tests against prod network
  run-tests-test-network    Run tests against test network
  start-test-network        Start test network via VIRL

If we look inside the Makefile (shown below) you will see that we:

• First export a number variables that our commands will require.
• Define each of the different steps that we will required to later build our pipelines.
• Use the rule .PHONY to specify that the target is not a file.
• @ is used to suppress the command output.
• $ is prepended to any variables. Using a single $ Make will try and match it against its own set of variables.

.DEFAULT_GOAL := help

export VIRL_HOST=172.29.236.133
export VIRL_USERNAME=guest
export VIRL_PASSWORD=guest
export ANSIBLE_HOST_KEY_CHECKING=False
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
export PATH=$$PATH:./venv/bin:/usr/local/bin:/usr/bin:/bin

.PHONY: help
help:
	@grep '^[a-zA-Z]' $(MAKEFILE_LIST) | \
	sort | \
	awk -F ':.*?## ' 'NF==2 {printf "\033[36m  %-25s\033[0m %s\n", $$1, $$2}'
	
.PHONY: install-deps
install-deps: ## Install pip
	apt-get update -y
	apt-get install python-pip -y

.PHONY: add-venv
add-venv: ## Install virtualenv, create virtualenv, install requirements
	pip install virtualenv
	virtualenv venv
	. ./venv/bin/activate
	@echo installing requirements.txt ...
	@venv/bin/pip install -q -r ./requirements.txt

.PHONY: format
format: ## Remove end of line spaces from yaml files
	find ./ \( -name *.yaml -o -name *.yml \) | xargs sed -i  "s/\s *$$//g"

...remainder omitted...

Ansible-Lint

lint, or a linter, is a tool that analyzes source code to flag programming errors, bugs, stylistic errors, and suspicious constructs.^[1]

For our CI/CD pipeline we will use ansible-lint to validate our yaml files are syntactically correct. As shown below:

find ./ansible/ \( -name *.yaml -o -name *.yml \) -exec ansible-lint {} +

Note the use of +. This ensures that if any invocation returns a non-zero value exit status, then the find utility will return a non-zero exit status.^[4]

One of the main errors you will see when running ansible-lint is around end of line spaces. Therefore, our pipelines will also perform a format so that the yaml files have no end of line spacing. Like so:

find ./ \( -name *.yaml -o -name *.yml \) | xargs sed -i  "s/\s *$$//g"

Coming Up

In part 3 of this series we will dive into each of the pipelines, their configuration, and then a final walk-through. See you there ...

References