Whether you like it or not, the networking industry is stuck with (and will be for many years to come), vendors and devices with no API, inconsistent interfaces, differing configuration and runtime CLI formats. This means that as much as you hate screen-scraping and regex, it's here to stay. In this post I'll dig into some parsing tools in the current landscape that will help you accomplish your network automation goals with minimum amounts of regex, which in turn will lead to minimum 🤬.
One point to make before diving in, is that there are two distinct use-cases. First, extracting structured data from semi-structured text output, and second, taking structured data and feeding that data into a template engine in order to generate something from it (think configuration generation with Jinja2 templating). This post is discussing the former.
Template Text Parser
Template Text Parser (ttp) is the newest tool to the landscape and it's one that I quite like. It is simple to use, but has some tricks up its sleeve if you need to parse something more complex.
If you are familiar with Jinja2 templating language, then you will like ttp. Think of it as reverse Jinja2 templating (not technically analogous to Jinja2, but syntactically similar). Here is a simple example from the documentation to parse interface data from a Cisco device.
Below is another example that is a little more complicated. You'll notice that if the status of a VIP is down, the output does not include the portion. So in the case that we get crappy, inconsistent output from a device, we still can deal with it fairly easily. The interesting part in this example, is that we can apply a macro to a block of the template, or to an individual line in the template. A macro in ttp is simply a python function that is wrapped into a <macro> tag inside of the XML template.
Macro (Python Function)
Overall, I really like Template Text Parser. I think it's a welcome addition to the network automation toolbox. I can see using ttp to extract data from existing configurations and building a data structure that can then be used as input for configuration generation using Jinja2 templating.
PyATS & Genie
PyATS and Genie are Cisco libraries that have been developed by Cisco and released to the general public for a year or two now. They are really starting to gain traction after their very public introduction at Cisco Live US 2019 on the DevNet floor. Most of the tasks you can perform with PyATS and Genie is exposed via a command-line utility, but you can also invoke the parsers directly from python as well. These examples are simply using it for it's parsing capability, however this is only one of its many features. It is powerful, and can do so much more than what is described in this example.
Here is an example of parsing a show command with one of the 1200+ parsers (more being added every month) using the genie cli tool.
This example is the same as above, but invoking it from python natively instead of from the cli command.
Here is another example, except this time, there is no parser written by Cisco. Also, it is not being run on a live device, but output that was already collected by some other means. This uses the generic tabular parsing functionality of Genie called parsergen.
This is just scratching the surface of the capabilities of the pyATS and Genie duo. It is a very powerful set of tools, and the parser functionality is no exception. It is definitely another great tool in the network automation toolbox.
Parse Genie - Ansible Plugin
Parse Genie is an Ansible filter plugin that exposes the functionality of Cisco Genie parsers (as shown above) to users of Ansible. Below is an example of utilizing Parse Genie in an Ansible playbook.
The above playbook renders this output which is just a test playbook that shows the parsing capability.
Genie Functionality From Netmiko
Netmiko is another tool for connecting and interacting with devices and it has recently added support for Cisco Genie as well. Just a simple parameter on the send_command method and you can parse the show command output coming back from a device in Netmiko (so long as there is a Genie parser).
Network To Code has developed a fairly robust library of TextFSM parsers called ntc-templates. The templates can be used directly in your code or utilized from a library such as Netmiko. Here is an example of using ntc-templates natively in python.
NTC Templates Functionality From Netmiko
This example is the same as above, except using ntc-templates instead of Cisco Genie parsers.
Yangify is an interesting project that aims to take native configuration and parse it into a standard JSON data structure that validates against a YANG model. It also has the ability to translate instance data (JSON) into native device configuration. From my understanding, it performs a similar operation to that of Cisco NSO's network element drivers (NED). I have not had any free time to give it a go, but if the need arises, it is yet another tool in the toolbox. Staying on topic, the parsing functionality is implemented in pure python and therefore is left up to the ability of the user to write an effective parser and/or translator. The main benefit here is that it gives you the ability to validate instance data against a YANG model, which will ensure that the data is valid before it gets deployed to the network.
NAPALM is a library to provide a consistent, vendor-agnostic API to the network. It supports many different device types, and has a lot of functionality around interacting and managing a live network. However, since this is a focused post around parsing, let's just focus on NAPALM's getters. These are get methods that pull back runtime information in a standard format. Example getters are get_arp_table, get_facts, etc. Here is a list of all the supported getters.
Here is an example using connecting and retrieving structured data from python.
Sublime Text is one of my all-time favorite tools for working with text. I use it for a lot of ad-hoc text manipulation due to it's advanced text capabilities. This example will show advanced find/replace using regex. Specifically, taking a list of vlans and names from a CSV file, and generating a configuration snippet. This is a simple, contrived example, but the possibilities are endless for taking data, simple regex with capture groups, and creating configurations.
Standard Linux command-line tools are also invaluable and save a lot of time. I use these tools for generating commands from data for very repetitive tasks that are highly error-prone. Here is a prime example; removing all ACL lines for which there are zero hit counts. It takes the output of show access-list from a Cisco ASA, and generates the no commands necessary to remove them all from the firewall. I recently used this to remove 2273 unused ACL lines in a customer firewall. Instead of spending hours pouring through the output and hand-crafting no commands, I just used grep, sed, and awk and had my maintenance procedure in a couple minutes.
Less regex I hope 😎. In summary, text parsing (and regex) is a necessary evil, but there are a lot of great tools available. You'll notice that most of the above examples require no use of regex at all. Leveraging the right tool for the job going forward lets us all focus on what we are here to do; operate networks. Hopefully this post serves as a fairly complete snapshot of the available tools out there and how to use them.