Data Manipulation using GRASP
GRASP is an acronym I came up with that’s built upon various utilities and programming languages that work together to create a “unified toolset”.
I’ve found it to be very useful for a Network Engineer.
All of these tools are standard on most Linux installations.
The reason I came up with the acronym GRASP is, like most engineers, I sometimes have a hard time remembering all of the tools available that I’ve used in the past. Using the mental association and recall abilities you get from an acronym, I’ve found that it’s now really easy to remember that I have these tools at my disposal.
I learned the power of this type of mental association many years ago when I was in my first networking class. The teacher gave us a way to remember the 7 Layers of the OSI Model using this phrase:
“All People Seem To Need Data Processing”
That did the trick. While it’s not an acronym, the concept is the same, and I’ve never forgotten recalling any layer in the OSI Model since then by using this mental association, e.g. Application, Presentation, Session, Transport, Network, Data-Link, and Physical: “All People Seem to Need Data Processing”.
OK, so let’s get to Data Manipulation using GRASP.
GRASP consists of:
GREP - (globally search a regular expression and print)
REGEX - In theoretical computer science and formal language theory, a regular expression is a sequence of characters that define a search pattern, mainly for use in pattern matching with strings, or string matching, i.e. "find and replace"-like operations.
AWK - a programming language designed for text processing and typically used as a data extraction and reporting tool. It is a standard feature of most Unix-like operating systems.
SED - (stream editor) is a UNIX utility that parses and transforms text, using a simple, compact programming language.
Python - a widely used high-level, general-purpose, interpreted, dynamic programming language.
Let’s see a few examples of how the GRASP toolset can help you as an engineer.
Example 1:
Let’s say you need to advertise all the BGP routes from a CE to PE connected to an MPLS cloud for a client.
Let’s say you need to make these prefixes available to another disjointed area of the network. (Also using the BGP protocol with an IGP in between)
You could grab the currently advertised prefixes from the CE to PE (Total paths shortened for brevity):
br-sao-rt1#sh ip bgp nei 10.127.246.157 advertised-routes
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 10.127.246.157 0 8035 13979 65108 65108 i
*> 10.0.32.0/24 10.127.246.157 0 8035 13979 65130 i
*> 10.0.64.0/24 10.127.246.157 0 8035 13979 65130 i
Let’s say you’ve saved this to a text file named bgp-routes.txt leaving only the prefixes:
*> 0.0.0.0 10.127.246.157 0 8035 13979 65108 65108 i
*> 10.0.32.0/24 10.127.246.157 0 8035 13979 65130 i
*> 10.0.64.0/24 10.127.246.157 0 8035 13979 65130 i
Now we can see the benefit of using GRASP as a complete tool-set.
First, we use grep to find all the text files in our Linux directory:
ls | grep txt
bgp-routes.txt
Now that we’ve used (G)rep, we can continue by using (R)egex, (A)WK, and (S)ED
In the end, we’d be using (P)ython to automate programming the router.
We’ll be using a regular expression inside of SED and that will complete our Swiss army knife “tool-set”
Using a single command, we can take the output of the “sh ip bgp nei 10.127.246.157 advertised-routes” and turn it into a working configuration to program into the router/s. (Minus a few manual items we’d add to the configuration such as the correct BGP AS, go into proper address-family, exit and write memory commands)
Again, lastly, we’d use Python to automate the configuration of the router/s for us.
First, let’s check out the text file from within Linux:
cat bgp-routes.txt
*> 0.0.0.0 10.127.246.157 0 8035 13979 65108 65108 i
*> 10.0.32.0/24 10.127.246.157 0 8035 13979 65130 i
*> 10.0.64.0/24 10.127.246.157 0 8035 13979 65130 i
Now, we run just one command to manipulate this into a configuration using REGEX, AWK, and SED
The command will be explained after the results below:
cat bgp-routes.txt | awk {'print $2'} | sed -e 's/\/24/ mask 255.255.255.0'/ | sed -e 's/^/network '/
network 0.0.0.0
network 10.0.32.0 mask 255.255.255.0
network 10.0.64.0 mask 255.255.255.0
You can see from above that using a single command, we can change the output into a usable configuration that can be programmed into the router.
Now, we run the same command but send/redirect the output to a new file called bgp-commands-for-python.txt
cat bgp-routes.txt | awk {'print $2'} | sed -e 's/\/24/ mask 255.255.255.0'/ | sed -e 's/^/network '/ > bgp-commands-for-python.txt
Now if I look at this new file it’s ready to go:
cat bgp-commands-for-python.txt
network 0.0.0.0
network 10.0.32.0 mask 255.255.255.0
network 10.0.64.0 mask 255.255.255.0
So, a few things are at play here:
We start by reading the file using the cat command in Linux, and pipe the output to awk which prints the second “contiguous line of characters”
From there, we pipe the modified output to sed which executes a substitution. This one is a little tricky, in that we have to match on a forward slash / which is part of the syntax that sed uses to separate match and replacements, so we first use a \ (backslash) escape character to let sed know to treat the / character that’s part of the CIDR output correctly (classless inter-domain routing slash notation), e.g. it’ll see the /24 as a /24 and use the replacement string of “ mask 255.255.255.0” (Notice the space before string “mask”)
Lastly, we pipe that to sed again and use a regular expression to match the beginning of each line using the carat ^ character (which matches the beginning of any string in regular expressions). We replace it with “network “. Again, notice the space after the string “network”. Also, notice each modification is “building” upon each other, e.g. we are only sending (piping) the “modified output” to the next pipe for further processing. This is where you can see that they work in concert with each other, so you can creatively and quickly take data in one format and turn it into another format
The > at the end just tells the shell to redirect the final formatted output to the file we specify. In this case it’s “bgp-commands-for-python.txt”
You can/will need to adjust for all your different masks (sed replacements). This can be done very easily if you have many masks as shown in the next example.
So, as you can see, in just one command, you can edit many networks, take the output from a “show” command and turn it into a working BGP configuration. At this point, you can use Python to automate programming the router.
This is just one example. The GRASP toolset can be used to easily convert prefix-lists into OSPF or BGP router advertisements (or convert BGP tables into prefix-lists too).
UC guys can use the tool-set, Security guys can use it. The tool-set is practice agnostic.
It can be used for anything you can think of, e.g. used to convert any format like CSV, an email with information all over the map someone wants programmed into the network, etc.
Example 2:
For example 2, we’ve got many different subnet masks to convert. (Again, total paths shortened for brevity – but this could be 500 prefixes and wouldn’t take any longer):
*> 10.0.0.0/9 10.44.7.241 20 32768 ?
*> 10.44.0.0/14 10.44.7.241 20 32768 ?
*> 10.56.0.0/18 63.158.142.129 200 0 209 65156 i
*> 10.56.128.0/20 63.158.142.129 200 0 209 65156 i
*> 10.57.0.0/19 63.158.142.129 200 0 209 65156 i
First, we use grep to find all the text files in our Linux directory:
ls | grep txt
bgp-routes-allot-of-prefixes.txt
Then, we run awk to get the 2nd column’s output and send the output to a new file called “bgp-allot-of-prefixes-for-web-tool.txt” (we are going to use a web based tool to convert CIDR notation into masks.
cat bgp-routes-allot-of-prefixes.txt | awk {'print $2'} > bgp-allot-of-prefixes-for-web-tool.txt
10.0.0.0/9
10.44.0.0/14
10.56.0.0/18
10.56.128.0/20
10.57.0.0/19
There’s a web based tool (Which probably uses python on the backend) that allows you to input line by line as many CIDR notation based routes as you’d like and it will convert it for you. You can then export it to a CSV file and/or copy it directly into a text file and from there, run parts of the GRASP toolset again to create your configuration file in a few minutes.
Go to:
Hit calculate and see output below:
At this point, you have your network address and subnet masks ready and you can save it to a CSV file to further process using the GRASP toolset (NOTE: I omitted the first column (CIDR) when I put it directly into a text file ,e.g MY first column is Network address).
So when I run the commands, I take that into consideration. (Just going to pull out the first and fourth columns): Network Address and Subnet mask.
I named the text file “grasp-article-example-2.txt”
So, I run one command using awk and sed and create the following output, I am extracting only columns 1 and 4 and inserting “network” in the beginning:
Again, the same one command would have created the configuration for 500 prefixes in the same amount of time.
cat grasp-article-example-2.txt | awk {'print $1 " " $4'} | sed -e 's/^/network '/ > grasp-example2.txt
Let’s take a look at the new file. It’s still not complete, as we have to run one last command.
cat grasp-example2.txt
network 10.0.0.0 255.128.0.0
network 10.44.0.0 255.252.0.0
network 10.56.0.0 255.255.192.0
network 10.56.128.0 255.255.240.0
network 10.57.0.0 255.255.224.0
Finally, we run one more command and this completes the configuration, all done in 10 minutes total.
cat grasp-example2.txt | awk {'print $1 " " $2 " mask " $3'} > final-config-allot-of-BGP-prefixes.txt
Let’s take a look at the final configuration:
cat final-config-allot-of-BGP-prefixes.txt
network 10.0.0.0 mask 255.128.0.0
network 10.44.0.0 mask 255.252.0.0
network 10.56.0.0 mask 255.255.192.0
network 10.56.128.0 mask 255.255.240.0
network 10.57.0.0 mask 255.255.224.0
This example comprised running three commands total. We really could have done it in one command as well, but just wanted to break it down a little easier for the article.
Your final commands file is just a .txt file that python looks for within a script, so it’s easy to automate programming the router using python.
Now, as mentioned previously regarding a high number of prefixes, imagine you have a BGP neighbor your sending 500 prefixes to that you need to do this on with many different subnet masks. This could be very time consuming to do this manually and could be subject to human error as well. Using things like grep, regex, awk, sed, and python make it super quick and less prone to human error.
Time wise, it’s pretty much the same whether it’s 5 prefixes (like the example above) or 500 prefixes. We are talking minutes here, as opposed to hours or days.
Again, the actual programming of the router or routers is the same thing. Python does the work for you.
To recap, we started with this output (Every prefix has a different mask):
*> 10.0.0.0/9 10.44.7.241 20 32768 ?
*> 10.44.0.0/14 10.44.7.241 20 32768 ?
*> 10.56.0.0/18 63.158.142.129 200 0 209 65156 i
*> 10.56.128.0/20 63.158.142.129 200 0 209 65156 i
*> 10.57.0.0/19 63.158.142.129 200 0 209 65156 i
And with three simple commands we turned it into:
network 10.0.0.0 mask 255.128.0.0
network 10.44.0.0 mask 255.252.0.0
network 10.56.0.0 mask 255.255.192.0
network 10.56.128.0 mask 255.255.240.0
network 10.57.0.0 mask 255.255.224.0
For five networks, of course you’d just do it manually, but for hundreds of prefixes, this is going to save you allot of time.
This tool-set is also great during the discovery phase too, giving you flexibility (and creativity) in your methodology and how you approach understanding the network. It’s going to help with extracting and analyzing the data in a way you want to see it, and in a way you understand it best.
That’s it…..Happy networking….
Here’s a few links to the various components of GRASP:
https://www.gnu.org/software/grep/manual/grep.html
http://regexr.com/
https://www.gnu.org/software/gawk/manual/gawk.html
https://www.gnu.org/software/sed/manual/sed.html
https://docs.python.org/2.7/
Best Regards,
Brett M. Spunt, CCIE 12745
