Story Time:
Shortly after I earned my CCIE I was faced with a packet analysis challenge. I was on-site visiting with some team mates who managed a customer’s network. A user at a site was complaining: “the network is slow”. My team has been working with this site for a few days checking all the routing and devices in the network path, testing traffic from all the network devices and more. The user was still having issues and sent a packet capture from their workstation to my team. I started shoulder-surfing as soon as I saw Wireshark open on their desktop. They were hoping their freshly minted CCIE coworker could quickly help them out. The problem was, I had no idea what I was looking at. There I was, the most certified person in the room, and no idea how to begin to properly analyze this packet capture to help narrow down where or what may be causing network performance issues. This was embarrassing for me. I wanted to make sure that anytime someone gave me a packet capture to analyze that I had a strategy, and an understanding of what something should look like under optimal conditions. This started my packet analysis journey.

Packet Acquisition

“Packet Acquisition” is the process and/or tooling used to acquire the packets or ‘actually capturing them’. There are countless blogs and articles about the TAP vs. SPAN arguments. This is not that.

Before you can acquire the packets you should answer 2 basic questions:

1. Where to capture from?
   
   • From a link using a TAP
   • Directly from a device
2. What tool to capture with?
   
   • The devices own capture capability
   • B.Y.O.T. (Bring Your Own Tools)

Where to Capture

In order to know “where is the best place to capture traffic” you’ll need to understand what it is you’re looking for and where that traffic flows. It wouldn’t do you any good to capture from a device or a link that your traffic doesn’t pass through. Sometimes, the best place isn’t an option for you, so you’ll also need to be prepared to capture from a less than ideal alternative location while still gathering the data you need to perform analysis or prove your hypothesis.

NOTE: This blog post makes some assumptions about your work experience, knowledge and understanding of networks and network traffic.

The answer to: “Where is the best place to capture traffic?” is relative, as different places throughout the network can give different vantage points. Regardless, if it is the best place or not, your choices will fall into 1 of two categories:

• Strategic placement of a TAP or visibility sensor
• Living off the Land (LoL)

Strategic TAP/Sensor Placement

Placing a TAP or a sensor exactly where you want in the network is the most ideal situation.

Examples of strategic TAP/Sensor placement:

• In front of a server where all users are experiencing delays.
• In front of a printer that frequently is ‘unreachable’.
• South of a FW to have visibility into East-West traffic.
• Between a FW and Router to have visibility on all the North-South traffic.

Using a network TAP/visibility sensor is optimal because it provides the true view with the true delay of exactly what is on the wire. This method is absent of other computationally expensive processes that can occur when you have to Live off the Land. This is especially important when diagnosing issues related to delay and timing. Many high end servers have specialized NICs with additional features or functions designed to ‘lighten the load’ on the processor. This can be a disadvantage when capturing directly from a device and trying to get the ‘true view’ of the packets in flight. For this reason capturing from a link with a TAP in-line is a superior choice.

Living off the Land

“Living off the land” is a phrase meaning: using a devices built-in capture capability without any additional tooling.

In a typical enterprise network there are networking components like: routers, switches, firewalls and more. There are also end-points and workstations. Most of these devices have some packet capturing capabilities built in. Capturing from each of these devices is a little different and can yield slightly different packet captures, each with their own view of the traffic, which can result in multiple-truths. As a network engineer or security analyst, it’s your job to understand the concept of multiple-truths and apply that to your analysis of the packet captures.

Multiple-Truths:

I love this image. I use it all the time. Multiple things can be true regarding a single issue and it’s only a matter of perspective.

Examples of Living off the Land:

• Using the packet capture capability of the router
• Leveraging the SPAN/Mirror port of a switched infrastructure
• Using tcpdump/wireshark directly on the end-host/server

Ultimately deciding “where to capture” becomes a logic exercise between what you want to see and where you can see it from. The bottom line is:

Having /a/ PCAP is better than no PCAP!

What to Capture With?

Depending on whether you are strategically placing a TAP/sensor or Living off the Land, might pre-determine the tools you use to capture the packets. For example, if your capture point is a Cisco router, you might be limited to using Cisco’s Embedded Packet Capture. On the other hand if you are placing a TAP between 2 devices and capturing using a fully featured operating system, you’ll likely have many tools available to you.

Living Off the Land: Routers

Most routers have some way to capture packets for diagnostics. Generally speaking this is not suitable for long-term captures because of the potential performance degradation. This should be used for short-term troubleshooting only.

• Cisco’s Embedded Packet Capture (EPC)
• Juniper
• OpenWRT
• VyOS
• pfsense

Warning: Routers are typically built for high-speed data-plane packet transfers. When capturing from a router you are pulling packets out of the high-speed data path and pushing them through the lower speed (lower resourced) control plane/management plane.

Living Off the Land: Firewalls

Similar to routers, most if not all, firewalls have a built-in packet capture capability. Some can even mirror traffic from one port to another. Again, this type of capture should be used for specific troubleshooting use cases and is not suitable for long term packet captures.

Note: If using Palo-Alto Firewalls and using tcpdump from the cli, by default it will only capture on the MGMT interface, not the data plane interfaces. Ask me how I know :D

Living Off the Land: Switches

SPAN ports, SPAN ports, SPAN ports.

To be clear, a SPAN port isn’t a tool to capture the packets, instead its part of the plumbing in the switch that duplicates the packets out an interface so you can capture them. Again, I won’t rehash the age old discussion of TAP vs. SPAN but if a SPAN is all you have, it’ll have to do. I am a fan of the SPAN. (<–I’m going to put that on a t-shirt) There are some cool things you can do with a SPAN port from a switch such as capturing an entire VLAN. This isn’t without it’s caveats, you can severely impact the performance of older switches, and you can over-subscribe the interface transmitting to your capture device. Also, incoming malformed frames aren’t duplicated but dropped on ingress to the switch. (<–If this was your issue you’d probably want to be able to see it). There was a really good article on packet-pushers blog about capturing from a switch that I think should be a must read.

There are lots of options when using a SPAN/mirror port and they differ from Vendor, model and software version.

WARNING: Over-subscription is your enemy when it comes to SPAN ports.

SPAN Port References

• Cisco:

  no monitor session 1
  default interface gi0/2
  
  monitor session 1 source interface gi0/1
  monitor session 1 destination interface gi0/2 
  

• Juniper:

  set interfaces ge-0/0/47 description "SPAN-PORT for Testing"
  set interfaces ge-0/0/47 unit 0 family ethernet-switching
  
  set forwarding-options analyzer SPAN_NAME input ingress vlan 10
  set forwarding-options analyzer SPAN_NAME output interface ge-0/0/47.0
  

B.Y.O.T. (Bring Your Own Tools)

If you are capturing from a TAP, visibility sensor or directly from an end-point device like a workstation you’ll have the option of choosing the best tool for the job. You should choose the tool that has the features you need and runs efficiently on the system you’re using. Depending on your environment you might be in a situation where there’s only one tool available, like a linux environment without internet access to get new packages. In this case most linux distro’s include tcpdump. Some of the most common packet capture software are:

Basic Usage

tcpdump: tcpdump is very fast and uses bpf syntax for filters. The most simplest usage example is:

# Dump packet summary to stdout
tcpdump -i [interface_name]

# Write packets to PCAP file
tcpdump -i [interface_name] -w [filename.pcap]

dumpcap: This is the capture utility the Wireshark GUI uses when you’re capturing from an interface. A simple usage example:

# Write packets to PCAPng file
dumpcap -i [interface_name] -w [filename.pcapng]

tshark: This is the reading and filtering engine Wireshark GUI uses. This is a command line utility. A simple usage example is:

# Dump packet summary to stdout
tshark -i [interface_name]

# Write packets to PCAPng file
tshark -i [interface_name] -w [filename.pcapng]

Wireshark: The GUI application can be invoked via CLI as well:

# Write packets to PCAPng file
wireshark -i [interface_name] -w [filename.pcapng]

Pro-tip: Become fluent in the 3-above examples. You might be in a situation where its the only tool available and you have to make it work.

Final Thoughts

Like networking skills, your packet analysis skills will mature. The more you do it, the more you’ll learn. First you get familiar with the tools, then the process, then the analysis. Eventually, you’ll become syntacticly fluent in multiple tools, able to get packets out of any environment and analyze traffic from unknown network topologies, quickly.

Remember: Packet Acquisition is step #1 to Packet Analysis.

Moment of Truth: I’ve tried to describe all of the above in a very scientific and methodical way that will yield positive results. Sometimes scientific methods can take the fun out of it. How about you just start capturing with what ever you have and see what you get. Have fun! I know I will :D

Remember: If women don’t find you handsome, they should at least find you handy.

While, I do fancy myself as someone who pays close attention to detail, this time I was bested.

I followed many guides to get docker running in WSL.

I’m really loving having native linux on my Windows 10 laptop. There’s a small learning curve when needing to troubleshoot if things go wrong.

Not being a docker expert myself or experienced with Windows Sub-system for Linux (WSL), I was following someone else’s guide. No where was it mentioned about the different versions of WSL or any problems that could cause.

Exhibit: A

https://dev.to/felipecrs/simply-run-docker-on-wsl2-3o8

# Ensures not older packages are installed:
sudo apt-get remove docker docker-engine docker.io containerd runc

sudo apt-get update

# Ensure pre-requisites are installed:
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

# Adds docker apt repository:
echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" |
    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Adds docker apt key:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg |
    sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Refreshes apt repos:
sudo apt-get update

# Installs Docker CE:
sudo apt-get install docker-ce docker-ce-cli containerd.io


# Ensures docker group exists:
$ sudo groupadd docker

# Ensures you are part of it:
$ sudo usermod -aG docker $USER

# Close your shell and reopen to make sure you're in the correct group (docker):
groups

# Open ~/.profile and add:
if service docker status 2>&1 | grep -q "is not running"; then
    wsl.exe -d "${WSL_DISTRO_NAME}" -u root -e /usr/sbin/service docker start >/dev/null 2>&1
fi

Even after manually starting the service (sudo service docker start), I would run the ‘hello-world’ test and get an error:

user@hostname:~$ docker version

Client: Docker Engine - Community
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        dea9396
 Built:             Thu Nov 18 00:37:06 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

The above error along with others was indicating iptables couldn’t be found. Even when I manually check for iptables, I found it’s not installed! How could this be? Now, were shifting from running and troubleshooting docker to investigation why iptables isn’t included in Ubuntu.

Turns out, iptables isn’t included with Ubuntu distro using WSL version 1. Unknowingly, I was using WSL1 not WSL2.

You can check that in a Windows command prompt:

wsl -l -v
  NAME      STATE           VERSION
* Ubuntu    Running         1

You can set a version number by:

wsl --set-version ubuntu 2

When I did this, I received an error from Microsoft about a “kernel update package” and a hyperlink: https://docs.microsoft.com/en-us/windows/wsl/install-manual#step-4—download-the-linux-kernel-update-package

You’ll need to install the Kernel Update Package as administrator.

Then, set the version of WSL for your distro:

wsl --set-version ubuntu 2

It takes a few minutes to convert, but once it does, I was able to open ubuntu and the iptables command worked perfectly as well as starting the Docker daemon.

I’ve set my default WSL version to “2” using: wsl --set-default-version 2

I am now a WSL lover and trying to incorporate it in many of my workflows.

• 2-Day Network Analysis and TCP Deep Dive with Wireshark (Chris Greer)
• Next Generation Protocols & Advanced Network Analysis (Phill Shade)
• Analyzing Pcaps Faster with Filters (Betty DuBois)

This year I presented at SharkFest! My presentation was titled “School from Home: Watching the wire with Wireshark” where I talked about some interested traffic I found coming from the school provided Chromebooks and how I took a deeper dive to taking advantage of implied trust. (When ever the video gets posted to YouTube I’ll include a link)

I spent most of the 2nd day of conference presentation playing the CTF. The CTF this year was really well done with tons of challenges and some really hard ones too. If you want a CTF that focussed on packet captures and packet analysis this is the place to be. Overall I got 5th place out of 68 teams and I’m happy with that.

“Something worth doing, is worth doing twice!” - Tony E.

Because of a configuration error I realised after completing all my slides and generating my example PCAPs, I needed to spend most of my time during the training classes pulling double duty. I was halfway participating in class and the other half remaking my slides and PCAPs. I even spent most of Day-1 from the conference finalizing my slides and semi-rehearsing for my presentation.

Imposter Syndrome

This was the first ‘conference’ I’ve spoken at. While I typically don’t have any major stage fright, I was feeling a bit nervous because everyone who was presenting were folks who I look up to as experts in the field of packet analysis and I was exeriencing some major imposter syndrome.

Despite my imposter syndrome, I felt good about the execution of my presentation. I would’nt have changed anything about my presentation except timing. I should’ve start of a little faster pace because at around the halfway mark I realized I only made it through 1/4th of my slides. I quickly picked up the pace and at that rate I felt was too fast and not comprehendible. I also had to gloss-over or skip some more in-depth analysis but… It is what it is. I’ll make that a lessons Learned.

Things I Learned From SharkFest

😀 Emoji’s in the Column Headers 😁

The Wireshark column headers supports unicode characters, such as emojis. As soon as I learned this I started graffiti’ing my profiles with pointless and rediculous smiley faces. This might seem useless but there are a few practical uses:

• Columns related to time can get a fancy stopwatch: ⏱
• Delta columns can get a: 🔺 (or any ‘delta’ style shape)

Coloring Rules 🟥🟧🟨🟩🟦🟪🟫

I’ve seen SharkFest presentations before where customizing you profiles can give you an advantage or efficiency when analyzing certain traffic. While it really just comes down to personal preferences, in the past I brushed this idea under-the-rug and focused on my protocol fundamentals. Well, maybe it’s a sense maturity, but I now recognize the value of coloring rules: It’s not just coloring rows for personal preference but also creating a column for your coloring rules and using the rule name as a filter.

Maybe this^ needs a blog post all on it’s own?

PCAP File Structure

Recently I’ve needed to dive into the file structure of PCAP and PCAPng file types. See: Modifying PCAPng File Structure Using A Raw Hex Editor and https://twitter.com/showipintbri/status/1437070853062156296.

I had no idea you could investigate the file structure itself within Wireshark. For that navigate to: View > Reload as File Format/Capture

^ This is exactly why I love attending conferences. You learn about cool projects people are doing. Tips & Tricks to make your job easier and an opportunity to as experts questions!

TCP SACK’s and TCP Stream Graphs

Last year I took Chris Greer’s TCP trainings prior to SharkFest. I learned a ton! Infact there was so much to learn I deffinitely didn’t absorb all of it. So, this year I signed up for the training again hoping that a second time going over the material it would sink-in a little more… and it did. My focus was on day 2 of the training as Day 1 is alot of TCP basics and Wireshark basics. Day 2 is some more advanced TCP topics such as TCP SACK and understanding stream graphs. <- This was the content I came for.

I was having trouble remembering which lines indicated the various TCP header components. Also, I just wanted more organic practice of looking at the graphs on real traffic to help train my eyes and my mind to recognize graph shapes and paterns.

Read Filters

I often have to open files greater than 1 GB. This is one of the reasons I lean on so many other tools for processing large files before diving in with the Wireshark Microscope. Processing large captures in Wireshark is dificult because each time you change or update a display filter it needs to re-scan the entire file which can takes time. Read filters will apply to when you open the PCAP so your subsequent display filters will be on a much smaller sub-set of traffic.

The Read filter can be fore when to open a PCA from the Wireshark interface: File > Open…

NOTE: It is will still take time on initial load.

NOTE: When a display filter is applied it can still take a long time if the after the read filter you still have alot of packets.

Window Scaling, OMG Window Scaling!!!

During the CTF there was a challenge about TCP Window Scaling. This one took me a while. Alot longer than it should have but I was stuck on the fact that Wireshark was calculating the ‘Calculated Window Size’ for me and I couldn’t figure out the math it was use or hos it was deriving the values it was using. Sure I could’ve just popped into the chat and asked one of th emany experts on the subject but I really wanted to figure this out on my own.

The Window Scale value is a single byte in width. The Window Scale Factor is 2^[window Scale Value]: 2 to the ___th power.

This value, if the option is agreed upon from both end points, is applied to all TCP packets after the SYN & SYN/ACK.

Reference: https://datatracker.ietf.org/doc/html/rfc1323#page-8

I title this section “Window Scaling, OMG Window Scaling” because there’s something to be said about memory and emotions. I don’t know the exact science behind it but you are much more likely to remember shomething if you have a specific emotion tied to that event or that thing. In this case, the emotion I felt when finally sorting out the window scaling I’ll never forget this now. Forever it will be burned in my brain as: “Two to the __nth!”

In Conclusion

While it’s been on my radar for much longer, Sharkfest has been amazing for me over the past 2 years. I have learned so much about packet analysis and understanding the network traffic that uses the plumbing we(network engineers) build. I also had fun hanging out in the “Developerd Den” where I was chatting and interacting with some of Wiresharks core developers. I was able to ask about certain features and make suggestions for upcoming features. Wireshark truly is an amazing project with a great community around it 🤘.

After reading a recent blog from the VyOS dev team demonstrating an example of supporting containers natively from the VyOS CLI, I knew I had to try it with Suricata. Combining an amazing routing platform, with best in class Network Security Monitoring (NSM) and Intrusion Prevention System (IPS), both of which are Free and Open Source Software (FOSS) has been a dream of mine.

INB4: Yes, I am well aware that pfsense and OPNsense has Suricata and FRR plugin’s available. I’ll withold my opinions of them from this post and save my commentary on them for a different post. (I’m not a fan! :hushed:)

The Problem Statement:

I needed to build a proof-of-concept to standardize the router/gateway/edge hardware for my companies locations.

• Hardware should be my companies own hardware.
• Platform must support automation via ansible and API.
• Platform should be FOSS.
• Software should be modular. (the ability to add, remove or modify components as necessary)
• Network Security Monitoring (NSM), Intrusion Prevention System (IPS) and other Next-Gen FW capabilities should be supported.

This blog doesn’t dive into automation or other resource and performance monitoring but, it is the first in a series of proof of concepts using my companies own hardware as an all-in-one platform that fits our needs. Hopefully many more fun usecases and experiments to come. 🤞

• Proof of Concept:
  
  • VyOS & Suricata
  • VyOS & Zeek
• Performance Testing:
  
  • VyOS & Suricata on SN-3000 Hardware
  • VyOS & Zeek on SN-3000 Hardware
• Automation Tooling
• Operational Playbooks

Hardware Specifications

I’m using my companies own SN-3000 series small form-factor node. While this unit can support larger specs, I opted for reduced spec’d components:

• Memory: 128GB RAM
• Storage: 128GB SSD
• Processor: Intel(R) Xeon(R) D-2183IT CPU @ 2.20GHz

Platform and Software Versions

I’m calling the base VyOS a “platform”.

• Platform: VyOS 1.4-rolling-202105192127 (sagitta)
• Podman: 3.0.1
• Suricata: 6.0.2 (container)

Network Architecture

I’m currently leveraging this box at my home, during the proof-of-concept period. It is acting as my internet gateway performing outbound NAT, basic stateful ACL’s and inter-VLAN routing (router-on-a-stick).

• The layer-2 trunk uplink from my switch to the SN-3000 is 1Gbps copper.
• The link from the SN-3000 to my ONT CE (FTTH) is 1Gbps copper with 1Gbps CIR from my ISP.

TL;DR: Here’s the Code

1. Edit registries.conf and add the docker registry:

sudo nano /etc/containers/registries.conf

2. Uncomment and change “example.com” -to-> “docker.io”, save and exit.

unqualified-search-registries = ["docker.io"]

3. Pull the Suricata container:

sudo podman pull jasonish/suricata:latest

4. Verify:

sudo podman images

5. Create a directory structure to store rules files, logs and configurations between container restarts.

Below is my preference:

   ~/suricata/
      |
      |-etc/ (suricata configurations and rule exclusions)
      |-logs/ (suricata stats.log, fast.log & eve.json)
      |-rules/ (suricata combined rules files)

cd ~ && mkdir suricata && cd suricata && mkdir etc && mkdir rules && mkdir logs 

6. Verify suricata was built with NFQ support:

# From within ~/suricata/ run:

sudo podman run --rm -it --net=host \
     --cap-add=net_admin --cap-add=sys_nice \
     -v $(pwd)/logs:/var/log/suricata \
     -v $(pwd)/etc:/etc/suricata \
     -v $(pwd)/rules:/var/lib/suricata \
         jasonish/suricata:latest --build-info

From the output printed, look for: NFQueue support: yes

🛑 STOP!

When you ran the command above in Step #6, the suricata container started, checked the directories for its configuration files, didn’t find any, and placed the default configuration files in the default directory: ~/suricata/etc/suricata.yaml, then it ran with the --build-info flag and stopped.

This means you now have a default suricata.yaml file in ~/suricata/etc/ that you can modify to configure suricata to run and operate how we need it.

Best practice is to make a backup copy of this default YAML file so you can always revert to default if you booger up your configuration.

🟢 GO!

7. Configure suricata.yaml

Edit ~/suricata/etc/suricata.yaml to suit your needs. All of the available configuration options in this YAML file is way to much to review in this short blog. Please see https://suricata.readthedocs.io/ for a much more verbose description of the configuration file content and options.

Below are just some of the options I changed for my use case.

outputs:
  - eve-log:
      community-id: false
      types:
        - anomaly:
	    enabled: yes

outputs:
  - eve-log:
      community-id: true
      types:
        - anomaly:
	    enabled: no

host-mode: auto

host-mode: router

nfq:
#  mode: accept
#  repeat-mark: 1
#  repeat-mask: 1
#  bypass-mark: 1
#  bypass-mask: 1
#  route-queue: 2
#  batchcount: 20
#  fail-open: yes

nfq:
  mode: accept
#  repeat-mark: 1
#  repeat-mask: 1
#  bypass-mark: 1
#  bypass-mask: 1
#  route-queue: 2
#  batchcount: 20
  fail-open: yes

The Suricata documentation has a really good section explaining the various nfq modes.

8. Start the suricata container:

NOTE: Since this is purely a proof of concept and not optimized for performance, I’m leveraging a single queue.

# From within ~/suricata/ run:

sudo podman run --rm -itd --net=host \
     --cap-add=net_admin --cap-add=sys_nice \
	 --name=suricata \
     -v $(pwd)/logs:/var/log/suricata \
     -v $(pwd)/etc:/etc/suricata \
     -v $(pwd)/rules:/var/lib/suricata \
         jasonish/suricata:latest -q 1

Verify: Check the: ~/suricata/logs/suricata.log file to verify everything is starting correctly.

Regarding this particular setup I’m looking for:

11/6/2021 -- 17:37:00 - <Info> - Enabling fail-open on queue
11/6/2021 -- 17:37:00 - <Info> - NFQ running in standard ACCEPT/DROP mode

9. Add the required iptables rule to forward traffic to suricata:

Because IPS rules/signatures are more computationally expensive than firewall rules(tuple matching/connection tracking). I follow the basic principal that firewall rules should be processed before IPS rules/signatures. While my SN-3000 has way more horsepower than I need for my home traffic and can process all my traffic through IPS I still follow the principal. For that reason I only send packets to Suricata only after they been filtered by iptables. This is why I insert this rule as #4 in the FORWARD chain. On the flip side I could insert a similar iptables rule first and send all traffic to suricata before it hits the iptables rules defined by my VyOS configuration.

To understand this a little please read through the suricata doumentation on NFQ.

NOTE: Since this is purely a proof of concept and not optimized for performance, I’m leveraging a single queue.

sudo iptables -I FORWARD 4 -p all -j NFQUEUE --queue-bypass --queue-num 1

If you need to remove: Assuming your rule is still number 4 in the FORWARD chain (verify with: sudo iptables -L FORWARD -n --line-numbers)

sudo iptables -D FORWARD 4

10. Enable rule sources and restart suricata:

sudo podman exec -it --user suricata suricata suricata-update
sudo podman exec -it --user suricata suricata suricata-update list-sources
sudo podman exec -it --user suricata suricata suricata-update list-enabled-sources
sudo podman exec -it --user suricata suricata suricata-update enable-source et/open
sudo podman exec -it --user suricata suricata suricata-update enable-source oisf/trafficid
sudo podman exec -it --user suricata suricata suricata-update -f

11. Check your logs

Check the logs directory and verify your files are growing. Verify their contents. Enjoy. Happy Hacking!

Circling Back

The inspiration of this proof-of-concept was derrived from the VyOS blog. In that blog, containers are instantiated using the native VyOS CLI and configuration style. The process I documented above doesn’t leverage the VyOS CLI or configuration style because I’m not sure how to leverage the CLI’s container statements and pass it some of the necessary --cap-add options. I don’t foresee any issues running this via native linux versus the VyOS configuration style. If we want suricata to start post boot we can add it to the /config/scripts/vyos-postconfig-bootup.script. I’m fine with manually starting it for now.

Fail open by design (my preference)

NOTE: This is my preference only! By tweaking some of the configuration this setup can be changed to fail-closed if you prefer. Please consult your organizations policy on fail-open network equipment.

Because this appliance is currently my internet gateway and I have an active household (school & work) I can’t let my mucking around with suricata affect the egress or ingress traffic. For my home-usecase having a blindspot from suricata is of much less importance than having 99.999% uptime.

There are 2 configurations that drive this “fail-open” design:

• --queue-bypass in our iptables rule
• fail-open: yes in our “nfq:” section of the suricata.yaml file

--queue-bypass is great because it allows VyOS to forward packets and filter them through the iptables/firewall rules that exist while, suricata is doing things like reloading a rules set. What this actually does is keeps the packets in the kernel and filtering them through iptables if a userspace application isn’t available to receive them from the specified queue.

Reference: https://ipset.netfilter.org/iptables-extensions.man.html

By default, if no userspace program is listening on an NFQUEUE, then all packets that are to be queued are dropped. When this option is used, the NFQUEUE rule behaves like ACCEPT instead, and the packet will move on to the next table.

For me this is perfect because of where in the FORWARD chain I put my NFQUEUE rule. I’m processing all my drop and reject iptables rules before the NFQUEUE rule is evaluated. If suricata isn’t available, then the packet is permitted. This doesn’t permit any unecessary packets that would otherwise be dropped because the following rule is an “ACCEPT all” rule. If I never bothered with suricata or if I don’t have suricata running, the packet would’ve passed anyway by ACL design.

fail-open: yes is an available option baked in to suricata which says: ‘if suricata can’t keep up with the packets in the queue (when the queue is full), mark the incoming packets as ACCEPT. For this proof of concept this is acceptable until a baseline of performance can be measured and predicted.

What to do with the logs and events?

If your hardware is powerful enough (and my SN-3000 is) you can host any number of databases, dashboards and search interfaces. Examples are: Splunk, Elasticsearch with Kibana, Graylog or something custom. If your hardware is not powerful enough to run other containers in addition to suricata I would recommend shipping them off the box with the recommended shipper for your database platform. Examples could be: Logstash or some custom scripting.

More to come

While this is the end of this blog post it isn’t the end of my on-going project. I still need performance testing, automation tooling and operational playbooks.

• Proof of Concept:
  
  • VyOS & Suricata
  • VyOS & Zeek
• Performance Testing:
  
  • VyOS & Suricata on SN-3000 Hardware
  • VyOS & Zeek on SN-3000 Hardware
• Automation Tooling
• Operational Playbooks

What other containers should I run on the SN-3000 using VyOS as a Platform?

Resources

• https://blog.vyos.io/vyos-project-may-2021-update (VyOS blog for demoing running containers via CLI)
• https://suricata.readthedocs.io/en/suricata-6.0.2/configuration/suricata-yaml.html#nfq (NFQ operation as explained by the Suricata Documentation)
• https://github.com/jasonish/docker-suricata (for a pre-packaged Suricata container)
• https://docs.mirantis.com/mcp/q4-18/mcp-security-best-practices/use-cases/idps-vnf/ips-mode/nfq.html (for help with understanding NFQ settings)
• https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/ (for help understanding NFQUEUE operation)
• https://suricata.readthedocs.io/en/suricata-6.0.2/setting-up-ipsinline-for-linux.html

tl;dr This is not a humble brag but if you have good experience and are a professional in at least 1 or more related fields, it might not be too difficult.

On Monday (March 29, 2021) I passed my GIAC: Network Forensic Analyst certification exam with a 92%. For study resources I used the SANS: FOR572 online video series, slides and books. I didn’t dedicate a ton of time into studying. Infact this probably took me nearly 100 days from the time I purchased the exam attempt and actually taking the exam. Within the 100 days were weeks when I didn’t crack a book or start a video.

SANS FOR572

In the SANS FOR572 series Phil Hagen(@PhilHagen) does a really great job of bringing you up from Zero-to-Hero throughout the course. In my opinion you definitely should have some basic experience in networking, security concepts like boundary defense, and understanding how things look on the wire and how they should look on the wire. If you only have a couple years of good experience in the field, spanning multiple domains, you should have no problem keeping up with the material. If you have less experience, it doesn’t mean you can’t pass the exam but, it may be more difficult. Phil speaks clearly and delivers the material very well, mixed with tales of personal experiences.

While I haven’t been doing incident response or digital forensics at all as part of my day job, I regularly participate in CTF’s chasing flags through an environment or hunting for flags through PCAPs. Most of the tools used throughout the course material I have running in production environments or have running here at my home for network visibility and “research”. I spent most of my career around traditional networking and network security products like firewalls, IPSs, Proxies and more. I understand multi-vendor and multi-capability security stacks well. So, when I saw the tools and concepts Phil Hagen was presenting in the material it felt very comfortable.

The Exam

I come from a Cisco testing background having gone from CCNA to CCIE, I was happy to NOT see what are know in the industry as “Cisco Style Questions” where there are multiple correct answers and you have to pick the most correct answer. Instead, the GIAC: NFA exam was clear and there was a single answer for each question. This alleviates a lot of frustration that can sometimes occur through certification exams. I finished the exam with lots of time to spare.

Testing outline:

• 50 questions
• 2 hours
• Open Book !!!

This was the first exam I’ve ever taken that was an open book test. It was weired and frankly I forgot to reference them throughout the test. There were a few questions where I paused as I was looking over the multiple choice answers and remembered: “Oh right, I can look this up!”, I would thumb through the book to the relevant section and skim and page or two until I could verify my intended answer. Without the books I still would’ve passed but not gotten as high of a score.

Index Method

I did not use the index method. I read as much as I could about this and watched a few ‘how-to’ videos but I just didn’t understand it. I started a spreadsheet but got about 25-rows done and gave up. It’s not for me. I see the value in it so I’m happy it works for many others but it’s not for me.

Practice Exams

I purchased 2 practice exams with my certification attempt. When I was about halfway through the material I took one, just to see if I was ready because frankly I was getting tired of studying and just wanted to get it out of the way. I scored well but ultimately decided to push through all the material before taking my certification attempt. The night before my certification attempt I took my 2nd practice exam and I scored even better. At that point I felt confident I would at least pass the exam, regardless of score. During the practice exams I didn’t use the books very often, 2 or 3 times maybe. I did feel like the practice exams were a good representation of the certification exam. I don’t believe they are from the same question pool, but they are similar so read the questions carefully and look at the diagrams carefully.

Why?

If you don’t do DFIR why take this exam?

I’m not sure why I pursued this as opposed to any other certifications. Personally, I always wanted to be the “anti-hacker”. The guy who could use a bunch of crazy commands CLI filtering to find the needle in the haystack. There’s something about the ‘hunt’ that I love. Also, “Network Forensic Analyst” sounds pretty cool! Ultimately, because I don’t do this everyday for a living, I just wanted a way to learn more, and by pursuing a certification with a proven curriculum of study material seemed the best way to learn.

What did I learn?

I learned a bunch of things. There were some tools that I was unaware of such as: silk, nfdump, and the value of proxy logs. Also, through demonstration and exercises I learned some additional filtering and flags for various CLI tools that I needed more practice on.

The Hug of Thunder

When I started this journey I put a question on on twitter to see who (if anyone) is pursuing a SANS certification other than myself. I received only a few replies but one very important was from Andre. He mentioned he was just starting SEC503. We became daily DM buddies, checking in on each others progress and what we’ve learned. If it wasn’t for this encouragement I might have let this one slip. For the record SEC503 and FOR572 overlap a little bit so we had some common ground between us. I wish him the best on his exam and because of all the great things I’ve heard and seen from the course material, I look forward to completing the SEC503 exam this year too!

Can you pass the cert without the training?

I’m not sure I can weigh in on this. Everyone is different. I even asked this question to Phil Hagan via a Slack we have in common. I think no matter what certification you are going for you should always take the tailored training. I know, in the case of SANS, it’s expensive, but so is failing the exam. It’s a gamble. I say TAKE THE TRAINING, you’ll learn something.

GIAC Advisory Board

Because I scored above a 90% I got an invite to the GIAC Advisory Board. It’s a mailing list with threads related to current events, certifications, and general IT and security focused queries.

…on to the next one!

WARNING: Throughout this post I reference “PCAP” and “PCAPng” interchangeably. I also perform operations on files who have the extension *.pcap while their file structure is actually *.pcapng . Everything in this post whether *.pcap or *.pcapng should be *.pcapng. Thank you.

In one of my recent blog posts I discussed how some packet analysis tools have trouble processing a PCAPng file containing more than 1 interface type. Specifically I had a PCAPng file containing:

• LINKTYPE_ETHERNET (1)
• LINKTYPE_LINUX_SLL (113)

See that post here.

At the bottom of that post I left with an indication that this could be done manually but I wasn’t quite sure how. This unanswered question has bothered me. I needed to know “Why?” and “How?”. This led to a path I never thought I’d cross: Manually editing raw hex to manipulate files.

My idea at the time was if we split the packets from the PCAPng file into 2 separate files each containing traffic from only 1 interface type, the packet analysis tools should have no problem processing them individually.

Using tshark I split the protocols into 2 files:

tshark -r <input_file.pcapng> -w eth.pcapng -Y "eth"
tshark -r <input_file.pcapng> -w sll.pcapng -Y "sll"

The above tshark commands will create 2 PCAPng files each containing only like-traffic using the display filter. With wireshark you can verify all the packets are alike within each file. Each tool tested (Zeek, Suricata, Brim & tcpdump) still gives the same error processing these new PCAPs as they did when processing the single PCAP containing both Layer-2 protocols. (see this blog post)

Hypothesis

I’m thinking there are still artifacts in the new PCAPng files referencing the additional interfaces from the original PCAPng file.

…but where?

A Clue

When you open the new PCAPng files (eth.pcapng & sll.pcapng), even though they each contain only 1 type of Layer-2 header, in the Capture File Properties you’ll see 2 types of interfaces listed under the “Link Type” column header.

Ah-ha! This data must still be contained in the file …but where?

Understanding the PCAPng File Structure

To understand what’s going on you’ll need to understand the PCAPng file structure.

To summarize our issue with this specific PCAPng file and condense the information I will NOT step through every header, block field and options, instead I’ll give a high level description. Everything you’ll need to know about the PCAPng file structure is in the documentation.

Below is an abstract image representing the compisition of our PCAPng file, reading from left–>right. The file is broken up into different sections called blocks. Below is a summary of the file blocks we’re interested in.

Block Types

• Section Header Block (SHB): Every PCAPng file must have at least 1 of these blocks but can have more. This does NOT contain packet data. It more like meta data around the file itself like the OS and version of the system the PCAPng file was created.
• Interface Description Block (IDB): This block contains the Linktype values. This is where an interface is identified as LINKTYPE_ETHERNET (1) or LINKTYPE_LINUX_SLL (113). The order of these blocks determines the interface ID value. The first Interface Description Block becomes interface ID: 0. All subsequent Interface Description Blocks not separated by SHB’s are incremented by 1.
• Enhanced Packet Block (EPB): This block contains the actual packet data from Layer-2 to Layer-7 and also indicates which interface ID this packet was captured from.

Our PCAPng file closely resembles the “complex example” figure 6 from the documentation.

Our PCAPng file has 2 IDB blocks and inside each IDB is a different Link Type.

This is the source of our problem.

PCAPng Hex Dump

Looking at the PCAPng file using a hex editor, I’ve outlined the sections using the same colors as the image in the previous section. I’ve also highlighted some values using bright yellow and green. The values highlighted in yellow represent the Link type and the values highlighted in green are the associated Interface ID for each EPB.

NOTE: The above hex dump should be read Little-Endian. Not all captures will be Little-Endian, that is left up to your local operating system.

Using the below image, something worth noting is, some of the info is explicitly contianed in the data (‘included data’) and some of the info is derived meaning it’s not actually reflected in the bytes or bits but is derived based on it’s placement in the file structure. The first IDB block becomes ‘interface id: 0’, the second IDB becomes ‘interface id: 1’, etc …:

In our case, using the example eth.pcapng file:

• All the packets reference interface ID: 1, which is the “LINKTYPE_ETHERNET” as their source interface.
• None of the packets reference interface ID: 0, which is the “LINKTYPE_LINUX_SLL”.

We should be able to change the Link type in the first listed IDB from 0x71 –> 0x01 without casing an issue.

This will indicate to applications reading the PCAPng file that ‘this file was produced on a system that was capturing on 2 interfaces and both were LINKTYPE_ETHERNET’.

My Process

1. Using tshark split out the Layer-2 protocols into separate PCAPng files: eth-before.pcap & sll-before.pcap
2. Manually edit the Interface Description Blocks (IDB) using a hex editor, making both LINKTYPE fields the same, either 0x71 or 0x01, and naming the files eth-after.pcap & sll-after.pcap, respectively.
3. Testing each pcap with: tcpdump, Suricata, Zeek & Brim

Definitions

Before

Each PCAP file contains only (1) Layer-2 protocol in the EPBs, but the SHB contains (2) IDBs with different LINKTYPEs.

After

Each PCAP file contains only (1) Layer-2 protocol in the EPBs and the SHB contains (2) IDBs which are the same LINKTYPE.

Results

After changing the values you can see from this screenshot, Wireshark now thinks both interfaces are the same.

Before

After

Compatibility Matrix

Example Processing the Before & After PCAPs

Zeek: eth-before.pcap & eth-after.pcap

Zeek: sll-before.pcap & sll-after.pcap

tcpdump: eth-before.pcap & eth-after.pcap

Wireshark Side-By-Side

This shows the meta-data around duration and file size the same, with the Hash values changing because of the 1 byte that was modified.

Packet View for Reference

If you wanted to correlate the Hex dump image from earlier in this post against how it’s shown in Wireshark, you would be looking at these 3 packets:

Questions

1. Why didn’t you just delete the unused/unreferenced Interface Description Block (IDB)?

   Because of how the interfaces are ID’d, if all the packets we referencing ID #1 (the second IDB), I would have to re-write all the interface ID references in all the EPBs and point them to interface ID #0. I really just wanted to make as few changes as possible. Leaving an unused ‘ghost IDB’ didn’t seem to hurt anything as long as the LINKTYPE was the same.

2. What hex editor did you use?

   I used the “hex editor” plugin for Notepad++. Since I don’t do this often, I don’t have a favorite tool.

3. How was the original PCAP created containing multiple LINKTYPE’s?

   I’m not sure how the PCAP’s author originally created file but if you use linux and capture packets on a system that has multiple interfaces and use: tcpdump -i any this should generate a PCAP using Linux Cooked Capture (sll) and then again but this time specifying a single interface: tcpdump -i eth0 and merge those files together, you’ll get a mixed IDB in the final PCAP file. NOTE: You may have to convert one or both to PCAPng file format first before merging.

Reference Links

1. https://pcapng.github.io/pcapng/draft-tuexen-opsawg-pcapng.html

Sometimes when loading a PCAP into various tools you get a cryptic error: an interface has a type 1 different from the type of the first interface. I had one PCAP that would generate various errors in different tools.

The Evidence

Brim:

See this Github issue I raised.

Zeek:

root@server:~/ctf$ /opt/zeek/bin/zeek -C -r ctf-dump-v2.pcap
fatal error: failed to read a packet from ctf-dump-v2.pcap: an interface has a type 1 different from the type of the first interface

Suricata:

root@server:~$ suricata -r ctf-dump-v2.pcap -c /etc/suricata/suricata-debian.yaml
10/4/2021 -- 23:10:01 - <Error> - [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -1 an interface has a type 1 different from the type of the first interface

tcpdump:

root@server:~$ tcpdump -r ctf-dump-v2.pcap
reading from file ctf-dump-v2.pcap, link-type LINUX_SLL (Linux cooked)
tcpdump: pcap_loop: an interface has a type 1 different from the type of the first interface

Wireshark & tshark:

Had no problems reading the PCAP.

The Solution

To fix this issue we need to iterate through the PCAP re-writing the SLL frame headers with standard ethernet headers. Tracewrangler makes quick work of this. The more correct solution is for the application developers to impliment compatibility with this interface type or a mix of interface types in the same PCAP file.

Tracewrangler

Tracewrangler is an awesome tool for manipulating PCAP files. It is written by Jasper Bongertz(@packetjay) and has gotten me out of a few jambs. This tool performs an number of common PCAP file manipulation tasks quickly. For example:

• Add or Remove a VLAN ID in the header
• Mask/re-write the src or dst MAC addresses
• Mask/re-write the src or dst IP addresses
• Slice off the payloads leaving only the headers
• ^^^ Just to name a few, and many many more!!!

Linux Cooked Capture Mode

This is “Linux Cooked Capture Mode” and you can find frames using this header type in your PCAP’s using the Wireshark display filter: sll. This header is 16-bytes and as you can see very different from a standard Ethernet header. Some tools actually don’t have a problem processing PCAP’s which contain only SLL, but most will choke when they contain a mix of various Layer-2 headers(in my case a mix of sll && eth). Frames using Ethernet headers are 14-bytes by default and can be isolated using the Wireshark display filter: eth.

NOTE: Alternatively, separate your different layer-2 protocols into seprate PCAPs. Most tools will be okay with this.

Let’s Get To It

1. Open TraceWrangler click the “Add Files” button and load the PCAP in question.
   

2. For this particular operation use the “Edit Files” button.
   

3. In the navigation tree select “Edit”, Check the box for “Replace Linux Cooked Header with Ethernet” and click “Okay”.
   

4. Back on the main TraceWrangler window click “Run” in the bottom left.
   

Alternative Solution

As mentioned above the root of the problem is the mix of interface types/Layer-2 header types in the same PCAP file. If you separate out the interface types you might be able to process the PCAPs separately. I haven’t found a clean way to accomplish this via CLI yet, I’ll update this blog post if I do.

# Note this doesn't solve the problem but it does seperate out the traffic. however the interfaces are both still listed in the capture properties.
# Even though this doesn't work as expected I'm leaving here as a seed of an idea until I have something better.

tshark.exe -r "c:\Users\Tony\Downloads\ctf-dump-v2.pcap" -w "c:\Users\Tony\Downloads\sll.pcap" -Y "sll"

tshark.exe -r "c:\Users\Tony\Downloads\ctf-dump-v2.pcap" -w "c:\Users\Tony\Downloads\eth.pcap" -Y "eth"

Links/References:

• https://wiki.wireshark.org/SLL
• http://www.tcpdump.org/linktypes.html
• https://pcapng.github.io/pcapng/draft-tuexen-opsawg-pcapng.html (Link Type: 1 & 113)
• https://linux.die.net/man/7/pcap-linktype (Link Type: 1 & 113)

Problem Statement:

I need to have Zeek log every UDP packet instead of only per UDP session/conversation

Because of Zeek’s session tracking it will only log one connection “uid” at the start of a session and for all subsequent packets which are part of the same session or conversation.

For example, using the popular smallFlows.pcap you can see there are 501 UDP packets:

# tcpdump

tcpdump -nr smallFlows.pcap udp | wc -l
501

# Wireshark Display Filter:
udp && !icmp
^^^ This count is 501 packets

# NOTE: You have to exclude 'icmp' because the Type-11's and Type-3's re-encapsulate the original UDP packet.
# You can identify these packets using the filter 'udp && icmp'(<-- this count is 22 packets).
# Using just 'udp' filter will result in also counting the icmp ecapsulated udp packets (<-- this count is 523)

Using Zeek on the same PCAP and filtering the logs showing only protocol UDP yields a much lower count:

zeek -C -r smallFlows.pcap
grep udp conn.log | wc -l
173

So, where did all the other packets go? The packets are there but Zeek only creates a new log entry for every unique session. Packets that are part of the same flow or conversation are counted together. In fact if you use the below command and add up all the numbers in the 6th and 7th columns you’ll get 501:

cat conn.log | /opt/zeek/bin/zeek-cut id.orig_h id.orig_p id.resp_h id.resp_p proto orig_pkts resp_pkts uid | grep udp

NOTE: The 6th and 7th columns are the number of packets seen with the Originator as the source or the Responder as the source (on a per-packet analysis)

How do we get Zeek to log each UDP packet instead of each session or conversation?

I’m glad you asked, I wrote a script for that!

This first script simply logs to STDOUT for every UDP request or UDP reply and logs them as individual unique connections.

Script #1: udp_script.zeek

event udp_request(u: connection){
# Choose one format to un-comment
#	print fmt("UDP Request: %s", u$id);
#	print fmt("%s UDP Request: %s %s --> %s %s", u$uid, u$id$orig_h, u$id$orig_p, u$id$resp_h, u$id$resp_p);
	print fmt("UDP Request: %s %s --> %s %s", u$id$orig_h, u$id$orig_p, u$id$resp_h, u$id$resp_p);
}

event udp_reply(u: connection){
# Choose the matching format from above to un-comment
#	print fmt("UDP Reply  : %s", u$id);
#	print fmt("%s UDP Reply  : %s %s <-- %s %s", u$uid, u$id$orig_h, u$id$orig_p, u$id$resp_h, u$id$resp_p);
	print fmt("UDP Reply  : %s %s <-- %s %s", u$id$orig_h, u$id$orig_p, u$id$resp_h, u$id$resp_p);
}

Lets run this script against our PCAP file:

# NOTE: I'm in the same directory as my pcap and my script file. The Zeek binary is in its default location.

sudo /opt/zeek/bin/zeek -C -r smallFlows.pcap udp_script.zeek

UDP Request: 192.168.3.131 57757/udp --> 239.255.255.250 1900/udp
UDP Request: 192.168.3.131 57757/udp --> 239.255.255.250 1900/udp
UDP Request: 192.168.3.131 57757/udp --> 239.255.255.250 1900/udp
UDP Request: 192.168.3.131 57757/udp --> 239.255.255.250 1900/udp
UDP Request: 192.168.3.131 57757/udp --> 239.255.255.250 1900/udp
UDP Request: 192.168.3.131 57757/udp --> 239.255.255.250 1900/udp
UDP Request: 192.168.3.131 68/udp --> 255.255.255.255 67/udp
UDP Request: 192.168.3.131 54600/udp --> 224.0.0.252 5355/udp
UDP Request: 192.168.3.131 54600/udp --> 224.0.0.252 5355/udp
UDP Request: 172.16.255.1 50983/udp --> 71.224.25.112 33695/udp
[ ... OMITTED FOR BREVITY ... ]

Sweet, now lets make sure we have the same number of log lines as we do UDP packets in the PCAP:

sudo /opt/zeek/bin/zeek -C -r smallFlows.pcap udp_script.zeek | wc -l
501

Awsome! While this is great and proof that we are generating logs the way we intended. It isn’t actaully logging anywhere and the results aren’t being picked up by other tools (Filebeat–>Logstash–>Elastic<–Kibana).

Leverage the Zeek Logging Framework

This script will generate a Zeek log file “udp_packets.log” with the columns: Timestamp, UID, Source IP, SRC Port, Destination IP, Dest Port

Script #2: udp_log.zeek

module Udplog;

export {
	redef enum Log::ID += { LOG };

	type Info: record {
		ts: time	&log;
		uid: string	&log;
		id: conn_id	&log;
	};
}

event zeek_init(){
	Log::create_stream(Udplog::LOG, [$columns=Info, $path="udp_packets"]);
}

event udp_request(u: connection){
	local rec: Udplog::Info = [$ts=network_time(), $uid=u$uid, $id=u$id];
	Log::write(Udplog::LOG, rec);
}

event udp_reply(u: connection){
	local rec: Udplog::Info = [$ts=network_time(), $uid=u$uid, $id=u$id];
	Log::write(Udplog::LOG, rec);
}

Running this against the same PCAP, Zeek will create a new log file: udp_packets.log. Zeeks standard log file format includes 8 lines of metadata pre-pended and 1 line appended.

wc -l udp_packets.log
510

# NOTE: Subtracting the 9 lines of metadata leave 501 lines of logs.

Using zeek-cut can clean-up the log for you:

cat udp_packets.log | /opt/zeek/bin/zeek-cut | wc -l
501

^^^ BOOM !

You do not need to run both of these scripts together at the same time. The first script was a proof of concept and prints to STDOUT while the second is a more permanent script to be used during live capture and when ingesting your logs into other tools. You can also send output as JSON if needed.

Thanks for reading.

Process Summary

1. Login to Google Cloud using your existing Google account or create a new account
2. Create a new project
3. Create the nested virtualization Ubuntu VM image
4. Create a new VM instance using the new image
5. Run the installation script
6. Run the setup wizard

1. Log In To Google Cloud

You can use your existing Google or Gmail account or create a new account for google cloud. https://cloud.google.com If this is your first time Google will give you a $300 free credit. This is more than enough to test this process and practice a bunch of labs.

2. Create A New Project

3. Create The New VM Image

Navigate to: Compute Engine –> VM instances and switch to your new Project from the upper left hand drop down.

NOTE: This takes a few minutes.

Once the Compute Engine is ready, click on the “Launch Cloud Shell” icon (in the upper right).

Make sure your Cloud Shell session is set to your newly created project. (It should have your project name in the command prompt.)

Enter the command: (this is 1 long command)

gcloud compute images create nested-virt-ubuntu --source-image-project=ubuntu-os-cloud --source-image-family=ubuntu-1604-lts --licenses="https://www.google.com/compute/v1/projects/vm-options/global/licenses/enable-vmx"

NOTE: It could take a few minutes to return to an interactive prompt. Be patient. You are ready when the shell returns “STATUS READY”:

user@cloudshell:~ (test-305418)$ gcloud compute images create nested-virt-ubuntu --source-image-project=ubuntu-os-cloud --source-image-family=ubuntu-1604-lts --licenses="https://www.google.com/compute/v1/projects/vm-options/global/li
censes/enable-vmx"
Created [https://www.googleapis.com/compute/v1/projects/test-305418/global/images/nested-virt-ubuntu].
NAME                PROJECT      FAMILY  DEPRECATED  STATUS
nested-virt-ubuntu  test-305418                      READY
user@cloudshell:~ (test-305418)$

Now, close the Cloud Shell terminal.

4. Create The New VM instace

Click on the “Create” button in the VM instances frame in the center of the screen.

• Name: Anything you want (cannot change)
• Labels: (optional)
• Region: Choose the region nearest you (cannot change)
• Zone: (cannot change)
• Machine Family: General Purpose
• Series: Must choose “Intel Cascade Lake” or “Intel Skylake”
• Machine Type: This is going to depend on what you need for labbing. You cannot change this unless you power-off your VM. Just to get started I choose “n2-standard-4”.
• Confidential VM service: Unchecked
• Container: Unchecked
• Boot Disk: Choose the image we created from the cloud shell.
  • It will be under “Custom images”:
  • Show images from: Your new project name
    • Image: the image name we created earlier
    • Boot Disk Type: SSD
    • Size: This is going to be determined by you on how you intend to use the VM for labbing. To get started I select 50GB.
• Identity and API access: leave default
• Firewall: Choose “Allow HTTP traffic”

Click “Create” at the bottom.

Wait for your VM to be provisioned, this could a minute or two.

5. Run The Installation Script

Once your VM is provisioned it will automatically be started. Use the built-in SSH feature to connect to the VM’s console.

Once you’re SSH’d into the VM become root, grab the installation script, update the package manager and upgrade current packages.

sudo -i

wget -O - http://www.eve-ng.net/repo/install-eve.sh | bash -i

apt update

apt upgrade

Reboot the VM.

6. Run The Setup Wizard

Connect to it again via the built in SSH.

You will be presented with a configuration wizard.

STOP!

When you are greeted with the wizard to enter a root password, don’t!

Hold CTRL and press C, become root sudo -i

This will restart the wizard and allow you to change root’s password.

Follow the initial configuration wizard.

• Enter Root Password:
• Enter Root Password Again:
• Hostname: anything you want (I leave default)
• DNS Domain Name: anything you want (I leave default)
• DHCP/Static IP: Choose DHCP/Static
• NTP: (leave empty)
• Proxy: Choose “Direct connection”

After hitting enter, the setup wizard will kick you out.

7. You’re Finished / What’s Next?

At this point the installation is finished. You should have a working EVE-NG server in Google Cloud.

Next, you should:

• Start uploading the images you need for labbing
• Start importing your already created labs
• Start thinking securing the ingress and egress traffic of your EVE-NG lab.
• Check out my other blogs and videos for more EVE-NG: Tips & Tricks

How do I give internet access to my running images?

Ever since I released my video How to run EVE-NG in Google Cloud in 2018, I have been inundated with this question and others. I finally put it all together for you and introduce some new ideas in my video series.

Skill Level Required: Medium

If you already know what to do with these commands then go ahead and get started.

If you don’t know what to do, I created a follow along video, along with other EVE-NG: Tips & Tricks:

Add an IP address to the 2nd bridge interface:

sudo -i

root@eve-ng:~# nano /etc/network/interfaces

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
iface eth0 inet manual
auto pnet0
iface pnet0 inet dhcp
    bridge_ports eth0
    bridge_stp off

# Cloud devices
iface eth1 inet manual
auto pnet1
iface pnet1 inet static     # <-- Change this to static
    bridge_ports eth1
    bridge_stp off
    address 10.199.199.1    # <-- Create an address
    netmask 255.255.255.0   # <-- Create a subnet mask
    
[ ... omitted for brevity ...]

Restart Networking:

root@eve-ng:~# systemctl restart networking

Turn on IPv4 Forwarding:

Edit /etc/sysctl.conf as root and uncomment:

# net.ipv4.ip_forward=1

so that it reads:

net.ipv4.ip_forward=1

Force sysctl to read the new values from the file:

sudo -i
sysctl -p /etc/sysctl.conf

Configure iptables to NAT outbound connections:

sudo -i
iptables -t nat -A POSTROUTING -s 10.199.199.0/24 -o pnet0 -j MASQUERADE

# The source "-s ##.###.###.#/##" needs to match the subnet you configured the "pnet1" interface with from above.

Make iptables changes persistent:

Save the current iptables rules (including the new rule we just added) and create a new file for our script:

sudo -i
root@eve-ng:~# iptables-save > /etc/iptables.rules
root@eve-ng:~# nano /etc/network/if-pre-up.d/iptables

Enter this content:

#!/bin/sh
iptables-restore < /etc/iptables.rules
exit 0

Save changes then edit/create next “iptables” file:

root@eve-ng:~# nano /etc/network/if-post-down.d/iptables

Enter this content:

#!/bin/sh
iptables-save -c > /etc/iptables.rules
if [ -f /etc/iptables.rules ]; then
    iptables-restore < /etc/iptables.rules
fi
exit 0

Save changes. Make both files executable:

sudo chmod +x /etc/network/if-post-down.d/iptables
sudo chmod +x /etc/network/if-pre-up.d/iptables

Troubleshooting

The Proper Process

I have had great success with this process:

1. Making the above configuration changes to the VM.
2. Adding the Cloud 1 network to your topology.
3. Adding any device images.
4. Connecting their MGMT interfaces to Cloud 1.
5. Power on the device images.
6. Configure their MGMT IP address, subnet mask and Default Gateway.
7. Ping the Default Gateway
8. Ping an external network (8.8.8.8)

Your first test should always verify you can ping your default gateway(pnet1 interface) from your lab images. If you cannot ping your default gateway try these steps:

• Shutdown your lab images conencted to Cloud 1 and restart them. Try to ping the Default GW.
• Delete your Cloud 1 from your topology, save and exit your topology. Launch your topology again and drop in the Cloud 1 network again, reconnect it to the device images mgmt interfaces and start the device images.