I spent a bit of the day troubleshooting a community safety monitoring (NSM) drawback. I assumed I might share the issue and my investigation within the hopes that it would assist others. The specifics are most likely much less essential than the final method.

It started with ja3. It’s possible you’ll know ja3 as a set of Zeek scripts developed by the Salesforce engineering staff to profile consumer and server TLS parameters.

I used to be reviewing Zeek logs captured by my Corelight equipment and by one in every of my lab sensors operating Safety Onion. I had protection of the identical endpoint in each sensors.

I seen that the SO Zeek logs didn’t have ja3 hashes within the ssl.log entries. Each sensors did have ja3s hashes. My first thought was that SO was misconfigured one way or the other to not report ja3 hashes. I rapidly dismissed that, as a result of it made no sense. In addition to, verifying that intution required me to begin troubleshooting close to the highest of the software program stack.

I made a decision to begin on the backside, or near the underside. I had a sinking suspicion that, for some purpose, Zeek was solely seeing visitors despatched from distant methods, and never visitors originating from my community. That will account for the creation of ja3s hashes, for visitors despatched by distant methods, however not ja3 hashes, as Zeek was not seeing visitors despatched by native purchasers.

I used to be operating SO in VirtualBox 6.0.four on Ubuntu 18.04. I began sniffing TCP community visitors on the SO monitoring interface utilizing Tcpdump. As I feared, it did not look proper. I ran a brand new seize with filters for ICMP and a distant IP handle. On one other system I attempted pinging the distant IP handle. Certain sufficient, I solely noticed ICMP echo replies, and no ICMP echoes. Oddly, I additionally noticed doubles and triples of among the ICMP echo replies. That frightened me, as a result of unpredictable conduct like that would point out some form of software program drawback.

My subsequent step was to “get beneath” the VM visitor and decide if the VM host might see visitors correctly. I ran Tcpdump on the Ubuntu 18.04 host on the monitoring interface and repeated my ICMP assessments. It noticed all the pieces correctly. That meant I didn’t must hassle checking the change span port that was feeding visitors to the VirtualBox system.

It appeared I had an issue someplace between the VM host and visitor. On the identical VM host I used to be additionally operating an occasion of RockNSM. I ran my ICMP assessments on the RockNSM VM and, sadly, I acquired the identical one-sided visitors as seen on SO.

Now I used to be frightened. If the issue had solely been current in SO, then I might repair SO. If the issue is current in each SO and RockNSM, then the issue needed to be with VirtualBox — and I won’t be capable of repair it.

I reviewed my configurations in VirtualBox, making certain that the “Promiscuous Mode” beneath the Superior choices was set to “Enable All”. At this level I frightened that there was a bug in VirtualBox. I did some Google searches and reviewed some discussion board posts, however I didn’t see anybody reporting points with sniffing visitors inside VMs. Nonetheless, my use case may need been bizarre sufficient to not have been reported.

I made a decision to attempt a special method. I puzzled if operating VirtualBox with elevated privileges would possibly make a distinction. I didn’t wish to take possession of my consumer VMs, so I made a decision to put in a brand new VM and run it with elevated privileges.

Let me cease right here to notice that I’m breaking one of many guidelines of troubleshooting. I am introducing two new variables, after I ought to have launched just one. I ought to have constructed a brand new VM however run it with the identical consumer privileges with which I used to be operating the prevailing VMs.

I made a decision to put in a minimal version of Ubuntu 9, with VirtualBox operating through sudo. Once I began the VM and sniffed visitors on the monitoring port, lo and behold, my ICMP assessments revealed either side of the visitors as I had hoped. Sadly, from this I erroneously concluded that operating VirtualBox with elevated privileges was the reply to my issues.

I took possession of the SO VM in my elevated VirtualBox session, began it, and carried out my ICMP assessments. Womp womp. Nonetheless damaged.

I spotted I wanted to separate the 2 variables that I had entangled, so I finished VirtualBox, and adjusted possession of the Debian 9 VM to my consumer account. I then ran VirtualBox with consumer privileges, began the Debian 9 VM, and ran my ICMP assessments. Success once more! Apparently elevated privileges had nothing to do with my drawback.

By now I used to be glad I had not posted something to any consumer boards describing my drawback and asking for assist. There was one thing concerning the monitoring interface configurations in each SO and RockNSM that resulted within the incapability to see either side of visitors (and keep away from bizarre doubles and triples).

I began my SO VM once more and seemed on the script that configured the interfaces. I commented out all of the entries under the administration interface as proven under.

$ cat /and so on/community/interfaces

# This configuration was created by the Safety Onion setup script.
# The unique community interface configuration file was backed as much as:
# /and so on/community/interfaces.bak.
# This file describes the community interfaces accessible in your system
# and easy methods to activate them. For extra info, see interfaces(5).

# loopback community interface
auto lo
iface lo inet loopback

# Administration community interface
auto enp0s3
iface enp0s3 inet static
dns-domain localdomain

#auto enp0s8
#iface enp0s8 inet guide
#  up ip hyperlink set $IFACE promisc on arp off up
#  down ip hyperlink set $IFACE promisc off down
#  post-up ethtool -G $IFACE rx 4096; for i in rx tx sg tso ufo gso gro lro; do ethtool -Ok $IFACE $i off; completed
#  post-up echo 1 > /proc/sys/internet/ipv6/conf/$IFACE/disable_ipv6

#auto enp0s9
#iface enp0s9 inet guide
#  up ip hyperlink set $IFACE promisc on arp off up
#  down ip hyperlink set $IFACE promisc off down
#  post-up ethtool -G $IFACE rx 4096; for i in rx tx sg tso ufo gso gro lro; do ethtool -Ok $IFACE $i off; completed
#  post-up echo 1 > /proc/sys/internet/ipv6/conf/$IFACE/disable_ipv6

I rebooted the system and introduced the enp0s8 interface up manually utilizing this command:

$ sudo ip hyperlink set enp0s8 promisc on arp off up

Fingers crossed, I ran my ICMP sniffing assessments, and voila, I noticed what I wanted — visitors in each instructions, with out doubles or triples no much less.

So, there seems to be some form of drawback with the best way SO and RockNSM set parameters for his or her monitoring interfaces, at the very least so far as they work together with VirtualBox 6.0.four on Ubuntu 18.04. You may see within the community script that SO disables a bunch of NIC choices. I think about a number of of them is the perpetrator, however I did not have time to work via them individually.

I attempted looking on the community script in RockNSM, but it surely runs CentOS, and I will be darned if I can not determine the place to look. I am positive it is there someplace, however I did not have the time to determine the place.

The ethical of the story is that I ought to have instantly checked after set up that each SO and RockNSM had been seeing either side of the visitors I anticipated them to see. I had taken that without any consideration for a lot of earlier deployments, however one thing broke not too long ago and I do not know precisely what. My workaround will hopefully maintain for now, however I must take a more in-depth have a look at the NIC choices as a result of I could have launched one other fault.

A second ethical is to watch out of adjusting two or extra variables when troubleshooting. If you do that you just would possibly repair an issue, however not know what change mounted the problem.

oracle virtualbox user manual,oracle virtualbox features,virtualbox 6.1 user manual pdf,90 oracle virtualbox,virtualbox dashboard,how to create vm in virtualbox,what is oracle virtualbox,how to use virtualbox