Using Deep Packet Analytics to Extract Specific Bytes

Why Extract Specific Bytes Out of a Packet?

Pulling specific bytes out of a packet is the best way to get to the real truth of the content. Getting to this level of the content can help you in many use cases.

One of the hidden features of NetMon’s DPA language is that you can extract specific bytes out of a packet inside of a packet rule. Although NetMon classifies over 3,100 applications and extracts many thousands of metadata fields, there is always more to learn about network traffic. You may need to dive into specific byte level analysis if you need to:

  1. Identify traffic that is not natively classified
  2. Extract details not normally extracted
  3. Look for signatures in data payloads for specific kinds of traffic

For example, many malware infiltration and C2 channels use well-known protocols in unusual ways or generate custom protocols over base channels. (Gh0st uses a custom protocol on TCP<link.

Many of these communication channels are actually in clear text. Being able to passively extract the traffic gives you a unique defensive advantage and the capability to closely examine the details of the traffic. Whether you are exploring a specific targeted threat or trying to create more general alarms for TTPs, looking at the raw bytes is often the only way to go. For instance, you could end up discovering which commands were issued through a reverse shell (making it easier to identify, isolate and remediate a threat). You might also create new alarms based on unique signatures of threats that you don’t have other means of catching.

Extracting Specific Bytes Out of a Network Packet Using DPA

We haven’t publicly exposed this method call before because there’s risk associated with any packet rule impacting performance. With our Rule Your Network contest running, we’ve decided it’s time to share the proper techniques for extracting specific bytes out of a network packet using DPA.

This method call is pretty simple, and it allows you to pull out up to 4 bytes cast as a long integer:

bytes = GetPacketBytes(packet, startbyte, endbyte)

It’s one line of code. How complicated can it be?

Although we’re really only talking about a single DPA method, this approach is complicated by several factors that require a specific use pattern:

  1. A packet rule executes on every packet: Your rule is likely to run hundreds or millions of times per second. Efficiency matters. Memory use matters. Bad logic can kill your system performance.
  2. A packet can be encapsulated with VLAN headers: Your rule has to account for VLAN wrappers to determine which byte or bytes to extract.
  3. Bytes are cast as long integers: You have to work with integer representation of the bytes or use bit masking flags to extract specific bits. Lua has a library for this, but it is extra work.

The Code Pattern

The high-level logic for a packet rule extracting bytes should be:

  1. Exit the rule as fast as possible.
    • Filter on whether the packet is classified.
    • Filter on protocol.
    • Filter on any other clear include/exclude criteria.
    • Filter if you’ve already processed critical information.
  2. Determine if there’s a VLAN offset.
  3. Extract the desired byte(s), adjusting with the VLAN offset.
  4. Process the bytes per your logic.
    • Analyze the byte as an integer or use bit masking.
    • Create custom metadata.
    • Vote to capture.

In addition, if you are converting byte flags to text strings, make sure you properly declare your lookup table as a global variable.

Exiting the Rule

The most efficient packet rule is one that never executes. Although it may feel strange in terms of logic, the recommended design pattern is to start the rule by applying “leave now” filters, such as:

  • Exit the rule if basic classification hasn’t yet occurred. This works well if you know the protocol that you are expecting and if NetMon classifies that protocol.
  -- only look at classified packets
      if not GetFlowClassified(dpiMsg) then
         return false
      end
  
  • Exit the rule if the protocol doesn’t match what you expect. This will prevent logic from running on packets that won’t have expected data or format. This approach works well with the first exclusion. However, be careful if you are processing a “nested” protocol like “tcp.” You may need to change the logic to look for tcp in the ApplicationPath rather than as the latest application. In this example, ICMP is a “terminal” protocol, so GetLatestApplication works.
  -- only look at ICMP
      if (GetLatestApplication(dpiMsg) ~= "icmp") then
         return false
      end
  

However, GetLatestApplication forces a string comparison, which takes more compute cycles than a numeric comparison. With a quick check on a NetMon dashboard, you can find the application ID for ICMP. Open the Analyze dashboard and filter for application:ICMP. Then, open any session and look for the “ApplicationID” field (967 for ICMP). You can also find this information in the help by looking for “application codes.” With this tiny bit of legwork, you can make this line more efficient by changing it to:

  -- only look at ICMP
      if (GetInt(dpiMsg, 'internal', applicationid) ~= 967) then
         return false
      end
  
  • Exit the rule if you’ve already processed a packet with the critical information. This prevents processing all the downstream packets in a flow. In this example, we’re looking for a particular custom metadata field that our rule adds. Note that GetCustomField returns a table, and we are looking to see if the “entry” in the table for our value is empty. Check out this link for more on the syntax of using tables.
  -- ignore packets if we've already filled in the command
      if (next(GetCustomField(dpiMsg, "ICMP_Type")) ~= nil) then
         return false
      end
  

Depending on your rule logic, some other possible exit filters could be:

  • Check source or destination IP address (or range). You may only want to process data coming from or going to certain specific addresses.
  • Check source or destination port. You may only want to examine traffic on port 80.
  • Check specific metadata. For multi-packet sessions, you can check for and act on additional metadata (e.g., SSL certificate name) once it has been parsed.

Determine if There is a VLAN Offset

The IEEE 802.1Q networking standard defines a method for enabling virtual local area networks over Ethernet. Although this is a wonderful standard and is used on nearly every network, it has one unfortunate side effect for our purposes.

Depending on the type of VLAN encapsulation, the packet we process may have 0, 4, or 8 extra bytes of “VLAN wrapper.” To get the correct offset to account for the VLAN, you have to check to see if the packet was wrapped in a single VLAN layer or “double tagged” with a service tag (ISP VLAN).

Fortunately, checking is as simple as pulling a different byte from the packet and comparing the value. Remember, you’re only looking to see if a VLAN wrapper is present. You don’t actually need to unpack the VLAN header or extract any data from it.

The logic for evaluating VLAN tagging involves looking at bytes 13 and 14:

  • Values of 0x8100, 0x9100, 0x9200, 0x9300, or 0x88A8 (in hex) indicate a VLAN tag. The offset is at least 4 bytes.
  • A value of 0X88A8 is reserved for an ISP VLAN s-tag. Digging into bytes 17 and 18, you will once again check of 0x8100 to confirm the nested VLAN tag. If you see 0x8100, then the offset needs to be 8 bytes.
  • You can assume that the value is 0x0000 (no VLAN) otherwise.

The code looks like this (written as a function you can copy and paste and then call in a DPA rule):

-- Determine if VLAN offset exists.
   --Return the number of bytes in the VLAN offset for the packet.
   function getVlanOffset(packet)
      local offset = 0
      local hex = GetPacketBytes(packet, 13, 14)
      if (hex == 0x8100 or hex == 0x9100 or hex == 0x9200
          or hex == 0x9300 or hex == 0x88A8) then
         offset = 4
         hex = GetPacketBytes(packet, 17, 18)
         if(hex == 0x8100) then
            offset = 8
         end
      end
      return offset
   end

Extract the Desired Bytes with VLAN Offset

Now we are finally ready to call our “one line” of code to extract the packet. We’ve already used the method in determining the VLAN offset:

local vlanOffset = getVlanOffset(packet)
bytes = GetPacketBytes(packet, startbyte+ vlanOffset,
                            endbyte+ vlanOffset)

Remember: You can extract up to 4 bytes at a time. If you try to extract more, you will generate a runtime error that will disable the rule.

Process the Bytes

The return value is an integer. Depending on what you need to do, you can evaluate the result as an integer, using hex notation, or using bit masks.

  • As an integer:
    • Compare the result to a number
    if (bytes == 33024) then…
    
  • As hex:
    • Compare using hex notation (as per VLAN check)
    if (bytes == 0x8100) then…
    
  • As bits:
    -- check if bit 2 is on
local bit = require("bit")
if (bit.band(bytes,0x02) == 0x02) then...
    

Wrapping Up

Once you get your answer, you can wrap up your rule by either voting to capture (or not), or by adding custom metadata.

SetCustomField(dpiMsg, "ICMP_Type", icmpType)

Monitor Performance

Just remember that a packet rule has to be high performance. The more packets you have, the more times the rule runs. Any rule, no matter how simple, will have an impact on your Network Monitor.

Do not develop these rules in your production environment and make sure you monitor them closely when you enable them to prevent runaway resource usage. To monitor the performance of the rule, enable the rule and then look at the following diagnostic charts:

  • Deep Packet Analytics –> Memory Stats for Deep Script Packet Rules: Look for high-sustained lines. If you see excessive memory usage, try to reduce the use of global variables or memory in your rules.
  • Deep Packet Analytics –> Engine Threads: Note how much additional CPU the threads are using after the rule is enabled. If you see threads pegged at 100 or holding steady in the high 90s, then try to reduce the computations in your rule.
  • Engine –> Packet Processing Statistics: If you see dropped packets with your rule enabled, the rule is consuming enough resources to prevent high-speed processing.
  • Engine –> Engine Memory Heap: If you see the engine going into swap space after starting the rule, you may be using too much memory. Look for ways to reduce the use of global variables.

Conclusion

Look for documentation on the GetPacketBytes method to be made public in a future release of NetMon. In the meantime, feel free to follow the guidance here.