Mitigation of L2FB (Layer 2 Fallback/Intrinsic HA) Events

Overview

This article addresses the mitigation of Layer 2 Fallback (L2FB) events occurring in TippingPoint TPS devices. It outlines symptoms, root causes, and detailed mitigation steps to resolve the issue.

Symptoms

When the TippingPoint TX-Series IPS enters an L2FB state, users may experience the following symptoms:

• The IPS intermittently enters YELLOW and RED thresholds for license utilization.
• Threshold errors may be logged, indicating that packet loss has exceeded acceptable limits.

Root Cause

The primary reasons for entering L2FB include:

• Excessive Packet Loss: If packet loss exceeds 90%, the IPS may enter L2FB.
• High Traffic Volume: An unexpected surge in traffic may lead to resource exhaustion.
• Configuration Issues: Suboptimal or misconfigured device settings can introduce latency and increase the likelihood of packet loss.

Inspection throughput is based on the traffic seen at the switch component entering the inspection engine, aggregated across all inspection segments (excluding management port traffic). To check current throughputs and high-watermarks, use the CLI command show np tier-stats. If throughput consistently exceeds the inspection license capacity, the device may enter fallback to preserve network connectivity and performance. The default behavior of L2FB allows all traffic to pass uninspected, but this can be configured to block all traffic if desired. Note that real-time inspection throughput will show as 0.0 while in fallback.

Mitigation Steps

Step 1: Verify Traffic

1. Take the device out of fallback.
2. Run the CLI command show np tier-stats. Verify the real-time throughput and high watermarks for Tier 1 (at switch component).
3. If the statistics look unexpected, clear them with clear np tier-stats.
4. After clearing, place the device inline and run show np tier-stats multiple times to observe trends and stabilization of the high watermark.

Step 2: Reboot the Device

1. Access the command line interface (CLI) of the TippingPoint system.
2. Execute the CLI command reboot full to perform a graceful power cycle, including hardware reinitialization and software restart.
3. Monitor the system after rebooting to see if the issue persists. This will help clear any issues related to memory leaks, process crashes, etc.

Step 3a: Reduce Traffic Flow into the Inspection Engine

• If the device regularly enters fallback due to excessive inspection throughput, consider reducing the amount of traffic being inspected.
• Add inspection bypass rules for trusted internal traffic (e.g., backups to a secure enclave).
• Configure Traffic Management Filters or Flow Management Filters to reduce load.
• Ensure that traffic is not being double-inspected due to misconfiguration.

Step 3b: Purchase and/or Attach a Higher Capacity Inspection Throughput License

• If possible, purchase a higher capacity inspection throughput license if not already at maximum.
• For TippingPoint TPS, you can detach/attach licenses via the TippingPoint License Manager (TMC).
• Log into TMC to manage the attached license for your device(s).

Step 3c: Upgrade the Device Model

• If you are already at maximum capacity and cannot reduce traffic, consider purchasing a TippingPoint device with higher inspection capability or setting up a stack with an additional device of the same model.

References

• What is a Traffic Management Filter?
• What are Inspection Bypass Rules?
• What are Flow Management Filters?
• TippingPoint Documentation Page