While the protocol decoding itself requires no configuration by the Administrator, understanding that the decoding done by this rule is essential for other DPI rules to function. You may notice that when you assign other DPI rules to protect your web server itself (i.e. Web Server IIS or Web Server Apache rules) that Deep Security will inform you that this rule is a required dependency. This is basically informing you that proper decoding of the HTTP protocol is required for that rule to be able to protect your system. Also, if you are using the Recommendation Engine to assign your DPI rules, you will notice that this rule is recommended on all systems where a web server is detected.
The URI normalization control portion of the HTTP Protocol Decoder is the tricky part that requires the most tuning. The easiest way to understand the URI normalization performed by the DPI engine is to describe the configuration options available in this rule.
This setting instructs the DPI engine to apply the same URI Normalization function that it would on the URI parameters of a GET request, to an HTTP POST body. The default setting for this option is enabled and does not typically cause false positives for most web applications.
This setting instructs the DPI engine to assume that the HTTP POST body is URI form encoded (application/form-urlencoded) in the instance where a Content-Type HTTP header has not been included in the HTTP request. The default setting for this option is enabled and does not typically cause false positives for most web applications.
This setting simply tells Deep Security to log the full packet data information when generating an event due to URI Normalization. For example, with this setting enabled, when double encoding was detected in an HTTP request, Deep Security would log the packet in which this was detected and attach it to the DPI Event for analysis. The default setting for this option is enabled due to the fact that it can be very challenging to investigate potential false positives in URI Normalization with this option disabled. Only when you are confident that you have properly tuned your URI Normalization options for you web application should you disable this setting. The only downside to leaving this option enabled is that there will be a slight increase (1500 bytes maximum) in the amount of space required in the database to log a URI Normalization event.
This setting allows an Administrator to disable URI Normalization on the HTTP POST body for specific URLs.
This setting allows you to choose which encoding type is in use on your web server. The choices are Latin-1 (i.e. %E8) or UTF-8 (i.e. %C3%AD). The default option for this is UTF-8. There is a possibility that your web server will use Latin-1 encoding. If you see many events with the reason “Incomplete UTF8 Sequence” and the encoding type appears to be a valid percent encoding (i.e. %E6) then you should switch your encoding type to be Latin-1 for this Computer or Security Profile.
Percent encoding is used to encode certain reserved characters in a URI. An example of percent encoding is when the reserved character ‘[’ is encoded as %5B. As some IDS/IPS devices had signatures to detect these percent-encoded characters, attackers discovered they could evade detection by double encoding the character. This was done by percent encoding the ‘%’ character, resulting in %25, in addition to the character itself. If you take a look at our previous example of encoding the ‘[’ character, when double-encoded the encoded value becomes %255B. In order to prevent this evasion attempt, Deep Security will block double-encoded data when this option is enable. The default setting for this option is enabled. Some web client browsers and applications will occasionally double-encode legitimate data, causing false positives with your web application. If you see numerous “Double Decoding Exploit” DPI Events in your logs, it could indicate false positives with this setting. You should examine the data portion of the DPI Events to determine if the data looks like an attempt to evade detection or if it is simply a web client that is sending double-encoded data. If the latter is true, you may need to disable this configuration setting for the computer hosting this application.
When percent encoding is used, it is in the form of a percent sign followed by two hex bytes that represent the character. When enabled, this setting will block any data this doesn’t appear to be correctly encoded. For example, if the string %2x is identified in the stream being inspected it would be blocked and an “Invalid Hex Encoding” event would be generated as 2x are not two valid hex bytes. If the web client browser or application intended to send the string %2x it would have been sent properly encoded as %252x. The default setting for this option is enabled and doesn’t typically cause false positives with most web applications.
HTTP Protocol Decoding has, by far, the most challenging set of configurable options for any rule within Deep Security. So if you do not plan on using Deep Security to protect your web applications, but require it as a dependency for rules that protect your web server, we recommend that you modify your HTTP Protocol Decoding configuration in this way:
- Disable the Use URI Normalization in body of HTTP POST option.
- Disable the Block double URI encoded data option.
It is important to note that making these changes will reduce the protection provided by the product but only in the area of web application protection. The HTTP protocol will still be decoded properly and other vulnerability and exploit rules specific to your web server will function normally.
We have moved configuration option Use a custom list of characters disallowed in a URI to a new DPI rule 1008646 - Detect Illegal Characters In URI.
