IDS - Evasion Techniques using URL Encoding

[Resheph]

Below is a description of encoding methods supported by Microsoft Internet Information Services (IIS). They are not ment to be utilized together but as Microsoft IIS allows for two passes when decoding it is actually possible to encode an URL that has already been encoded once. It is also possible to encode different pieces of the URL with different encoding methods.
It is a very nice feature that IIS supports this many encoding methods and that more than one method can be used in the same URL. IIS is very user friendly however that may make defending the site slightly harder. The defending party has to forsee the complete attack vector to keep a determined attacker at bay.
Needlesl to say - encoding can be used in many other situations such as evading input validation and filtering.

HEX Encoding
The HEX encoding method is one of the RFC compliant ways for encoding an URL.
It consists of escaping a hexadecimal byte value for the encoded character with a %.

%41 = A

Double Percent HEX Encoding
The Double percent HEX encoding is based on the normal method of HEX encoding.
The percent is encoded using HEX encoding followed by the hexadecimal byte to encode.

%2541 = A

The percent is encoded with the %25 which equals a %.
The value is then decoded again with the value %41 which equals A.

Double Nibble HEX Encoding
The Double Nibble HEX encoding is based on the normal method of HEX encoding.
Each hexadecimal nibble (half-byte, tetrade or four bits) is encoded using the standard HEX encoding.

%%34%31 = A

The normal HEX encoding for A is %41.
In Double Nibble HEX Encoding the hexadecimal nibble values are each encoded in the normal HEX encoding format
So the first nibble, 4, is encoded as %34 (ASCII 4) and the second nibble, 1, is encoded as %31 (ASCII 1).
This will result in two decodings. The first one for the two nibbles, the second one for %41.

%%34%31 -> %41 -> A

First Nibble HEX Encoding
First Nibble HEX encoding is very simillar to Double Nibble HEX Encoding.
Just, only the first nibble, 4, is encoded seperately.

%%341 = A

This encoding method will be decoded in two passes. First %34 will be decoded, then the resulting %41 will be decoded.

%%341 -> %41 -> A

Second Nibble HEX Encoding
Second Nibble HEX encoding is the same as First Nibble HEX Encoding except here the second nibble will be decoded seperately instead of the first.

%4%31 = A

The %31 gets decoded to the numeral 1 (ASCII 1) in the first decoding pass, resulting in %41 that will then be decoded in the second pass.

%4%31 -> %41 -> A

UTF-8 Encoding
UTF-8 encoding allows values larger than a single byte (0-255) to be represented in a bytestream.
HTTP web servers uses UTF-8 encoding to represent Unicode code points that are outside of the ASCII code point range (1-127).
Unicode code point values are usually in the range of 0-65535 (two bytes).
Any code point value above 127 uses UTF-8 encoding in a HTTP URL.
The remaining 65408 code points are in place to represent characters in languages such as Hungarian or Japanese.
UTF-8 works by assigning a special meaning to the high-bits in a byte.

A UTF-8 two byte sequence looks like this

110xxxxx 10xxxxxx

A three byte UTF-8 sequence looks like this

1110xxxx 10xxxxxx 10xxxxxx

The first byte in a UTF-8 sequence is the most important one because it contains the amount of bytes in the complete sequence.
This is determined by counting the amount of high bits that are represented as 1 before the first 0 bit is located.
For a two byte UTF-8 sequence (110xxxxx 10xxxxxx) example the first byte has two high bits set to 1, then a 0 bit.
This indicates that this sequence is a two byte UTF-8 sequence.
The rest of the bits after the first encountered 0 bit in the first byte are bits in the final value to be decoded.
The same format is used for the second byte in the two byte sequence.
The first two 1 bits in the second byte (11xxxxxx) are in place to identify a UTF-8 byte and the following six bits are used for value to be decoded.
When using UTF-8 as URL encoding each byte is escaped with a % sign.

%C1%81 = A (11000001 10000001 = 10000001) Two byte sequence
%E0%81%81 = A (11100000 10000001 10000001) Three byte UTF-8 sequence.

UTF-8 Bare Byte Encoding
UTF-8 Bare Byte Encoding is the same as traditional UTF-8 encoding except that the UTF-8 byte sequence is not escaped with a %.
The byte sequence is sent with the actual raw byte.

0xC1 0x81 = A

Microsoft %U Encoding
Microsoft %U Encoding presents a different way to encode Unicode code point values up to 65535, two bytes.
%U is a prefix followed by 4 HEX nibble values that represent the Unicode code point value.

%U0041 = A

Putting it together
Since IIS will do a double decode on an encoded URL we can use more than one encoding in the same URL.
If we want to encode A with, say, with the Microsoft %U encoding (%U0041) and make it less readable we can encode the U part of %U with HEX encoding.

A -> %U0041 -> %%550041 (%55 is the U)

We can then encode 0041 with HEX encoding or one of the other encoding methods. This is where the attacker has the option to be creative.

Request Pipelines
Request pipeline evasion is a type of invalid protocol parsing evasion.
It obfuscates the URI by using protocol characteristics of a request pipeline in HTTPv1.1.
The request pipeline standard allows a web client to send several requests within a single packet.
This may opens the posibillity to evade some older IDS systems because they may only inspect the first request in the packet.

A payload using request pipelining

GET / HTTP/1.1\r\nHost: \r\n\r\nGET /foobar.html \r\nHost: \r\n\r\nGET /cgi%2Dbin%2Fph%66 HTTP/1.1\r\nHost: \r\n\r\n

The request contains the following three requests

GET / HTTP/1.1
Host:

GET /foobar.html
Host:

GET /cgi%2Dbin%2Fph%66 HTTP/1.1
Host:

Source: SOURCEfire HTTP IDS Evasions Revisited.
https://snort-org-site.s3.amazonaws....S_evasions.pdf