Why do I need a deny rule to make an allow rule for a custom protocol work correctly?

Protocol definitions include lists of primary connections, secondary connections, and associated application filters. Each primary connection includes a port range, which may cover one or more port numbers. When traffic is sent to an ISA Server computer, ISA Server uses the port on which it arrives to identify its protocol.

When a policy rule allows traffic for a certain protocol, the Firewall service checks the definition of the protocol and passes the traffic to all the application filters associated with the protocol definition for processing.

If traffic of a specific type is sent to a port corresponding to a predefined protocol that is associated with an application filter and you do not want the application filter to process this traffic, you can define a custom protocol that has the same primary and secondary connections as the predefined protocol but is not associated with the application filter. Then you can use your custom protocol in a policy rule that allows this traffic. Protocol definitions with primary connections that include the same port are called overlapped protocol definitions.

As an example, let’s say that you have an internal server to which VPN clients send nonstandard HTTP traffic through TCP port 80. When this traffic is allowed by a matching access rule that uses the predefined HTTP protocol and is configured to allow traffic from the VPN Clients network to the Internal network, the Web Proxy Filter is invoked and rejects this traffic because it does not comply with HTTP standards. To allow nonstandard HTTP traffic to reach your nonstandard HTTP server, you can create a custom protocol definition that has a primary connection for outbound traffic through TCP port 80 and is not associated with the Web Proxy Filter. We will refer to this protocol as the CustomHTTP protocol.

Note: The steps provided here may not allow client access to some nonstandard HTTP servers. This usually happens when ISA Server rejects responses from the HTTP server because the behavior of the HTTP server is not RFC-compliant. Analyzing the traffic between the ISA Server computer and the HTTP server should help to reveal the underlying cause.

To allow the nonstandard HTTP traffic, you need to create two access rules:

· An access rule that uses the CustomHTTP protocol and allows traffic from the VPN Clients network to the computer object representing the nonstandard HTTP server.

· An access rule that uses the predefined HTTP protocol and denies traffic from the VPN Clients network to the computer object representing the nonstandard HTTP server.

The new allow rule must come before your original rule that allows HTTP traffic from the VPN Clients network to the Internal network in the ordered list of policy rules, and the new deny rule should be placed immediately after the new allow rule.

The following table lists the rules that you should have to enable traffic in this scenario:

So why do I need the new deny rule (the second rule)? The short answer is that this rule is needed to prevent the third rule or any other rule from invoking the Web Proxy Filter for traffic that matches the first rule.

To understand why this rule is needed, you need to know how ISA Server processes traffic sent to a port that is associated with overlapped protocols. When traffic arrives at a port that is associated with overlapped protocols, the first policy rule that matches the traffic for each of the overlapped protocols (the HTTP and CustomHTTP protocols) is found, and the rule that is highest in the list of rules is applied. In our case, that would be the first rule with the CustomHTTP protocol, which allows traffic to the nonstandard HTTP server but does not invoke the Web Proxy Filter. In addition, all the rules for the overlapped protocols in the ordered list of rules are processed, their secondary connections are added to the session, and the application filters associated with them are invoked until an access rule that denies traffic is encountered. In our case, the second rule, which is a deny rule, stops this processing. Without the second rule, the third rule would be processed for traffic that matches the first rule, and the Web Proxy Filter would be invoked for it.

If the Web Proxy Filter would be invoked by the third rule, the Web Proxy Filter would discover that the traffic does not conform to HTTP standards. The Web Proxy Filter would then block the traffic and add an entry to the Web Proxy log indicating that the Allow HTTP to Internal Servers rule blocked the traffic.

Pesach Shelnitz

ISA Server Team

Comments

• Anonymous
  January 01, 2003
  It does not work with ISA 2006! Someone has another workaround?

• Anonymous
  January 01, 2003
  please be aware, that this workaround doesn't work for Web Proxy Clients! Only SecureNat and Firewall Clients will work.

• Anonymous
  January 01, 2003
  We've just successfully implemented this on ISA 2006 Enterprise due to an issue with our custom HTTP Protocol and the rule base being evaluated twice by ISA.

• Anonymous
  January 01, 2003
  Is there similar solution for ISA 2006? This solution works great on 2004 server but not on 2006.

• Anonymous
  January 01, 2003
  anyone came up with solution for 2006 ISA?

• Anonymous
  December 30, 2006
  Wow great site! Some really helpful information there.,Wow great site! Some really helpful information there.

• Anonymous
  November 08, 2007
  The comment has been removed

• Anonymous
  October 09, 2012
  Can this method be used for FTP and FTES?  FTP needs the custom FTP application filter enabled for port 21 enabled whereas FTPES does not.   I am trying  to make both FTP and FTPES work through ISA.

• Anonymous
  January 10, 2016
  I am trying to use this method for an outbound HTTP access rule to port 80 on windowsupdate.com from a WSUS server behind the TMG. The Allow rule containing my custom HTTP protocol is not processed, it seems, because the connection from the WSUS hits the Deny rule containing the standard HTTP protocol. Should this be working for me, or am I trying something that can't be done? Should I be seeing the rule with the custom protocol listed under Web Access Policy? Only the Deny rule containing standard HTTP is listed there.