This help contains the following topics:

·     Introduction

¡     URL

¡     URL filtering rule

¡     URL category

¡     URL filtering profile

¡     URL filtering whitelist/blacklist rule

¡     URL filtering cloud query

¡     URL filtering action

¡     URL filtering mechanism

·     Restrictions and guidelines

¡     Restrictions and guidelines: Text-based URL filtering rule configuration

¡     Restrictions and guidelines: Regular expression-based URL filtering rule configuration

¡     Restrictions and guidelines: Whitelist

¡     Restrictions and guidelines: URL filtering profile configuration

·     Configure URL filtering

¡     Configure a URL category

¡     Configure the cloud query server

¡     Configure a URL filtering profile

Introduction

URL filtering controls access to the Web resources by filtering the URLs that the users visit.

URL filtering supports filtering only HTTP packets.

URL

A URL is a reference to a resource that specifies the location of the resource on a network and a mechanism for retrieving it. The syntax of a URL is protocol://hostname[:port]/path/[;parameters][?query]#fragment. Figure 1 shows an example URL.

Figure 1 URL syntax

 

Table 1 describes the fields in a URL.

Table 1 URL field descriptions

Field	Description
protocol	Transmission protocol, such as HTTP.
host	Domain name or IP address of the server where the indicated resource is located.
[:port]	Optional field that identifies the port number of the transmission protocol. If this field is omitted, the default port number of the protocol is used.
/path/	String that identifies the directory or file where the indicated resource is stored. The path is a sequence of segments separated by zero or multiple forward slashes.
[parameters]	Optional field that contains special parameters.
[?query]	Optional field that contains parameters to be passed to the software for querying dynamic webpages. Each parameter is a <key>=<value> pair. Different parameters are separated by an ampersand (&).
URI	Uniform resource identifier that identifies a resource on a network.

 

URL filtering rule

A URL filtering rule matches URLs based on the content in the URI or hostname field.

URL filtering provides the following types of URL filtering rules:

·     Predefined URL filtering rules—Signature-based URL filtering rules. The device automatically generates them based on the local URL filtering signatures. In most cases, the predefined rules are sufficient for URL filtering.

·     User-defined URL filtering rules—Regular expression- or text-based URL filtering rules that are manfully configured.

A URL filtering rule supports the following URL matching methods:

·     Text-based matching—Matches the hostname and URI fields of a URL against text patterns.

When performing text-based matching for the hostname field of a URL, the device first determines if the text pattern contains the asterisk (*) wildcard character at the beginning or end.

¡     If the text pattern does not contain the asterisk (*) wildcard character at the beginning or end, the hostname matching succeeds if the hostname of the URL matches the text pattern.

¡     If the text pattern contains the asterisk (*) wildcard character at the beginning, the hostname matching succeeds if the hostname of the URL matches or ends with the text pattern without the wildcard character.

¡     If the text pattern contains the asterisk (*) wildcard character at the end, the hostname matching succeeds if the hostname of the URL matches or starts with the text pattern without the wildcard character.

¡     If the text pattern contains the asterisk (*) wildcard character at both the beginning and the end, the hostname matching succeeds if the hostname of the URL matches or includes the text pattern without the wildcard characters.

Text-based matching for the URI field works in the same way that text-based matching for the hostname field works.

·     Regular expression-based matching—Matches the hostname and URI fields of a URL against regular expressions. For example, if you set the regular expression for hostname matching to sina.*cn, URLs that carry the news.sina.com.cn hostname will be matched.

URL category

URL filtering provides the URL categorization feature to facilitate filtering rule management.

You can classify multiple URL filtering rules to a URL category and specify an action for the category. If a matching rule is in multiple URL categories, the system takes the action for the category with the highest severity level.

URL filtering supports the following types of URL categories:

·     Predefined URL categories.

The predefined URL categories contain the predefined URL filtering rules. Each predefined URL category has a unique severity level in the range of 1 to 999, and a category name that begins Pre-. Predefined URL categories cannot be modified.

·     User-defined URL categories.

You can manually create URL categories and configure filtering rules for them. The severity level of a user-defined URL category is in the range of 1000 to 65535. You can edit the filtering rules and change the severity level for a user-defined URL category.

URL filtering profile

A URL filtering profile can contain multiple URL categories, and each category has an action defined for packets that match a filtering rule in the category. You can also specify the default action for packets that do not match any filtering rules in the profile.

URL filtering whitelist/blacklist rule

The device supports using URL-based whitelist and blacklist rules to filter HTTP packets. If the URL in an HTTP packet matches a blacklist rule, the packet is dropped. If the URL matches a whitelist rule, the packet is permitted to pass through.

URL filtering cloud query

You can enable cloud query in a URL filtering profile to improve URL filtering accuracy for HTTP traffic.

With cloud query enabled, the device will send URLs that do not match any local URL filtering rules to the cloud server for further query. The device determines the action to apply according to the query result returned from the cloud server.

·     If a matching rule is found for the URL, the action specified for the URL category to which the rule belongs apply. If the rule belongs to multiple URL categories, the action specified for the category with the highest severity level apply.

·     If no matching rule is found, the device executes the default action of the URL filtering profile on the packet. If the default action is not configured, the device permits the packet to pass through.

URL filtering action

You can specify URL filtering actions for URL categories and specify the default action for a URL filtering profile.

The device supports the following URL filtering actions:

·     Blacklist—Drops matching packets and adds the sources of the packets to the IP blacklist. If the IP blacklist feature is enabled, packets from the blacklisted sources will be blocked for the blacklist period. If the IP blacklist feature is not enabled, packets from the blacklisted sources are not blocked.

For more information about the IP blacklist feature, see attack defense online help.

To configure the blacklist period for the blacklist action, go to Objects > APPSecurity > Security Actions > Block.

·     Drop—Drops matching packets.

·     Permit—Permits matching packets to pass.

·     Redirect—Redirects matching packets to a webpage.

·     Reset—Closes the TCP connections for matching packets by sending TCP reset messages.

·     Logging—Logs matching packets. This action takes effect only after the logging function is enabled in the URL filtering profile.

URL filtering mechanism

URL filtering takes effect after you apply a URL filtering profile to a security policy rule.

As shown in Figure 2, upon receiving a HTTP packet, the device performs the following operations:

1.     The device compares the packet with the security policies.

If the packet matches a security policy that is associated with a URL filtering profile, the device extracts the URL from the packet.

2.     The device compares the extracted URL with the rules in the URL filtering profile.

¡     If the URL matches a rule, the device determines the action for the packet as follows:

-     If the matching rule is a whitelist rule, the packet is permitted to pass through.

-     If the matching rule is a blacklist rule, the packet is dropped.

-     If the matching rule is a URL filtering rule, the devices takes the action specified for the URL category to which the rule belongs.

If the URL filtering rule belongs to multiple URL categories, the action specified for the URL category with the highest severity level apply.

¡     If the URL does not match any rule in the policy, and cloud query is disabled in the profile, the default action specified for the policy applies. If the default action is not configured, the device permits the packet to pass through.

¡     If the URL does not match any rule in the policy, and cloud query is enabled in the policy, the device performs step 3.

3.     The device forwards the URL to the cloud server for further query.

¡     If a matching rule is found for the URL, the action specified for the URL category to which the rule belongs apply. If the rule belongs to multiple URL categories, the action specified for the category with the highest severity level apply.

¡     If no matching rule is found, the device executes the default action of the policy on the packet. If the default action is not configured, the device permits the packet to pass through.

Figure 2 URL filtering mechanism

 

Restrictions and guidelines

Restrictions and guidelines: Text-based URL filtering rule configuration

Follow these guidelines when you use the asterisk character (*) in the text pattern for hostname or URI matching:

·     For hostname matching, the asterisk (*) can appear only at the beginning or end of the text pattern as a wildcard character to match zero or more characters.

·     For URI matching, the asterisk (*) can appear at the beginning or end of the text pattern as a wildcard character to match zero or more characters, or appear in the middle as a non-wildcard character.

Restrictions and guidelines: Regular expression-based URL filtering rule configuration

·     The regular expression pattern can contain a maximum of four branches. For example, 'abc(c|d|e|\x3D)' is valid, and 'abc(c|onreset|onselect|onchange|style\x3D)' is invalid.

·     Nested braces are not allowed. For example, 'ab((abcs*?))' is invalid.

·     A branch cannot be specified after another branch. For example, 'ab(a|b)(c|d)^\\r\\n]+?' is invalid.

·     A minimum of four non-wildcard characters must exist before an asterisk (*) or question mark (?). For example, 'abc*' is invalid and 'abcd*DoS\x2d\d{5}\x20\x2bxi\\r\\nJOIN' is valid.

Restrictions and guidelines: Whitelist

·     If a packet matches a whitelist rule, the URL Category column in the URL filtering log list displays Whitelist.

·     If the referer header of an HTTP request matches the URL filtering whitelist, the URL Category column in the URL filtering log list displays Referer Whitelist.

·     With whitelist mode enabled, if a packet fails to match any whitelist rules, the URL Category column in the URL filtering log list displays Blacklist.

Restrictions and guidelines: URL filtering profile configuration

For configuration changes (create, edit, and delete settings) in an URL filtering profile to take effect immediately, click Submit. If you do not click Submit, the configuration changes will automatically take effect after 40 seconds.

Configure URL filtering

Configure URL filtering as shown Figure 3.

Figure 3 URL filtering configuration procedure

 

 

Configure a URL category

Perform this task to create a user-defined URL category and configure filtering rules for it to meet specific URL filtering requirements.

Procedure

1.     Click the Objects tab.

2.     In the navigation pane, select APPSecurity > URL Filtering > URL Categories.

3.     Click Create.

4.     Create a URL category.

Table 2 URL category configuration items

Item	Description
Name	Enter a name for the URL category. The category name cannot start with Pre-.
Description	Enter a description for the URL category.
Severity level	Assign a unique severity level in the range of 1000 to 65535 to the URL category. The larger the value, the higher the severity level.
Include predefined category	Select a predefined URL category to add all its rules to the URL category.

 

5.     Add URL filtering rules to the URL category:

a.     Click Add.

b.     From the Match pattern list, select the match pattern type for the host name field. Options are Text and Regular expression.

c.     Enter the match pattern for the host name field.

d.     From the Match pattern list, select the match pattern type for the URI field. Options are Text, Regular expression, and –NONE—.

e.     Enter the match pattern for the URI field. This step is not required if the –NONE— option is selected for the match pattern of the URI field.

f.     Click OK.

g.     Repeat the preceding steps to add more URL filtering rules.

6.     Click OK.

The URL category is displayed on the URL Categories page.

Configure the cloud query server

Perform this task to configure the cloud query server for URL filtering.

Procedure

1.     Click the Objects tab.

2.     In the navigation pane, select APPSecurity > URL Filtering > URL Categories.

3.     Click Configure next to the Cloud server connection status field.

4.     Configure the cloud query server.

Table 3 Cloud query server configuration items

Item	Description
Server address	Enter the IP address or hostname of the cloud query server. Only the cloud query server of our company is supported.
Max cached URL entries	Specify the maximum number of URL entries that can be cached. The device creates a URL cache entry for each unique URL submitted to the cloud query server for cloud query. The cloud query result will be stored in the cache entry.
Min cache period	Specify the minimum cache period for a URL cache entry in minutes. Setting the minimum cache period for URL cache entries ensures that the entries will not be deleted during the specified period of time. When the URL filtering cache is full, the system identifies the cache period of the oldest URL cache entry to determine whether to overwrite it: ·     If the cache period of the entry is equal to or less than the minimum cache period, the system does not delete the entry. The new entry is not cached. ·     If the cache period of the entry is greater than the minimum cache period, the system overwrites the entry with the new entry. However, if the configured max cached URL entries is less than the currently cached entries, the system will delete the oldest cache entries even if their cache periods are equal to or less than the minimum cache period.

 

Configure a URL filtering profile

You can configure the following settings in a URL filtering profile:

·     Enable cloud query.

·     Specify the default action for packets that do not match any URL filtering rule.

·     Configure whitelist and blacklist rules. The whitelist rules take precedence over the blacklist rules.

·     Specify actions for URL categories.

A URL filtering profile contains all URL categories on the device. You can specify the actions for individual URL categories in the URL filtering profile. If an HTTP packet matches a URL filtering rule in a URL category, the action specified for the category applies to the packet. If the matching rule is in multiple URL categories, the action specified for the category with the highest severity level is taken.

Procedure

1.     Click the Objects tab.

2.     In the navigation pane, select APPSecurity > URL Filtering > Profiles.

3.     Click Create.

4.     Create a URL filtering profile.

Table 4 URL filtering profile configuration items

Item	Description
Name	Enter a name for the URL filtering profile.
Default action	Select the default action to take on packets that do not match any URL filtering rules. Options are Permit, Drop, Reset, Redirect, and Blacklist.
Cloud query	Select the box to enable cloud query.
Logging	Select the box to enable logging for packets matching URL filtering rules. Before you select this item, configure the default action first.
Enable HTTPS URL filtering	Select the box to enable URL filtering on undecrypted HTTPS traffic. If the SSL decryption action is selected in any enabled proxy policy, this feature does not take effect. For more information about SSL decryption, see proxy policy online help.
Enable referer whitelist	Select the box to allow an HTTP request to pass if its referer header matches the URL filtering whitelist.
Whitelist mode	Select the box to allow users to access the websites in the URL filtering whitelist only.
Whitelist	Add whitelist rules to the URL filtering profile as needed. To add a whitelist rule to the URL filtering profile: 1.     In the Whitelist area, click Add. The Add Whitelist Rule window opens. 1.     From the Match pattern list, select the match pattern type for the host name field. Options are Text and Regular expression. 2.     Enter the match pattern for the host name field. 3.     From the Match pattern list, select the match pattern type for the URI field. Options are Text, Regular expression, and –NONE—. 4.     Enter the match pattern for the URI field. This step is not required if the –NONE—option is selected for the match pattern of the URI field.
Blacklist	Add blacklist rules to the URL filtering profile as needed.
URL categories	In the URL categories area, select the actions for individual URL categories. Supported actions are Permit, Drop, Reset, Redirect, Blacklist, and Logging. Before you select the logging action for a URL category, select an action among the Permit, Drop, Reset, Redirect, and Blacklist actions first.

 

5.     Click OK.

The URL filtering profile is displayed on the URL Filtering Profiles page.

6.     Use the URL filtering profile in a security policy.

For more information about security policies, see security policy online help.

7.     To have the configuration take effect immediately, click Submit.

If you do not perform this operation, the configuration will automatically take effect after 40 seconds.