Author: theninja

Web/Internet filtering

Internet filtering used to be relatively scarce but it’s extremely common now and takes a variety of forms.  The two most basic forms are URL and content filtering .

URL Filtering

Typical examples of URL filtering is where the requested URL of a web site is intercepted by the proxy or firewall and compared to a big list of ‘bad urls’.  If the URLs match then the request is denied and blocked.  In  this case the user is normally redirected to an error page, although in some cases the request will be logged and an administrator alerted.   It’s not a great system as if you have an extensive list of URLs it can have a big performance impact – and remember this impact is for all requests even those that don’t contain a blocked site.

In recent years some performance improvements have been made to alleviate the issues.  For instance some URL filtering systems use hash values of the URLs rather than the addresses themselves.  The hash values can be ordered so that the system can locate information faster (by jumping to specific points in the list rather than searching from start to finish).   Most systems you’ll find in corporate environments will use URL filtering to some extent.

There can be lots of other problems with filtering simply based on a list especially if you use the hash value searching system.  The URLs have to complete and only that exact, specific address is restricted.   Many websites have multiple domain names and aliases so any list has to have all these URLs listed too.

Content Filtering

Just like URL filtering has a noticeable impact on performance, the same can be said of content filtering.   Content filters look inside the data being transmitted – their goal is not only to block access to inappropriate sites but also to check for security risks.  A content filtering system will often be set to filter out specific objects like Java or ActiveX.   They also check for viruses and other security problems entering the network.

These filtering systems are very sophisticated – analyzing the actual packet data though is bound to have an impact on any networks performance.  Content filters will usually defeat the use of anonymous proxies as the end URL is irrelevant – the data itself is being scanned which will reveal both the proxy address and the destination URL.   An example of one of the most widely used content filters is WebSense – which uses a variety of plug ins and runs on dedicated hardware strategically placed with a tap into all network traffic.

Ninja Bypassing of Filtering Systems

To defeat the URL filtering system is normally fairly straight forward, most anonymous ninja proxy servers available on the internet will suffice.  The only difficulty is that most URL lists contain a large selection of these sites – so if the one you use is on the list you’re going to get blocked.   Not only that but the administrator will likely be informed that someone is deliberately trying to bypass corporate restrictions.  If you set up your own using a hosting account and a Glype installation then you’ll likely be able to surf under the radar.

Unfortunately the mass majority of filtering devices now use both URL and Content filtering technology. The normal web proxy sites you’ll see on the internet promising you complete anonymity and the ability to bypass filters are completely useless. The content filter will look into the packet itself – the fact you are using a proxy and a fake ip are irrelevant.

There is only one effective way to defeat a genuine content filter and that is to encrypt your surfing. In this case the URLs and sites you are visiting are unable to be read by the content filters.   The ideal method is to use a residential VPN that is a VPN to encrypt combined with a standard residential address to bypass IP filters. Although standard datacenter IPs will encrypt they will cause flags to be raised particularly on e-commerce sites like Supreme. This is because they are commonly used for proxy and fraud detection.