When trying to test a particular business using Insites, you may get a warning stating that the site is unable to be analysed. Here's a helpful guide to help you understand why, and give you some tips and tricks to solve it.
One of the most common issues we can't access a site is simply due to downtime. If the website is overloaded or experiencing a technical issue - we just won't be able to access it.
Sometimes a website owner or their hosting company decides to blocks crawlers like Insites from accessing them. This commonly comes up as a 403 error but can also be other status codes like 401, 429, 503 amongst others.
In most cases we are able to detect blocking, and our technology will try different combinations of crawl settings automatically so that the audit can be generated. However, if we can't automatically bypass the blocking, you will see an error like this:
While our technology is able to avoid getting blocked most of the time, it is a cat-and-mouse game, so inevitably in some cases we're unable to test certain websites.
Sometimes a website owner may block visitors outside of their country of operation. This has become more common since the introduction of the GDPR data protection laws in Europe - many US websites have chosen to simply block visitors outside the US, rather than attempt to comply.
Since Insites servers are based in the EU (Dublin, to be precise) our crawler sometimes falls foul of this kind of blocking. In most cases, our platform is able to detect and resolve this automatically, by routing the traffic through a US-based proxy.
Enabling a proxy on an audit
To enable a proxy for a given audit, first browse to the audit in question. In the top right, head to the Actions drop down menu and choose Edit audit settings. From the left menu, choose the Website tab. Scroll down to find Proxy to use and change this from auto to US or Residential - you'll need to decide which option applies depending on the kind of blocking you're trying to circumvent. Once you've hit Save changes, you'll be prompted to run a new audit with these settings enabled.
Enabling a US proxy for all websites
If you are US based and you run into blocking issues frequently, we recommend that you turn on the Always use US proxy setting, that can be found in the Analysis section of the Account settings, under General settings: