When trying to test a particular business using Insites, you may get a warning stating that the site is unable to be analysed. Here's a helpful guide to help you understand why, and give you some tips and tricks to solve it.
Downtime
One of the most common issues we can't access a site is simply due to downtime. If the website is overloaded or experiencing a technical issue - we just won't be able to access it.
Check the website is down in your own browser. You can also use services like "Down for everyone or just me" to see if the website is having issues in specific locations.
Blocking
Sometimes a website owner or their hosting company decides to blocks crawlers like Insites from accessing them. This commonly comes up as a 403 error but can also be other status codes like 401, 429, 503 amongst others.
In most cases we are able to detect blocking, and our technology will try different combinations of crawl settings automatically so that the audit can be generated. However, if we can't automatically bypass the blocking, you will see an error like this:
While our technology is able to avoid getting blocked most of the time, it is a cat-and-mouse game, so inevitably in some cases we're unable to test certain websites.
If you own or administer the website in question, reach out to our team so that we can work with you to ensure our crawler is able to pass through your blocking.
If you don't own or administer the website, you can try manually enabling a US proxy or a residential proxy and see if the website can be tested.
If this still doesn't work, then unfortunately there may not be a lot we can do. The website owner has chosen to block automated crawlers and this is entirely out of our control. However, feel free to log a case with us by chat or email and we'll have our team take a look and see if there's anything that can be done.
Geographical restriction
Sometimes a website owner may block visitors outside of their country of operation. This has become more common since the introduction of the GDPR data protection laws in Europe - many US websites have chosen to simply block visitors outside the US, rather than attempt to comply.
Since Insites servers are based in the EU (Dublin, to be precise) our crawler sometimes falls foul of this kind of blocking. In most cases, our platform is able to detect and resolve this automatically, by routing the traffic through a US-based proxy.
If Insites isn't able to detect the geographic blocking automatically, you can try manually enabling a US proxy from the audit settings.
Enabling a proxy on an audit
To enable a proxy for a given audit, first browse to the audit in question. In the top right, head to the Actions drop down menu and choose Edit audit settings. From the left menu, choose the Website tab. Scroll down to find Proxy to use and change this from auto to US or Residential - you'll need to decide which option applies depending on the kind of blocking you're trying to circumvent. Once you've hit Save changes, you'll be prompted to run a new audit with these settings enabled.
Enabling a US proxy for all websites
If you are US based and you run into blocking issues frequently, we recommend that you turn on the Always use US proxy setting, that can be found in the Analysis section of the Account settings, under General settings:
