How to Analyse Your Website for SEO – Introducing Screaming Frog

The foundation for any successful SEO strategy is to ensure you’re building from the ground, up. Understanding why there is a need to implement a SEO strategy is essential so to understand the objectives and how performance will be monitored and measured for the benefit of your business. It’s common with many SEO strategies to jump straight into applying keyword research techniques, commencing a link-building campaign or the necessity to develop and create a production line of content generation to then optimise and roll out.

In fact, a quick search on Google for “Common reasons for an unsuccessful SEO Strategy” and the most popular posts are more about suggestions around more tactical approaches rather then dealing with the fundamental issues of why your SEO strategy is not succeesding. In other words, has your website been built with a sound information architecture?

Applying tactical SEO is all good (link building, guest blogging, content marketing, on & off page best practise SEO) but if the architecture of how your site is built then the tactical side of your campaigns will result in failure.

To explain SEO to my colleagues and the wider business, I like to reference the Pyramid (below):









The SEO pyramid is a very effective and straight forward way to communicate how SEO can impact and influence an entire business and that the key to a successful strategy is a lot more in depth than applying the right keywords and driving the right links! In fact this diagram is a few years old and the SEO landscape has become a lot more complicated but it still provides the key stages to consider for best practise SEO. AS Moz suggests, “For optimal results, start with a strong base and build your way up.”

Whether you are a digital marketer or SEO specialist, the purpose of this post is to introduce you to a SEO tool to help assist you and your team to ensure you are building your strategy from the ground up and to help you do this, I’d like to introduce you to the Screaming Frog SEO Spider

I caught up with their co-founder and Director, Dan Sharp to help explain what this tool does, the fact you don’t need to be a technical genius to make best use of the tool and his views and how to extract actions from Screaming Frog to steer your SEO Strategy





Tell us a little about Dan…

I’m co-founder and Director of Screaming Frog, a digital marketing agency based in Henley-on-Thames, Oxfordshire. I run and manage the business and smart team we have here, which has now grown to over twenty people. As an agency we develop digital strategies for large brands internationally, as well as small and medium sized businesses in the UK.

We are particularly well known for our software development, the Screaming Frog SEO Spider crawler, which spiders’ websites’ and allows SEOs, marketers and webmasters to gather data to help audit onsite SEO. You can follow (and chat!) to me via @screamingfrog on Twitter, although I tweet as much about Arsenal and what I am up to, as much as I do SEO and digital!


When did your digital career start?

It started over a decade ago, after I had graduated from university and taken a year out to travel. I eventually came back to the UK and realised I needed a ‘proper job’. Like most, I wasn’t entirely sure what I wanted to do and by chance my very first interview and soon to be role was working at a search marketing agency called Guava (who were eventually acquired by a pan-European agency).

My role grew over the years alongside the company, I worked as an affiliate in my spare time and eventually many years later, I left to work for myself as an SEO Consultant under the name ‘Screaming Frog’. This is when the SEO Spider development started and a little later, I joined forces with Graeme (former MD of Guava) and Jason (former Director of Guava) to incorporate ‘Screaming Frog’ the business.


Screaming Frog SEO Spider – can you sell it to us in a tweet?

I’m not a big fan of selling anything (we don’t actually hire any sales people in the company either), so I’d actually just recommend giving it a go and see if it helps (as it’s free in lite form!). To sum it up in a tweet, simply –

The Screaming Frog SEO Spider allows you to crawl a website for common SEO issues & analyse the data to improve your website organically.


Screaming Frog has very quickly become the tool of choice for SEO’s wanting to look “under the hood” of a website to really get a better understanding of the architecture. How have you managed to become the tool of choice in such a short space of time?

I don’t think there’s a big secret, I just think we built something that a lot of people were having a similar problem with as us. So solving or helping to alleviate those frustrations went a long way in our initial success. After that, I believe it’s because we understand and listen to our users.

We live and breathe SEO and share a lot of the same frustrations with our customers, which aids our development. When you work in the trenches, you know what weapons are required. The tool was originally built and used by us internally, before we released it publically for example.

But more importantly, we’ve had, and still receive, lots of amazing feedback from users and we regularly incorporate this into the tool. Lots of businesses say they listen, but don’t in reality. We listen and we really care.


How do you pitch the tool to a non-technical/SEO audience?

To be honest, I don’t. I have been pretty upfront that we built the SEO Spider for SEOs, or those with the expertise and knowledge to use it. We could make it appeal more to beginners, but I didn’t want to water it down equally, there is a balance.

Obviously we do get plenty of beginners using the tool and we try and provide guidance and help them as much as possible via our support.

I think the tool can be a little overwhelming for someone without experience, but it’s also very intuitive and it will start to make far more sense alongside the moz beginners guide to SEO or Google’s own SEO starter guide.


Can you recommend a good place to start to understand Screaming Frog for newbies?

I’d recommend watching the demo video on our SEO Spider page, reading the FAQs and user guide which we keep as up to date as possible. I also highly recommend the SEER Interactive guide to doing almost anything in Screaming Frog, which is pretty damn awesome.


Getting actionable data and insights to structure your digital strategy is essential. It seems you could feel a little intimidated by the range of data Screaming Frog can offer – what would you recommend are the key metrics/insights a user should focus on to start with?

I agree, the tool can be a little intimidating at first, but it’s also very simple to use. Essentially the tabs are focused around key onsite elements (such as page titles, meta descriptions etc) and you can use the filters to view common SEO issues or items within that element.

I’d recommend crawling a fairly small website to start with and reviewing the basics, such as response codes to find broken links, redirects or server errors by using the relevant tab, and filters contained within them. Then I’d recommend sorting the address column alphabetically (by clicking the ‘address’ column header) and reviewing the content of the page titles, meta descriptions and headings tabs for example.

Are they relevant, descriptive and targeted towards your products, services and audience, based upon your keyword and market research? You can then use the filters to check for common issues, such as those which might be missing, or duplicated, or if any are particularly long or short etc.

You can then start to review technical implementation in more detail, such as directives, things like canonicals, use of meta robots or rel=”next” & rel=”prev” etc.








Can the tool be used to support other digital tactics e.g. Social, affiliate marketing, Mobile?

Not a huge amount – The SEO Spider can be used to identify links or code on a website to social profiles, or affiliate links etc. So it could help find all external affiliate links on a website let you know if there are any problems. You can also test and crawl mobile websites, by changing the user-agent to Googlebot for smartphones for example.


Any advice for a newbies moving into digital?

I like Carl Hendy’s recent ‘A career in SEO’ post, which provides some good advice on starting in SEO. We always suggest that anyone new at Screaming Frog starts their own website or blog (if they haven’t already) about something they are passionate about, to help them learn.

I certainly believe the best way to learn is to give things a go and experiment and we try and provide our team some time each month to work on personal projects.

Actually, the first website one of our team created, now receives over half a million visits each month, which makes a pretty considerable amount of money via Google Adsense, too.


What’s on your reading list?

We started a Screaming Frog library a couple of years ago of ‘must read books’ for the team (where they can submit books for us to buy and keep for the team), and I’ve still not read a lot of them. That said, I’ve just started ‘Predatory Thinking’ by Dave Trott as I’ve heard such good things about it.


Finally, I have to ask, what’s your predictions on Arsenal finishing in the top 4 this season?

You left the best question until last! 😉

Most Arsenal fans (myself included) expect Arsenal to finish in the top 4 this season. Wenger always seems to find a way, even when we haven’t had the funds to really compete and were forced to sell some of our key players over the past 5-6 years.

Now we do have the money, FFP has helped to an extent and we’ve added some really top players in the last year or so. But we haven’t yet found the right balance; as usual we are struggling with numbers due to small squad and injuries, particularly at the back, and defensive quality in the middle (the much talked about holding role). But with our attack, it should be enough to see us make the top 4!

Alexis is the signing of the summer and Arsenal have built a very talented British core, which should be exciting over the next few years.


One thought on “How to Analyse Your Website for SEO – Introducing Screaming Frog

Leave a Reply

Your email address will not be published. Required fields are marked *