301 redirect checker script

This tutorial contains 301 redirect checker script which I have created after spending more than half an hour searching the Internet for something suitable I can use for my needs. While I was searching I found only separate web pages with fields where you can check one – two URLs or heavy applications that can be installed, and yet not covering my needs. Certainly, there were paid 301 redirect checker solutions which can be used to check 301 Permanent Redirect, but I am not sure whether it does worth to buy such. The purpose of this checker in most cases is only to see if a particular pages redirect permanent covering the SEO (Search Engine Optimization) requirements.

In my case part of the OnlineHowTo.net is moving over Free Tutorials Submit dot com, and from SEO perspective, the tutorials moved have to be with 301 redirect to their new URLs.

I decided to write it as Linux bash 301 redirect checker script because it can be run on almost any Linux machine with curl installed and easily added to the Crontab. I believe the script is rather easy to be understood and used, even by not so experienced administrators.

What the redirect script is doing is to gather the URLs you want to check from an external file (in my case: /opt/scripts/301URLs.txt ), where every URL is placed on a separate line like this:

I chose this way as most of the get-all-urls-from-domain applications provide the listing like this, but certainly you can choose (and then modify a little bit my script) different format.
Then using simple bash loop it reads every line one by one, processing it with CURL extracting the headers (curl -I option).
Every header contains information similar to:

Once we have this information we can easily egrep for the 301 Moved Permanently part and if it exist, the 301 redirect checker script continues with the next URL. If the redirect is something different it writes down the URL into the /tmp/301report.txt file.

Finally the script checks whether the /tmp/301report.txt file exist – it will be created only if there is even one different from 301 Moved Permanently redirect – and will send it via e-mail to the address you have set in the EMAIL variable. If there are no such file, the script will end with no report.

Here is the whole script which you can freely use and modify as per your needs:

 

To be honest, I am using a little bit modified 301 redirect checker script, which is also checking whether the redirected URL is the one I need, but it is more complicated and may confuse you if you have to modify it.
Also, more elegant solution will be if you decide to use array, but this redirect checker script fits my needs perfect.
Also I have put sleep command as if the number of the URLs is too big, it may lead to server abuse and blocking the IP you are checking the redirect from. If you do not need the script to wait 5 seconds for every URL, just remove that line.

Hope this scrip will cover your needs as well. Your comments and suggestions are welcome.

Tags

Filed Under: Bash ScriptingLinuxScriptingServer Administration UtilitiesWeb forwardingWeb serversWeb-hosting

Anthony Gee About the Author: Anthony G. is an IT specialist with more than 9 years of solid working experience in the Web Hosting industry. Currently works as server support administrator, involved in consultative discussions about Web Hosting and server administration. One of the first writers in the Onlinehowto.net website, now writing for Free Tutorials community - he is publishing tutorials and articles for the wide public, as well as specific technical solutions.

Comments (4)

  1. LLTolkin says:

    Nice 301 redirect checker. I did’n find such for free also, but your one is rather different than the one I need. You are using external file, but I need my one to be web based.
    Can you tell me is it possible (and how) your one to be rewritten in PHP?

  2. Well, I needed this redirect checker script as bash script, and I chose cURL . With PHP 5 there is a function called ‘get_headers’ which may be used for such script.

    For example:

    will print the header as array. You may create the code to extract or look for something particular like 301 permanent redirect.

  3. Varun says:

    Hi Anthony,
    It would be great if you can upload the modified script too, the one which checks if the redirected url is the one you need.

    • Anthony Gee Anthony Gee says:

      Something simple like this maybe:

      < ?php $file1 = "./test.txt"; $lines = file($file1); foreach($lines as $line_num => $line) {

      echo ‘Count: ‘ . $line_num . ‘ – ‘ . getsz($line);

      }

      function getsz($url) {
      $a = get_headers($url);
      return $a[‘Location’];
      }
      ?>

Leave a Reply