PHP - Detecting Spam Requests With Raw Analytics

If you own a website, chances are you have some sort of analytics software installed such as Google Analytics to report on traffic. But Google Analytics doesn't report on spiders/bots/referral spammers that crawl your site using your bandwidth and resources. In this article, I discuss the approach I took to try and block these spiders and spammers.

First, some background. A few weeks ago I noticed a high number of referrals appear in my Google Analytics. The referral host names didn't seem related to the contents of my website and a quick web search revealed that these referrals were indeed spammy jokers. It seemed other website owners were having the same problem. Below are just a few hosts that have plagued my website with there ghost referrals.

Please do not visit any of these sites.


To understand what was going on, I decided to start monitoring details of each request. After capturing data for a 24 hour period, I went through each of the requests and noticed not only a high number of spam referrals that Google Analytics didn't report on but also a very high number of search bots.

To make matters even worse, one of my articles was spammed by various hosts. Again this did not appear on Google Analytics. Further investigation revealed that most of these referral hosts shared the same IP address and each one of these hosts would request a page up to three times in one visit.

My initial though was to add some entries in my .htaccess file that would redirect these hosts else where, but the problem was I would need to know which referral hosts to redirect and that meant monitoring HTTP requests. Eventually I decided to develop a simple Raw Analytics PHP control panel that would allow me to view recent requests and take action if needed. I created a block list, that allowed me to block a request based on IP address, host name and user-agent.

After 24 hours of setting up the block list, I noticed a huge reduction in crawlers and spammy referrals and after several weeks of using the block list, I've noticed my bandwidth usage reduce. Of course I get the occasional unwanted crawler or referrer.

You can download the application from the link below.


C# - Group Chat
If you want to know how to develop a group chat application, have a look at this project. The project includes a server and a client. The server listens for TCP client connections. Once a client is connected to the server, the server maintains the connection in a collection. Every time a message is sent from a client to the server, the server broadcasts the message to all clients in the collection.
C# - Custom InputBox
C# unlike Visual Basic, does not have an Input DialogBox class to capture user input. Buts that's not a problem as you can just add a reference to Microsoft.VisualBasic.dll and use the static InputBox class from the Microsoft.VisualBasic.Interaction namespace. But if you want a custom InputBox, then checkout this project.
PHP - JsonDb A Flat File Database
JsonDb is a simple flat file database library designed to perform CRUD operations on a flat file. In this article we take a look at how JsonDb can be used to store data in a json file.
PHP - MicroMVC A Simple Framework
The MVC pattern is a widely used pattern for developing web applications. PHP has a vast range of open source MVC frameworks that help to ease development. This article presents a lightweight MVC framework that you can use as a bootstrap.
ASP.Net - Simple File Upload
Asp.Net makes uploading files as simple as a few lines of code. In this article, I explain how to upload files using Web Forms and C#.
C# - Text To Speech
The .Net Framework makes developing Text to Speech applications very simple with the System.Speech assembly. In this article, I explain how easy it is to get started with Text to Speech.
C# - Image Slider Control
This Windows control can be used to view images in a slide show with captions. The control is configurable with a few useful settings. The list below contains a set of properties which can be used to configure the ImageSlider control.
C# - Custom Button Control
Creating custom controls with the .Net Framework is relatively easy. In this article I explain how to create a custom button control using GDI.
C# - Reflection
The .Net Framework provides a reflection capability that allows you to obtain information about assemblies, modules and types as well as being able to dynamically create instances of types. In this article, I explain how to get type information and create instances dynamically.
PHP - Simple Database ORM
DataContext is a simple ORM class that can be used to manage CRUD operations on a database. Standard PHP objects or arrays are used to insert/update data.
PHP - Introduction To PDO Part 1
PHP Data Objects (PDO) is a simple database abstraction extension that allows you to interact with databases. This tutorial introduces you to the PDO extension and provides sample code to help you get started.
PHP - Introduction To PDO Part 2
In part 1, I explained how to connect to a MySQL database server using the PDO extension. In this tutorial, I explain more about the PDOStatement class and how it's used to fetch data.
PHP - Simple Template Engine Part 1
In this two part tutorial, I will explain how to develop your own template engine that uses a simple find and replace technique. In part 1, you will learn how to find and replace special tags known as placeholders with scalar values and in part 2, I will explain how to use simple objects as placeholders.
PHP - Simple Template Engine Part 2
In part 1, I explained how to develop a simple find and replace template engine which substituted placeholders in a template file with scalar values. In this tutorial, I will explain how to access object properties and array elements in the template.
A Custom Windows Message Box It's taken a while, but I've finally, updated a custom MessageBox class, that I had developed a few years ago. In this article, I'll explain some of the new changes and why I made them.
Introduction To Stored Procedures And Functions Part 1 MySQL introduced Stored Routines as of 5.0 making it possible to execute SQL code stored on the server. In this article I will explain what Stored Routines are and how they can be used with examples.
ASP.Net C# MessageBox Modal boxes are great to get a users response. Sometimes, they replace the traditional JavaScript alert and confirm dialogs giving the end user a better experience. But managing modal boxes in code ca...
Accordion Menu In Window Forms In this tutorial, I'll explain how to develop a custom Accordion menu class, that you can use in your Windows Forms application to group controls together.
JSON Viewer At some point you might have been exposed to JSON data whilst working on a project. Sometimes the data can be hard to read if your not using a decoder and your working with large data.