Developing a Website Scanner

I run a traffic exchange and a common occurance is for people to advertise sites that cause problems while surfing. Sometimes it’s a mistake, and other times it’s on purpose.

Part of my job is to keep things in rotation clean. I’m doing this better by developing a website scanner in PHP. So far it’s working great. Here are some tips if you are doing something similar:

fsockopen: If you use this to open a site, only pass the port if it’s not port 80. I found that passing port 80 will get me a different site than the one that’s intended.

proxies: Use a large list of proxies for two reasons: 1) It’ll make it more difficult for users to know when the site is being scanned. 2) If you use only one proxy, they may not like having all your automated traffic.

https: Remember to pass port 443 if a different port isn’t passed, otherwise you won’t get a connection.