spiderbar
Parse and Test Robots Exclusion Protocol Files and Rules