robotstxt
From MaRDI portal
Software:100982
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Last update: 3 September 2020
Copyright license: MIT license, File License
Software version identifier: 0.7.13
Related Items (2)
This page was built for software: robotstxt
