Select a tag to browse associated projects and drill deeper into the tag cloud.
html-snapshots
H
Analyzed about 1 year ago
A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input
Licenses:
mit
robots.txt-go
R
Analyzed about 1 year ago
robots.txt exclusion protocol implementation for Go language (golang).
Contains logically separate parser and checker.
Parser may be used to check robots.txt file for correctness.
Checker may be used for may-i-visit-this-url kind of queries against parsed robots.txt data.