mirror of
https://github.com/Grasscutters/mitmproxy.git
synced 2024-11-23 16:17:49 +00:00
ee6937f948
- robots.txt prevents crawling of everything but /stable - error.html has a simple meta refresh to the latest stable docs. This will also help crawlers find the right path.
3 lines
52 B
Plaintext
3 lines
52 B
Plaintext
User-agent: *
|
|
Disallow: /archive/
|
|
Disallow: /master/ |