Robots gone rogue

After the title and description of this challenge, you really shouldn't need a hint 😜

On websites, what file keeps web crawlers, spiders away from certain files and directories?  What's another name for these web crawlers and spiders?

A robots.txt file is a simple text file used by websites to give instructions to web crawlers (also known as robots or spiders) about which pages or directories they are allowed or disallowed to access. It is part of the Robots Exclusion Protocol (REP) and helps site owners manage which parts of their site should be crawled and indexed by search engines like Google, Bing, etc. 

To solve this challenge, have a look at the robots.txt file here:
https://erniejohnson.ca/cybersecurity/ctf/robot_fan_page/robots.txt

From there, you can see we're trying to hide the 'badrobots' directory.  But you, a cybersecurity expert, can easily access it with the full URL: https://erniejohnson.ca/cybersecurity/ctf/robot_fan_page/badrobots/

🎓 Here's some extra learning about robots.txt (ChatGPT generated, but it's accurate) and how it ties to cybersecurity.