robots.txt and forgotten folders
The Polo Nord ice cream shop has a simple website and a slightly naive owner: he tells search engines not to look in certain folders and thinks that keeps them safe. Explore the site, find out what he wants to hide and take home the flag.
Lab 1 of 7 in the Web Exploitation path. If you get stuck you have 3 hints, from the vaguest to the most precise: they cost 0, 30 and 100 XP.
What you learn
Objectives complete themselves while you work in the terminal; each one is worth experience points:
- Map the site pages with gobuster (50 XP)
- Read robots.txt (75 XP)
- Open the folder "not to be indexed" (75 XP)
Topics: web, robots-txt, enumeration.
Commands you will use
The terminal is simulated and understands about forty Linux commands. In this lab you will mostly need these:
gobuster: automatically tries a list of folder and page names on a website, to find what no link shows.curl: makes HTTP requests from the terminal and shows the server’s raw answer, headers included.
The simulated environment
The target is an invented host on a simulated private network: you can only reach it from the game terminal.
The target exposes these services:
- SSH: encrypted remote command-line access; it is protected with keys instead of passwords and without direct root login.
- Web (HTTP): a web server with pages, APIs or admin panels: the most exposed part of almost every organisation.
The defensive lesson
Once the lab is solved, the write-up explains what went wrong and how to protect yourself:
robots.txt politely asks search engines not to index certain pages. It is not protection: it is a "keep out" sign hung on an open door, and since it is public it becomes a map of the interesting places. Defensive lesson: protect what is private with logins and permissions on the server (the folders that answered 403 held, the open one did not), never list secrets in robots.txt and keep directory listing switched off.
Before and after this lab
To unlock it you must have solved:
- Mapping a host (Recon & Scanning, easy)
After this lab you can move on to:
- The cookie that is trusted too much (Web Exploitation, medium)
- The manufacturer's manual (Web Exploitation, medium)
The Web Exploitation path
Web Exploitation brings the basics into the world of websites and web applications, where a huge share of real attacks happen. You explore invented sites with HTTP requests from the terminal and discover folders someone hoped to hide, cookies the server trusts too much, factory credentials never changed, comments forgotten in the page source and error messages that say too much. The lessons follow the OWASP Top 10 categories, such as Broken Access Control and authentication failures, and each one explains the fix.