:: TSP Menu
- Home
- Authors
- Glossary
- RSS Feed

:: Servers
- CPanel
- Hosting
- Linux
- Windows

:: Webmaster
- Databases
- Design
- PHP
- Search Engines

:: Scripts
- Scripts

:: Documentation
- PHP Manual

:: Other
- TechPunt
- Wojjie

»
TheServerPages Articles
»
Webmasters
»
Search Engines

Search Engine Optimization (SEO)

Articles:

Robots.txt Format

by Wojjie (2004-07-19)
When search engines crawl through the internet, they check for the existance of the 'robots.txt' file. This file tells the search engine what urls it should not crawl to, and has a simple file format.

Links: Game-Monitor