Why we have to write robots.txt in website application? [closed]

by

It lists URLs that you don’t want bots to hit, so you can keep pages out of search indexes and stop CPU intensive scripts from being repeated hit by automated processes.

The syntax is described at robotstxt.org and is basically a series of:

User-agent: $PATTERN

each followed by any number of 

Disallow: $PATH_PREFIX

More Related Contents:

  1. html/css empty space at bottom of the website
  2. How to display an icon when you add a site to favorites in web? [closed]
  3. How can i achieve this style in Html/Css?
  4. Chrome: Automatic shift of web elements
  5. What does #/ means in url?
  6. Is there a minlength validation attribute in HTML5?
  7. An invalid form control with name=” is not focusable
  8. How to line-break from css, without using ?
  9. How to create uneven rounded sides on a div?
  10. Why are nested anchor tags illegal?
  11. CSS Cut out circle from a rectangular shape
  12. Problems with CSS sticky footer
  13. What is quirks mode?
  14. href syntax : is it okay to have space in file name
  15. How to select a text node with CSS
  16. How can I make a CSS glass/blur effect work for an overlay?
  17. How to exclude particular class name in CSS selector?
  18. How to make div element auto-resize maintaining aspect ratio?
  19. Do I need a “/” at the end of an or tag, etc.? [duplicate]
  20. How do I prevent the padding property from changing width or height in CSS?
  21. Div width 100% minus fixed amount of pixels
  22. Force to download image instead of opening url link to image [duplicate]
  23. Can XHTML and HTML class attributes value start with a number?
  24. How to specify model to a ngInclude directive in AngularJS?
  25. HTML parsing in perl
  26. How do I style the HTML form validation error messages with CSS?
  27. What are the default values for justify-content & align content?
  28. What is the difference between phrasing content and flow content?
  29. Rendering a hierarchy of “OPTION”s in a “SELECT” tag
  30. Dashed border not showing in firefox [duplicate]

Leave a Comment