There are three main search preferences meta tags custom robots txt and custom robots header tags special precaution would be taken to customize these three main search preferences because in case of incorrect use of these may cause permanent ignored from search engines see the below
For search engines better visibility all these should be in enabled form