Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
Digital strategist Dr Dave Chaffey is co-founder and Content Director of marketing publisher and learning platform Smart Insights. Dave is editor of the 100+ templates, ebooks and courses in the digital marketing resource library created by our team of 25+ digital marketing experts. Our resources are used by our Premium members in more than 100 countries to Plan, Manage and Optimize their digital marketing. Free members can access our free sample templates here. Dave is a keynote speaker, trainer and consultant who is author of 5 bestselling books on digital marketing including Digital Marketing Excellence and Digital Marketing: Strategy, Implementation and Practice. To learn about my books, see my personal site Digital marketing books by Dr. Dave Chaffey. In 2004 he was recognised by the Chartered Institute of Marketing as one of 50 marketing ‘gurus’ worldwide who have helped shape the future of marketing. Please connect on LinkedIn to receive updates or ask me a question.
Off page SEO: This type of SEO focuses on all of the activity that takes place "off the page" when looking to optimize your website. "What activity not on my own website could affect my ranking?" You might ask. The answer is inbound links, also known as backlinks. The number of publishers that link to you, and the relative "authority" of those publishers, affect how highly you rank for the keywords you care about. By networking with other publishers, writing guest posts on these websites (and linking back to your website), and generating external attention, you can earn the backlinks you need to move your website up on all the right SERPs.