How to Optimize Your Robots txt File

- HD качество

How to Optimize Your Robots txt File Your Robots.txt file is your “anti sitemap”. Instead of telling Google and other search engines about the content you want them to find and index, a Robots.txt file tells them what pages on your site you don’t want them to find. In this video you’ll learn how to block search engine spiders from accessing the pages on your site that you don’t want people to be able to find via a search engine. Video Transcript Hey, what’s up everybody, it’s Brian Dean from Quick Sprout. In this video I’m going to show you how to optimize your robots.txt files for SEO. First of all, what is a robots.txt file? It’s basically the opposite of your site map. A site map exists to tell search engines what pages you want them to index. And, a robots file is the opposite – it tells them what pages you don’t want them to index. You’re actually looking at a robots.txt file right now. I’m going to walk you through what each part of the robots file means and how you can modify it for your site. The first thing you want to do is check to see if you have one, and if you do, look at what settings are already there. To do that, head into your favorite browser. Type in your site name followed by robots.txt. If you see something that looks like this then you have a robots file. If not, then you don’t. If you want to modify this file you’ll need to install a plugin in WordPress. Head over to WordPress. Hover over plugins and click on add new. Then, under the search field put in robots. Click on search plugins. You want to install this one, WP Robots Txt. I’ve already installed it. You just install it and activate it. Once that’s done, hover over settings. Click on reading. Then, scroll down. Now you have this new section to the reading area called robots.txt content. This is where you can modify your robots file. I do want to give a warning that you usually don’t want to mess around with this too much. The only reason you would want to change your robots.txt content is if there are pages on your site that you don’t want Google to crawl and index, or if there are duplicate content issues on your site and you want certain pages to be blocked. So, if you have two identical pages you would choose one and add it here to the robots.txt file. Then, search engines wouldn’t crawl it. Later on I’m going to talk about another step you can take to make sure that those pages don’t get indexed. Let me walk you through the language of the robots.txt file. User agent represents the bot that you’re speaking to. In this case it has a little asterisk there, and that speaks to all bots. This is Google bot, Yahoo, Bing, every search engine. You’re basically telling them under disallow you don’t want this page to be indexed and you don’t want this page to be indexed. So, disallow means that you don’t want that page to be crawled by the spiders. User agent, you can change this, you add specific ones, and I’m going to teach you later on in the video how to find the bots that are visiting your site and to see maybe which ones you might want to block. But, in general, you want to keep this like it is. Let’s say you want to add another user agent that you may want specific rules for. So, say in general you don’t want any spiders to index these two pages. You can add a new user agent, like Google bot, and then you can add a new disallow. You can disallow them from accessing, let’s say, Google page. Now, you don’t want them for some reason to index this page. That’s just Google. Bing is okay to index that page. That would depend on how your site is set up. You can also disallow certain file types. You could say *.jpeg and that would block Google from accessing any images on your site.

Вы попали на страничку онлайн видео по фразе How to Optimize Your Robots txt File. Если онлайн видео How to Optimize Your Robots txt File Вам понравилось и Вы остались довольны результатом или наоборот, можете продолжить дискуссию комментарием ниже...

Мир в онлайне