{"id":677,"date":"2023-02-04T18:41:13","date_gmt":"2023-02-04T18:41:13","guid":{"rendered":"https:\/\/www.viu-data.com\/?page_id=677"},"modified":"2025-04-30T07:32:41","modified_gmt":"2025-04-30T07:32:41","slug":"robots-txt-pre-check","status":"publish","type":"page","link":"https:\/\/www.searchviu.com\/en\/robots-txt-pre-check\/","title":{"rendered":"Robots.txt pre-check"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-page\" data-elementor-id=\"677\" class=\"elementor elementor-677\" data-elementor-post-type=\"page\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-6213c95 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"6213c95\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;,&quot;shape_divider_bottom&quot;:&quot;split&quot;,&quot;shape_divider_bottom_negative&quot;:&quot;yes&quot;,&quot;ekit_has_onepagescroll_dot&quot;:&quot;yes&quot;}\">\n\t\t\t\t\t<div class=\"elementor-shape elementor-shape-bottom\" aria-hidden=\"true\" data-negative=\"true\">\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 1000 20\" preserveAspectRatio=\"none\">\n\t<path class=\"elementor-shape-fill\" d=\"M519.8,0.2c-11,0-19.8,8.5-19.8,19c0-10.4-8.8-19-19.8-19L0,0v20h1000V0.2H519.8z\"\/>\n<\/svg>\t\t<\/div>\n\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-a56c83c\" data-id=\"a56c83c\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-a264cb8 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"a264cb8\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;ekit_has_onepagescroll_dot&quot;:&quot;yes&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-16a4ca5\" data-id=\"16a4ca5\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-265bf4a elementor-widget elementor-widget-heading\" data-id=\"265bf4a\" data-element_type=\"widget\" data-e-type=\"widget\" data-settings=\"{&quot;ekit_we_effect_on&quot;:&quot;none&quot;}\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h5 class=\"elementor-heading-title elementor-size-default\">Use case<\/h5>\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-a2322e7\" data-id=\"a2322e7\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-c828a85 elementor-widget elementor-widget-text-editor\" data-id=\"c828a85\" data-element_type=\"widget\" data-e-type=\"widget\" data-settings=\"{&quot;ekit_we_effect_on&quot;:&quot;none&quot;}\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<div class=\"wpb_text_column wpb_content_element vc_custom_1602597132967\"><div class=\"wpb_wrapper\"><div id=\"custom_post_widget-4525\" class=\"content_block\"><div class=\"wpb_text_column wpb_content_element vc_custom_1602599897715\"><div class=\"wpb_wrapper\"><div id=\"custom_post_widget-4523\" class=\"content_block\"><h4>Ensure that changes to the robots.txt file lead to the desired outcome and that no traffic or revenue are lost.<\/h4><\/div><\/div><\/div><div class=\"wpb_text_column wpb_content_element \"><div class=\"wpb_wrapper\"><p>There is often an extremely high number of URLs, especially with large websites or websites with a lot of dynamic content. This means that Google no longer crawls all URLs and cannot index important sub-pages. To make optimal use of the \u201ccrawl budget\u201d, it often makes sense in such cases to exclude pages via robots.txt.<\/p><p>It should be ensured that no relevant pages are excluded.<\/p><\/div><\/div><\/div><\/div><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-9166818 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"9166818\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;ekit_has_onepagescroll_dot&quot;:&quot;yes&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-2adf7c0\" data-id=\"2adf7c0\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-f0211c9 elementor-widget elementor-widget-heading\" data-id=\"f0211c9\" data-element_type=\"widget\" data-e-type=\"widget\" data-settings=\"{&quot;ekit_we_effect_on&quot;:&quot;none&quot;}\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h5 class=\"elementor-heading-title elementor-size-default\">This is how we can help<\/h5>\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-cda01a2\" data-id=\"cda01a2\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-6b1622a elementor-widget elementor-widget-text-editor\" data-id=\"6b1622a\" data-element_type=\"widget\" data-e-type=\"widget\" data-settings=\"{&quot;ekit_we_effect_on&quot;:&quot;none&quot;}\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<div class=\"wpb_text_column wpb_content_element vc_custom_1602597132967\"><div class=\"wpb_wrapper\"><div id=\"custom_post_widget-4525\" class=\"content_block\"><p>Before actually changing the robots.txt, we crawl the website with the new version stored with us and compare the crawl with the last live crawl of the website. So we can, among other things, check the following structural factors:<\/p><ul><li>Can content be indexed further with clicks or impressions \u2013 if not, which ones?<\/li><li>Have important distribution pages been omitted, and is content with clicks or impressions no longer linked internally?<\/li><li>Are there externally linked pages that are no longer crawlable for the Googlebot?<\/li><li>Have relevant pages fallen behind in the internal link structure?<\/li><\/ul><\/div><\/div><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Use case Ensure that changes to the robots.txt file lead to the desired outcome and that no traffic or revenue are lost. There is often an extremely high number of URLs, especially with large websites or websites with a lot of dynamic content. This means that Google no longer crawls all URLs and cannot index [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"elementor_header_footer","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"class_list":["post-677","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Robots.txt pre-check - SEO Data Warehouse And Monitoring<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.searchviu.com\/en\/robots-txt-pre-check\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Robots.txt pre-check - SEO Data Warehouse And Monitoring\" \/>\n<meta property=\"og:description\" content=\"Use case Ensure that changes to the robots.txt file lead to the desired outcome and that no traffic or revenue are lost. There is often an extremely high number of URLs, especially with large websites or websites with a lot of dynamic content. This means that Google no longer crawls all URLs and cannot index [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.searchviu.com\/en\/robots-txt-pre-check\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Data Warehouse And Monitoring\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-30T07:32:41+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@searchviu\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.searchviu.com\\\/en\\\/robots-txt-pre-check\\\/\",\"url\":\"https:\\\/\\\/www.searchviu.com\\\/en\\\/robots-txt-pre-check\\\/\",\"name\":\"Robots.txt pre-check - SEO Data Warehouse And Monitoring\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.searchviu.com\\\/#website\"},\"datePublished\":\"2023-02-04T18:41:13+00:00\",\"dateModified\":\"2025-04-30T07:32:41+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.searchviu.com\\\/en\\\/robots-txt-pre-check\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.searchviu.com\\\/en\\\/robots-txt-pre-check\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.searchviu.com\\\/en\\\/robots-txt-pre-check\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.searchviu.com\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Robots.txt pre-check\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.searchviu.com\\\/#website\",\"url\":\"https:\\\/\\\/www.searchviu.com\\\/\",\"name\":\"SEO Data Warehouse And Monitoring\",\"description\":\"SEO Solutions For Teams And Agencies\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.searchviu.com\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.searchviu.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.searchviu.com\\\/#organization\",\"name\":\"searchVIU\",\"url\":\"https:\\\/\\\/www.searchviu.com\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.searchviu.com\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.searchviu.com\\\/wp-content\\\/uploads\\\/2023\\\/02\\\/d3babe41-searchviu-logo2x.png\",\"contentUrl\":\"https:\\\/\\\/www.searchviu.com\\\/wp-content\\\/uploads\\\/2023\\\/02\\\/d3babe41-searchviu-logo2x.png\",\"width\":340,\"height\":340,\"caption\":\"searchVIU\"},\"image\":{\"@id\":\"https:\\\/\\\/www.searchviu.com\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/x.com\\\/searchviu\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/searchviu\\\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Robots.txt pre-check - SEO Data Warehouse And Monitoring","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.searchviu.com\/en\/robots-txt-pre-check\/","og_locale":"en_US","og_type":"article","og_title":"Robots.txt pre-check - SEO Data Warehouse And Monitoring","og_description":"Use case Ensure that changes to the robots.txt file lead to the desired outcome and that no traffic or revenue are lost. There is often an extremely high number of URLs, especially with large websites or websites with a lot of dynamic content. This means that Google no longer crawls all URLs and cannot index [&hellip;]","og_url":"https:\/\/www.searchviu.com\/en\/robots-txt-pre-check\/","og_site_name":"SEO Data Warehouse And Monitoring","article_modified_time":"2025-04-30T07:32:41+00:00","twitter_card":"summary_large_image","twitter_site":"@searchviu","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.searchviu.com\/en\/robots-txt-pre-check\/","url":"https:\/\/www.searchviu.com\/en\/robots-txt-pre-check\/","name":"Robots.txt pre-check - SEO Data Warehouse And Monitoring","isPartOf":{"@id":"https:\/\/www.searchviu.com\/#website"},"datePublished":"2023-02-04T18:41:13+00:00","dateModified":"2025-04-30T07:32:41+00:00","breadcrumb":{"@id":"https:\/\/www.searchviu.com\/en\/robots-txt-pre-check\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.searchviu.com\/en\/robots-txt-pre-check\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.searchviu.com\/en\/robots-txt-pre-check\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.searchviu.com\/en\/"},{"@type":"ListItem","position":2,"name":"Robots.txt pre-check"}]},{"@type":"WebSite","@id":"https:\/\/www.searchviu.com\/#website","url":"https:\/\/www.searchviu.com\/","name":"SEO Data Warehouse And Monitoring","description":"SEO Solutions For Teams And Agencies","publisher":{"@id":"https:\/\/www.searchviu.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.searchviu.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.searchviu.com\/#organization","name":"searchVIU","url":"https:\/\/www.searchviu.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.searchviu.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.searchviu.com\/wp-content\/uploads\/2023\/02\/d3babe41-searchviu-logo2x.png","contentUrl":"https:\/\/www.searchviu.com\/wp-content\/uploads\/2023\/02\/d3babe41-searchviu-logo2x.png","width":340,"height":340,"caption":"searchVIU"},"image":{"@id":"https:\/\/www.searchviu.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/searchviu","https:\/\/www.linkedin.com\/company\/searchviu\/"]}]}},"_links":{"self":[{"href":"https:\/\/www.searchviu.com\/en\/wp-json\/wp\/v2\/pages\/677","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.searchviu.com\/en\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.searchviu.com\/en\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.searchviu.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.searchviu.com\/en\/wp-json\/wp\/v2\/comments?post=677"}],"version-history":[{"count":7,"href":"https:\/\/www.searchviu.com\/en\/wp-json\/wp\/v2\/pages\/677\/revisions"}],"predecessor-version":[{"id":6508,"href":"https:\/\/www.searchviu.com\/en\/wp-json\/wp\/v2\/pages\/677\/revisions\/6508"}],"wp:attachment":[{"href":"https:\/\/www.searchviu.com\/en\/wp-json\/wp\/v2\/media?parent=677"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}