Project

General

Profile

Actions
Copy link

Support #8122

closed

Meta data robots twice

Added by Samuel Quevauvillers about 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
02/23/2021
Due date:
% Done:

0%

Estimated time:

Description

Hello,
I have found the 2 topic abouts Metadata header tags.
https://redmine.webtoolkit.eu/boards/2/topics/16644
https://redmine.webtoolkit.eu/boards/2/topics/15203

I have set the meta tag for robots to "index, follow" in the wt_config.xml, but I always get a second meta tag in my header
<meta name="robots" content="noindex, nofollow" />

I get this as final :

<head>
  <meta name="robots" content="all"/>
  <meta name="description" lang="fr" content="Forestimator: outils cartographiques pour la gestion des forêts Wallonnes">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta name="robots" content="noindex, nofollow">

How could I get rid of it ?
Many thanks

Actions
Copy link
 #1

Updated by Korneel Dumon about 3 years ago

  • Status changed from New to Resolved

As the second link says, Wt will only add this second header when it does not recognize your browser as a bot. This can be a bit tricky to test, try something like this:

curl  -H "User-Agent: Googlebot/2.1" http://localhost:8080

The list of bots for which this works is also in wt_config (under <user-agents type="bot">)

Actions
Copy link
 #2

Updated by Samuel Quevauvillers about 3 years ago

Ok thank you but there is still something I don't understand. How can I allow bots to index my site ?
I notice with the the curl cmd that if I run it as bot or not, I always get a <meta name="robots" content="noindex, nofollow">

Actions
Copy link
 #3

Updated by Samuel Quevauvillers about 3 years ago

Ok now this is working, I had to set back the bot list in my wt_config.xml : 

<user-agents type="bot">
            <user-agent>.*Googlebot.*</user-agent>
            <user-agent>.*msnbot.*</user-agent>
            <user-agent>.*Slurp.*</user-agent>
            <user-agent>.*Crawler.*</user-agent>
            <user-agent>.*Bot.*</user-agent>
            <user-agent>.*ia_archiver.*</user-agent>
            <user-agent>.*Twiceler.*</user-agent>
            <user-agent>.*Yandex.*</user-agent>
            <user-agent>.*Nutch.*</user-agent>
            <user-agent>.*MJ12bot.*</user-agent>
            <user-agent>.*Baiduspider.*</user-agent>
            <user-agent>.*Ezooms.*</user-agent>
            <user-agent>.*Sogou web spider.*</user-agent>
            <user-agent>.*AhrefsBot.*</user-agent>
    </user-agents>
Actions
Copy link
 #4

Updated by Roel Standaert almost 3 years ago

  • Status changed from Resolved to Closed
Actions
Copy link

Also available in: Atom PDF