# Search engine crawlers restrictions. # # Copyright (C) 1999, 2000 The SourceForge Crew # Copyright (C) 2000-2006 Mathieu Roy # Copyright (C) 2014, 2016, 2017 Assaf Gordon # Copyright (C) 2015-2017 Bob Proulx # Copyright (C) 2001-2011, 2013, 2017 Sylvain Beucler # Copyright (C) 2013, 2014, 2017-2024 Ineiev # # This file is part of Savane. # # Code written before 2008-03-30 (commit 8b757b2565ff) is distributed # under the terms of the GNU General Public license version 3 or (at your # option) any later version; further contributions are covered by # the GNU Affero General Public license version 3 or (at your option) # any later version. The license notices for the AGPL and the GPL follow. # # Savane is free software: you can redistribute it and/or modify # it under the terms of the GNU Affero General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # Savane is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Affero General Public License for more details. # # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see . # # Savane is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # Savane is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # This file is checked by crawlers only if installed at the root "/" # of the website (it won't be read if you installed Savane in a # subdirectory such as http://yoursite.tld/savane/), so paths in this # file are relatives to the root of the website. User-agent: * # Disallow pages without indexable content. Disallow: /include/ Disallow: /images/ Disallow: /siteadmin/ # Disallow login.php because everypage links to it with a 'uri=' # parameter - this generates lots of useless pages to index, and lots # of traffic for nothing. Disallow: /account/login.php # There's one page per user and it doesn't contain any indexable # content - let's exclude it: Disallow: /sendmessage.php # Disallow those which are hit thousands of time for some reason: Disallow: /bugs/?func=search&group= Disallow: /cookbook/?func=search&group= Disallow: /patch/?func=search&group= Disallow: /support/?func=search&group= Disallow: /task/?func=search&group= # Disallow dynamically generated gpg key pages. Disallow: /project/memberlist-gpgkeys.php Disallow: /project/release-gpgkeys.php Disallow: /people/viewgpg.php # Disallow identical pages - less crawler traffic. Disallow: /*reload=1 # Disallow multi-fields sorts which exhibit thousands of URLs. Disallow: /*func=browse Disallow: /*func=digest # Don't index spam. Disallow: /*func=viewspam