mainSavannah Administration - Support: sr #111187, Webpages hosted on www.gnu.org or...

 
 

sr #111187: Webpages hosted on www.gnu.org or www.nongnu.org unreachable from GitHub infrastructure

Submitter:  EricFromCanada <ericfromcanada>
Submitted:  Thu 06 Feb 2025 04:03:47 PM UTC
   
 
Category:  Project webpages Priority:  5 - Normal
Severity:  3 - Normal Status:  None
Privacy:  Public Assigned to:  None
Operating System:  None Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 06 Feb 2025 09:58:27 PM UTC, comment #5: 

Using a desktop user-agent for a CI/CD check is disingenuous. I know it is the path of least resistance, but currently web hosts are seeing a large number of disingenuous user-agents most of which are AI startups with custom crawlers that are breaking pages. When I see them in the logs, I tend to block them. If you were to append something like (https://brew.sh/ci-bot; mailto: bot@brew.sh) to the user-agent with a page about what it is doing, I would not block such a thing upon manual review. That is the commonly used format that I have seen in the wild.

I probably did not even block homebrew specifically. I probably saw abuse from some other GitHub Action that was not checking annually. I believe the GitHub Actions CIDRs are listed here: https://api.github.com/meta I can compare my firewall rules with those and see what my note was. Fixing this for your annual check will likely open us up to the reason I blocked the address in the first place.

Our webpages are very old and change infrequently. I would like to opt-out of this check.

Michael McMahon <thomzane>
Site Administrator
Thu 06 Feb 2025 08:15:51 PM UTC, comment #4: 

comment #1:

> What is the purpose of this check?


These checks ensure that the formulae in the official "homebrew/core" repository (https://github.com/Homebrew/homebrew-core/) have accurate information.

> What is the frequency of this check?


The scheduled GitHub Action is defined here: https://github.com/Homebrew/homebrew-core/blob/master/.github/workflows/scheduled.yml
It's set to test each formula annually, in blocks distributed throughout the year.
This is what its homepage check looks like when run locally:

$ brew audit --online --skip-style --only homepage binutils --debug
/opt/homebrew/Library/Homebrew/brew.rb (Formulary::FromNameLoader): loading binutils
/opt/homebrew/Library/Homebrew/brew.rb (Cask::CaskLoader::NullLoader): loading binutils
==> Auditing Formula binutils on os sonoma and arch arm
/opt/homebrew/Library/Homebrew/brew.rb (Formulary::FromPathLoader): loading /opt/homebrew/Library/Taps/homebrew/homebrew-core/Formula/b/binutils.rb
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Mozilla/5.0\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 10_15_7\)\ AppleWebKit/605.1.15\ \(KHTML,\ like\ Gecko\)\ Version/17.0\ Safari/605.1.15 --header Accept-Language:\ en --connect-timeout 15 --max-time 25 --retry 3 --retry-max-time 25 --dump-header - --output /private/tmp/20250206-52585-sylsg6 --location https://www.gnu.org/software/binutils/binutils.html


> Why does this check not identify itself with a user-agent pointing to a link describing the purpose for this check?


It uses a desktop user agent to more closely mimic what a user would experience. Are there any standard headers that such status checks should be sending to identify themselves?

> Can this check be disabled for us?


Given this check amounts to fewer than 150 requests annually, I'm hoping that won't be necessary.

EricFromCanada <ericfromcanada>
Thu 06 Feb 2025 06:12:32 PM UTC, comment #3: 

I forgot to login. I wrote comment #1. As mentioned by rwp, the blocks are due to significant abuse seen since August. Related article: https://www.fsf.org/bulletin/2024/fall/fsf-sysops-cleaning-up-the-internet

Michael McMahon <thomzane>
Site Administrator
Thu 06 Feb 2025 05:03:48 PM UTC, comment #2: 

I am going to make a comment so that you know that the Savannah team side has read your problem report and has responded but I do not believe there is anything for Savannah admins to do here. Noting that Savannah and all of the public servers are under a big botnet abuse wave right now and IP based mitigations blocking it are in effect and some problems have been reported which we then try to manage. I have passed a pointer to this ticket over to the web team admins managing www.gnu.org for comment.

Bob Proulx <rwp>
Site Administrator
Thu 06 Feb 2025 04:59:15 PM UTC, comment #1: 

Hi, Eric!

What is the purpose of this check? What is the frequency of this check? Why does this check not identify itself with a user-agent pointing to a link describing the purpose for this check? Can this check be disabled for us?

I probably blocked this. I see a considerable amount of checks that are unidentified like this. If it was just homebrew, it wouldn't be a problem and I would not have blocked it in the first place.

Anonymous
Thu 06 Feb 2025 04:03:47 PM UTC, original submission:  

Homebrew (brew.sh) does automated, periodic checks of its formulae (packages), including testing the presence of homepages. These checks are run on GitHub infrastructure, and any formulae with homepages hosted on www.gnu.org or www.nongnu.org consistently fail the check since the server at 209.51.188.116 does not respond. For comparison, savannah.gnu.org and savannah.nongnu.org (209.51.188.113) are unaffected, which leads me to believe the block is accidental rather than intentional.

Related issue: https://github.com/Homebrew/homebrew-core/issues/206757

EricFromCanada <ericfromcanada>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by thomzane (Posted a comment)
  • -email is unavailable- added by rwp (Posted a comment)
  • -email is unavailable- added by ericfromcanada (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.14-7003.
    Corresponding source code