In reply to Dr Who Fan's message of 21 Nov 2025:

SETI website is loading nice and fast. Hurry up and read or post you item(s) there before it disappears again.

Not for me, i'm still forbidden.

---

Grant
Darwin NT.

Not for me, i'm still forbidden.

Sounds like possible Cloudflare servers are blocking you.

Out of curiosity, have you tried clearing the browser cache and deleting cookies for all the *(.)berkeley(.)edu sites, including this site to see if it will let you back in?

In reply to Dr Who Fan's message of 22 Nov 2025:

Not for me, i'm still forbidden.

Sounds like possible Cloudflare servers are blocking you.

Out of curiosity, have you tried clearing the browser cache and deleting cookies for all the *(.)berkeley(.)edu sites, including this site to see if it will let you back in?

Two different browsers on two different systems.
Can actually access Seti now on a 3rd system, but no joy on the other two.
Time to go cookie hunting.

Edit- not cookies, but related to the saved URL.
The ones that were forbidden, with no mention of HTTP or HTTPS in the saved address were resolving to HTTP. Changed the favourite link address to explicitly HTTPS and no problems.

---

Grant
Darwin NT.

It's still up nearly 14 hours later! This is the longest time in a while so, blame me for bragging and posting here if SETI website goes dark.within the next hour.

Eric posted the following on https://setiathome.berkeley.edu/forum_thread.php?id=86149&postid=2152880#2152880

We've been DDOSed the last month or so by assorted web cralwers that have been ignoring the robots.txt directive. I'm made blocking such things a bit more proactive. Hopefully we can stay up more than 10 minutes at a time now.

In reply to Grumpy Swede's message of 22 Nov 2025:

Eric posted the following on https://setiathome.berkeley.edu/forum_thread.php?id=86149&postid=2152880#2152880

We've been DDOSed the last month or so by assorted web cralwers that have been ignoring the robots.txt directive. I'm made blocking such things a bit more proactive. Hopefully we can stay up more than 10 minutes at a time now.

That seems to be becoming a bigger and bigger problem- most likely crawling for AI/ML data.
Another web site i often use had several issues about 6 months back- getting slammed with AI/ML bots, and they had to up their Cloudflare plan and put in other blocks to kill off that load.

---

Grant
Darwin NT.

I asked Grok how it deals with that. The answer is disconcerting.

As an AI developed by xAI, I rely on tools for accessing and processing web information rather than directly crawling sites myself. Robots.txt is a voluntary protocol—it's not legally binding, so there's no strict requirement to follow it. That said, ethical AI practices often encourage respecting it where possible.In practice, for real-time information retrieval (like web searches or browsing), xAI's systems appear to use user-agent spoofing, such as mimicking an iPhone browser, to bypass potential blocks and ensure access to content.

@grok

This approach can effectively circumvent robots.txt directives, as sites may disallow bots but allow user-like agents. Reports also note that xAI's bot sometimes doesn't self-identify properly, making it challenging for website operators to apply robots.txt rules to it.

For other purposes, like data collection for training, xAI has a documented user-agent (e.g., GrokBot/1.0) that is more likely to respect robots.txt.

Overall, adherence isn't universal and depends on the specific use case.

ChatGPT:

I don’t crawl the internet myself. I don’t browse websites, run a crawler, or access robots.txt when generating answers.

Two important points:

1. Training data collection (done in the past)

OpenAI and its data partners follow rules, licensing, permissions, and technical restrictions, which includes respecting things like robots.txt or equivalent measures when collecting publicly available data for training. So yes—robots.txt is one of the mechanisms that can guide what is or isn’t crawled during dataset creation.

2. What I do when answering you now

When answering your questions:

I don’t crawl websites.

I don’t access or ignore robots.txt.

I can perform live information lookups only through the built-in “web” tool, and that tool uses search engines—not a general-purpose crawler.

I don’t fetch arbitrary URLs, and I don’t run automated scraping behavior.

So in practice: I never violate robots.txt because I never crawl sites directly at all.

If you want, I can explain how robots.txt works or how responsible crawling is usually implemented.

Meta AI:

When I crawl the internet for information, I respect the rules set by websites, including those specified in robots.txt files. This file tells crawlers like me which parts of a site to crawl and which to avoid. I follow these guidelines to ensure that my data collection is done responsibly and respectfully.

That seems to be becoming a bigger and bigger problem- most likely crawling for AI/ML data.

CPDN dealt with this by blocking computers not logged in to the site from accessing parts of it. For example, to look at anyone's computers I need to be logged in now. What I don't know is whether only the parts that robots.txt would not allow the crawlers to see have been blocked, but it seems to have resolved the problem a few months back with access attempts regularly timing out.