Let's stop polluting the root of a domain!

[1] https://llmstxt.org/

  <link rel="alternate" type="text/markdown" href="https://example.com/foo.md" title="Markdown version of the &lt;Foo&gt; page">

I have created an `llms.txt` for my website anyhow. I use a fixed LLM prompt to generate it from the internal links in `index.md`.

https://code.claude.com/docs/en/overview and

https://code.claude.com/docs/en/overview.md

https://photostructure.com/coding/hugo-markdown-output/

(It includes the grandparent's head link suggestion, but it's not just "change .html to .md" because I'm old skool and as a wee nerd was told that URLs ending in .html or .php or whatever we're frowned on, so the above link's markdown is available by appending /index.md )

And anything else that might tell them not to access something.

If they want it, they will take it, polite directives in text files will have no effect.

The author is on record as trying to remove HTTP 418 "I'm a teapot" support from NodeJS, which resulted in backlash and Python adding support for it.

https://en.wikipedia.org/wiki/Hyper_Text_Coffee_Pot_Control_...

Which bots will then ignore.

Alternative, no SNI required

https://web.archive.org/web/20260619061625if_/https://mnot.n...

Why password-reset instead of a more generic link tree?

Why discord domain verification instead of domain-verifications with a dynamic list on entries?

Seems like a waste of time. I would just define my own spec outside of well known for my use case.

The password-reset well-known endpoint is used by password managers to show a "Change password..." button in their interface, which magically links to the password change page described in that well-known file.

> I guess the more flexible it is, the harder adoption becomes

Yeah. If there is one account management related URL that password managers care about, it’s the change password page. You don’t really need to change email on your account that often, but it is probably a good idea to rotate your password once in a while. So I guess it’s a good idea to make it as easy as possible to adopt – which means just a single URL redirecting to another.

> If the website implements it.

That’s a good catch, though. I guess right now password managers would still have to make a “preflight” request just to see if /.well-known/change-password is implemented before showing it to the user. (But that can go away if most websites adopt it.)

It’s not really a catch? Like robots.txt it’s just something you probe if you have the capabilities to use it. You can just cache the info afterwards.

The TXT record itself is already a dynamic list of entries. It's far simpler and easier to iterate through the list and compare the start of each value with your search string until you find "discord domain verification" directly than it would be to do anything else.

Example:

    ;; ANSWER SECTION:
    ycombinator.com.        300     IN      TXT     "openai-domain-verification=dv-QbhxxK0G0JK0dnyZ4YTsNAfw"
    ycombinator.com.        300     IN      TXT     "v=spf1 include:_spf.google.com include:mailgun.org a:rsweb1-36.investorflow.com include:_spf.createsend.com include:servers.mcsv.net -all"
    ycombinator.com.        300     IN      TXT     "MS=ms37374900"
    ycombinator.com.        300     IN      TXT     "anthropic-domain-verification-0qe2ww=yK576oHdDgyTcXgkPfj1KXgGt"
    ycombinator.com.        300     IN      TXT     "ZOOM_verify_2ndw8KZxSRa8PT8NmdyXvw"
    ycombinator.com.        300     IN      TXT     "google-site-verification=KsI69Y_jEVkp4eXqSQ9R9gwxjIpZznvuvrus6UolB9Y"
    ycombinator.com.        300     IN      TXT     "ca3-4861b957e83847c188e45d04ec314ee3"
    ycombinator.com.        300     IN      TXT     "apple-domain-verification=WG0sP5Alm7N6h1Te"
    ycombinator.com.        300     IN      TXT     "dropbox-domain-verification=asc63coma4mv"
    ycombinator.com.        300     IN      TXT     "google-site-verification=GJKdQskycEclAGPua3yXB9m_nVhxbrsVps_y-t9SXV0"
    ycombinator.com.        300     IN      TXT     "Wayback verify for support request 741082"
    ycombinator.com.        300     IN      TXT     "google-site-verification=rivq8jKu6AADGtbbEzJhmOpcqq08B7QxIzXxYV8DtyU"
    ycombinator.com.        300     IN      TXT     "rippling-domain-verification=a660f7a4ab77a3de"

    for i in $(echo "ycombinator.com 500px.com box.com ebay.com google.com hm.com lenovo.com nordstrom.com realtor.com tmz.com wired.com");do echo -en "${i}:  ";dig +short +nocookie -t TXT "${i}"|wc -l;done|sort -rn -k2
    nordstrom.com:  39
    lenovo.com:  38
    realtor.com:  36
    ebay.com:  36
    hm.com:  34
    box.com:  28
    wired.com:  27
    tmz.com:  22
    500px.com:  17
    ycombinator.com:  13
    google.com:  13

In unbound.conf:

    local-zone: ycombinator.com typetransparent
    local-data: 'ycombinator.com. TXT "[ddos redacted]"'

    dig +short +nocookie -t txt ycombinator.com
    "[ddos redacted]"

Why the echo? "for" should handle a list of terms just fine.

Pedantic assholery aside, genuine question. Is this some sort of shell expansion injection countermeasure of which I am unfamiliar?

And for the record I quite enjoy employing the useless use of cat. It turns pumping a file into a pipeline from a screwball shell meta command into a command isometric to any other command. I sort of wish tee had a "suppress stdout flag" so it could be used more naturally as cat's counterpart.

Would it be mean if I said I do that to expose cat rectum? I used to cat to tac to cat but that was too on the nose. Another fun one is mixed case HtMl elements.

Here's [1] something to play with. not my repo

[1] - https://github.com/bashfuscator/bashfuscator

If your site allows user-created subdomains it shouldn't allow leading underscore. This is reserved somehow.

At the very least, it should be "domain-verification-discord", "-google" and so on. Maybe even "-com.discord", "-com.google"? And the first part clearly standardized and registered, instead of one entity using "domain" and another one "site".

That’s on Discord. They’re not in the registry: https://www.iana.org/assignments/well-known-uris/well-known-...

> Why password-reset instead of a more generic link tree?

[edit: answered in more detail in a sibling thread https://news.ycombinator.com/item?id=48596286]

I did read one before while working with github oidc, and I did find it very useful.

What is it with technical documentations that go deep describing what it is in plenty words but refusing to give a single example? This far from the first case I've ran into either.

I don't know how that can be, since you claim to have found the RFC; the RFC straight-forwardly states,

> 5. IANA Considerations

> This specification updates the registration procedures for the "Well-Known URI" registry, first defined in [RFC5785]; see Section 3.1.

& then of course directs IANA to establish a registry. We'd expect this section, given the very nature of the RFC is that it establishes a collection of things, so that there is an IANA considerations section should be wholly unsurprising…

If you see the linked section…

> The "Well-Known URIs" registry is located at <https://www.iana.org/assignments/well-known-uris/>.

And there's a link to a listing of every standardized .well-known URI there is.

> What is it with technical documentations that go deep describing what it is in plenty words but refusing to give a single example?

The RFC provides an example in the form of "example", but also in the form of "robots.txt" (as a "it could have used this, had this existed", but what else could it have done?).

Here's one I could find: https://accounts.google.com/.well-known/openid-configuration

But how does one even find this?

In the latter case you just probe, for instance if you’re a password manager and you have a password for site A you hit A/.well-known/change-password and if they returns something you can surface a change password link to your user.

The one you found is for OIDC provider discovery (https://openid.net/specs/openid-connect-discovery-1_0.html#P...) so someone tells you they want to log in via Google, you hit that endpoint, and it lets you setup Google as an oidc provider rather without needing to hard-code providers. Even if you just want to support Google as a provider, you hit that and you get the entire configuration rather than have to hunt down the same information in the docs.

That said, I still find it very bizzare that it's so hard to find a tangible example to see how it is in practice.

The rfc has none. Another spec including the use of it has none. In the end only completed service provider/implementers show it.

Before programmatic access happens, it needs to be written by a human. Yet the whole thing feels so human-unfriendly.

Perhaps I am biased robots.txt sets a high bar on how easy it is to find and work with?

It's a JSON array of public keys which you can use to validate a JWT which is what an OIDC token is.

Making it an array means you can rotate keys whenever but the validator is typically caching the public keys.

https://www.hanko.io/blog/understanding-jwks

Actually... found it https://datatracker.ietf.org/doc/html/rfc6749

And here's a PHP implementation that is perfect. https://github.com/thephpleague/oauth2-client

Seriously, what an oxymoronic name. "/index.html" is a well known URL, literally: most of web-developers are aware of it. But inventing a bunch of URLs with predefined semantics and then slapping the "well-known" label on it... well, it won't magically make them actually well-known.

Sure, if everyone knows what they're doing, it's not a problem. But we all know how long that assumption lasts.

I'm not saying it's good or bad how things turned it, but the choice of a dotfile for this sure did not pan out well as the web went the exact opposite direction it would have been relevant in.