We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year.

I'm sure there's other things happening in the background, but I can not help but believe that this is directly correlated with the increase of LLM usage.

Though I would love to hear someone else's pet theory how a rock of the internet went from four+ nines of uptime to maybe one.

As someone who has spent many years working in high load environments, this is not an uncommon pattern.

You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.

And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.

This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.

It could be that the extra usage just pushed them past one of those capacity thresholds.

I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration.

I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.

Seems pretty conclusive. Very similar story when they bought skype.

Though I'm sure some of the blame can go to internal slop code.

I thought they should rename to Anglia Busways and have bus replacement trains instead.

We have many agents per employee working in parallel pushing way more commits than was humanly possible before AI, triggering GitHub actions a lot more than the workflows were built for, causing Actions costs to escalate (they really aren't cheap if you compare to hosting it yourself), meanwhile working with YAML workflows is just a pain, and just writing code would be so much more fun and AI compatible[1].

At the same time, GitHub has about ~3 different PR review UIs? And they're all half-bad? Any decently sized PR triggers their "optimized for large PRs" UI which jumps around randomly in my experience. If you don't get that UI and keep the scrolling one (there's an old and a new one btw) then god forbid you click a line number because at some point your browser will randomly scroll back to that line and it won't unstick. Now Linear[2] (and others) is replacing the PR review experience for the agentic era.

I'd love to see a solid AI first Git + CI + reviews.

[1] Cloudflare CI https://blog.cloudflare.com/ci-workflows/

[2] Linear PR reviews https://linear.app/changelog/2025-01-23-pull-request-reviews

This is annoying and I'm here because it's down. But it would have to be far worse to come close to actually being worth changing.

If I could right now:

1. go sign-up elsewhere 2. Log into GitHub and point Actions to that new host 3. All my actions files immediately worked without question

I'd probably give it a spin and make a wiki page explaining how to swap back and forth. No meetings. No design issues. No scheduling. Just a flip switch on who to pay for computers.

I think people were so excited to move away from jenkins to something 'managed' just because of how much a dinosaur jenkins is and how much a pain in the ass it is to upgrade it... but now we are seeing how managed can bite you in the ass if the manager is incompetent.

But for people who either don't pay anything at all or phenomenal amount one 9 of up time is all you need.

If you're actually trying to run a business I guess you can call and gitlab and get an Enterprise contract

This is multiple times this month that this has been a problem.

Has GitHub completed it's internal migration to Azure yet? Or is it still ongoing? None of our devs want to switch away from GH, but we will have to at this point.

Step one - migrate my build workflows to Docker.

My Github actions are now basically: "checkout / set env vars from secrets / docker-compose builder run make".

I used large machine runners to run full the Docker (Podman actually) on Github first to avoid dealing with docker-in-docker complications. This step also provided some very nice robustness advantages, as I can now trigger deployments from my laptop if needed.

Step two:

Migrate to self-hosted runners. I used my former homelab server to set up a build machine. It has 16Tb of fast NVMe SSDs and thanks to Podman container layer caching, my entire lint workflow now takes 30 seconds. Faster than just one "npm install" on Github before.

And Github's self-hosted runners are actually surprisingly easy to set up and use. They are also somewhat more robust.

Step three:

Swap Github for something else.

Github is just the laziest default. It's not _terrible_, but it's also not great.

Now I'm stuck twiddling my thumbs with PR checks stuck/failing...

Especially troublesome in the middle of trying to fix a high score security vulnerability when the release vehicle is Github.

Maybe they're hosting in us-east-1 though :)

It was before it became a news and a trend in X.

Really frustrating experience.

There is no better time to self-host.

[0] https://news.ycombinator.com/item?id=22868406