On my laptop with Fedora I just configured Linux to hibernate to disk after 15 minutes of suspend. Powering memory off ensures that bugs like this Debian-specific would not matter.

Plus what Debian extension to Linux tooling does although nice in theory, but in practice if one really worries about cold-boot attacks, then all keys and important documents has to be wiped out from memory, not only LUKS keys.

So hibernating is really the only proper way to protect against cold boot.

This is one of those regressions that's easy to miss because everything still "works." Security bugs often don't announce themselves.

Right! Which is why integration tests for these kinds of features are all the more important.

It was also fun to write, and enabled git-bisecting to isolate the specific kernel refactoring which introduced this bug: https://github.com/NixOS/nixpkgs/pull/532499

While it is certainly an interesting bug, I kinda feel that the title is click bait? Because this `cryptsetup luksSuspend` from what I understood is not really officially supported but an extension done in Debian, so if anything this regression only affected Debian? I am not sure if you can blame the kernel for something that is not supported or even widely tested.

I still find this impressive, and it is nice that we now have a test (NixOSTests BTW are awesome, I agree with OP) to avoid this regression from coming back. But from the title it seems to be a widespread issue, not something that affects only one Distro.

I don't see any other way? When you sleep (suspend to RAM), everything is stored in RAM and is encrypted but the master key is present in kernel memory (if I recall correctly).

However, if you hibernate (suspend to disk) the entire contents of RAM (including the master key) is written/encrypted to disk and the RAM is cleared.

When you wake the machine up you have to re-enter the passphrase to decrypt the master key to re-load disk contents back to memory.

Yes, if you simply suspend your laptop on most stock Linux distributions, then everything including the master key is still kept in memory. But Debian pioneered the (optional) cryptsetup-suspend addon. This issues a luksSuspend command which is supposed to wipe the key from memory, and on resume asks you to resupply your passphrase.

Up to kernel 6.8, this worked as described; starting with kernel 6.9, it silently didn't.

So you would still be asked for a passphrase, even though it's already available?

Exactly. Cryptsetup wouldn't know about the extra copy of the volume key in kernel memory. Which is why, dramatically, it appeared secure ("surely I wouldn't be asked to resupply the passphrase if the volume key is still in memory, right?").

FYI: VeraCrypt is not the defacto encryption software for Windows.

Oh, which one is it?

(You don't mean BitLocker, right?)

Okay, yes, sure. It definitely is the most-used encryption software for Windows.

But I would never trust it a second, being property and known for issues. You likely know that, but for the benefit of others:

38C3 - Windows BitLocker: Screwed without a Screwdriver https://media.ccc.de/v/38c3-windows-bitlocker-screwed-withou... https://www.youtube.com/watch?v=5eNtT2p12cM

veracrypt lost their drivers license so afaik you should avoid it since it cannot update its drivers any longer. didnt see any news about them reacquiring that license

Only if you store your key with Microsoft, which is not required or the default if you're using a local account which I assume most privacy sensitive people are.

Bitlocker can use keys that are local only, but the default for home editions of Windows was to use the online account to back it up.

'Happily' is also a stretch, as they really don't have a choice if served a valid court order.

If you want encryption that is safe from the US government, keys need to be stored in your head. Anything physical is subject to court orders.

for enterprises, where this doesn't really matter, bitlocker is great.

if by "great" you really mean "fine".

It's still brittle, awkward and puzzlingly awful UX despite being the literal standard for the platform.

Compare it to any of the actively maintained alternatives, Filevault for MacOS (which is wonderful and never sends your key to be kept somewhere else) or LUKS on Linux.. heck, even Veracrypt is actually easier to understand and more robust.

Veracrypt is more difficult to set up - whether on one machine or a fleet. Bitlocker is a few buttons in the UI, configurable via Group Policy, and so much more.

What is brittle or awkward?

>if by "great" you really mean "fine".

no, i mean great.

managing a fleet of 100+ laptops with bitlocker is a breeze. its so seemless that the users don't even realize its enabled (i.e. no UX issues, at all).

on the other hand, i am not managing 100+ laptops that use veracrypt. sounds absolutely awful. i've never managed an apple fleet, so i can't speak to that, and will take your word on it.

for personal use, i do not recommend bitlocker (or windows, really), but for already-windows enterprises? absolutely

Managing an Apple fleet is similarly fine, and that includes using any of the MDM tooling that also does key escrow on enterprise Filevault devices.

Does that mean it's not the de facto standard on Windows?

So exactly like FileVault?

Both Intel/AMD CPUs produced in the last 5 years or so support full transparent (to the OS) memory encryption. So cold boot attacks are a thing of the past if you enable this feature (it's typically disabled because it reduces RAM speed by about 0.5%).

I don't have to re-enter my boot password after Sleep, so obviously the encryption key is still in memory.

Obviously your distro isn’t using cryptsetup-luksSuspend.

Correct.

The point being made is: If one isn't re-entering their passphrase after suspend, how are they surprised that the encryption keys are somewhere in memory during suspend?

> The point being made is: If one isn't re-entering their passphrase after suspend, how are they surprised that the encryption keys are somewhere in memory during suspend?

If that was the case for the people using the debian extra secure extension that should have wiped the memory clean then someone would have found this bug much earlier than two years. Their password was required to be re-entered even though the key was still in memory somewhere.

The reason this bug is unexpected is that the user is expecting to have to enter their password (because they expect the key to be wiped on suspend), and then _they are_ asked for their password. But there was a copy of the key elsewhere in kernel memory that was never cleared.

Because debian users with luks-suspend have to re-enter their boot password after sleep.

> Except that, for more than two years, the encryption key remained resident in memory across suspend, leaving it there for the taking by anyone who seized the still-powered laptop.

I don't get it. Obviously, the laptop is locked when it resumes, how is that key "for the taking by anyone"? I'm not saying it is impossible to read out RAM from a locked laptop, but surely not by "anyone".

There are attacks that allow dumping RAM if the device is powered on though and you have physical access. Depending on config it may be very easy (just plug in a dumper over Thunderbolt on USB C and do direct memory access) or hard (freeze and swap physical RAM to an unlocked machine).. but the idea was defense-in-depth here; a well configured device should both be hard to dump RAM on and it should not give encryption keys if an attacker succeeds.

Anyone with physical access. I think it is understandable from the phrase.

There is a common misconception about how lock-screens in general work - they usually just prevents using the current hardware and software as it is to access the current OS. But the disk encryption is the main thing that prevents modification and other kind of access to actual data. And if the disk encryption key is lying in the memory, then effectively, the disk encryption is bypassed if someone can access the machine physically and assuming that there are no sufficient tampering protections in place for that machine.

Anyone with physical access, significant tools, and experience. The FBI has people who can pull data out of memory after freezing the RAM but the average laptop thief doesn’t so how serious this is depends significantly on your threat model. If you’re not a major criminal, bitcoin whale, or intelligence target this is almost certainly academic.

> If you’re not a major criminal, bitcoin whale, or intelligence target this is almost certainly academic.

Thanks, that's what I thought.

> Anyone with physical access. I think it is understandable from the phrase.

Sorry, I'm probably dense, I still don't get it. You steal a laptop, you open it, the screen is locked with a password/fingerprint whatever. How do you read out the RAM from that laptop?

Several options. One is you restart and boot from a live system where you are root, and then dump all memory. This is described in the paper with the witty title "Lest We Remember: Cold Boot Attacks on Encryption Keys":

https://www.usenix.org/legacy/event/sec08/tech/full_papers/h...

Other options: DMA attacks. Also you never know what the Intel Management Engine hidden in your computer is doing. It's running a version of Minix you don't have any control over, and it has full access to memory.

>How do you read out the RAM from that laptop?

the term to look up is "cold boot attack" (https://en.wikipedia.org/wiki/Cold_boot_attack).

tons of cool live demonstrations of how it works on youtube if you've got the 20-40 minutes to spare

Still, this is a pretty crazy definition of "anyone".

And I don't use GUIs, but it doesn't mean I have to be a jerk to people who are happy when their GUI gets better :-).

> That's a you problem. I shutdown my machine when I'm not using it.

"We designed the antennas correctly, you're holding the phone the wrong way."

It's not a good analogy. Something is still on in suspend. Good you can control Linux kernel, but what about all other chips which may be an attack vector?

Except shutting down and hibernate are two actions the user can literally select from the same menu.

I shutdown mine too but only because suspend is still a crapshoot on linux

There will always be more suspend/resume bugs to work through. It varies a lot per device. I feel it's necessary to paint the picture for people who are curious what it means for it to be a crapshoot, so indulge me while I share my experiences.

For work I have a ThinkPad T16 Gen 4 with the newer AMD gfx1151 iGPU. Works great. I have yet to witness any issues with suspend/resume. I suspect this is the case because it is running Ubuntu with Lenovo's own support package. Theoretically, from firmware to kernel, this is all tested and validated by Lenovo, like what certainly happens with every Windows laptop and all of the components that go into them.

I also have a gen 1 Framework 16. I have seen it crash on suspend, but it is pretty rare, so I've just shrugged it off for now. It would be hard to debug, I don't see it every month despite using the thing every day.

All of my desktops currently have perfectly reliable suspend resume, you can slam it all day and all night. The last time I ran into issues was a use-after-free issue in AMDGPU. Pretty alarming, although to be clear it never hit any LTS or vendor kernels that I am aware of. I hit it because I prefer to run the latest kernel on my personal machines.

I have certainly owned laptops where suspend basically didn't work, or it would not stay suspended. I think this mainly went away when I started specifically picking laptops for Linux support.

For Intel iGPUs and dGPUs, the track record has been flawless for me. I have a few of the new Battlemage cards that default to the xe kernel driver and those have been working very well as expected. So that's nice.

I don't think this situation will be fixed until more hardware vendors are taking part in validating their stuff on desktop Linux and keeping track of the kernels. The current Linux model seems to be just dealing with whatever the vendors crap out for Windows, often full of weird ACPI behaviors and buggy firmware. It's not to say that the fault of the problems don't often lie with code in the Linux kernel, but they do not seem to wish to be bug-compatible with Windows and I think that is perfectly reasonable, so for problems that come from essentially broken firmware, it simply is going to need vendors to actually fix their shit.

(And that includes AMD. The drivers are good in some regards, but it's hard to ignore AMD's stability issues even still. At this rate, more of the long outstanding AMD driver issues will get resolved by Claude than AMD engineers... Like with Panel Self Refresh on 7040 iGPU, apparently.)

I am too lazy for that, and I hate that after boot you need to launch everything again.

Suspend to (encrypted) swap might be a good middle ground between you and grandparent. Suspend to memory will (at best) protect your LUKS volume key, but other sensitive data remains.

A couple of years ago, three security researchers from the TU Munich implemented a prototype for also encrypting (most) parts of the memory just before suspend, to address this limitation; but as far as I know, it was not upstreamed or developed further: https://www.sec.in.tum.de/i20/publications/fridgelock-preven...

You can usually change that in the settings of the Desktop environment.

There is no universal support for restoring state between the apps. For example, Terminal won't run the scripts that were running, the browser will not automatically restore the pages etc, some apps might not launch or launch with wrong state.

Gnome desktop environment cannot even remember the position and size of console windows, you are expecting too much.

Bugs happen in all code. The difference is, anyone can fix stuff in open source. Closed source bugs are out of control and must be worked around. Usually by switching to OSS

Of course it's (indirectly) a symptom of that.

What's the alternative? Proprietary closed-source operating systems owned by corps who can be compelled to insert covert backdoors?

If BSD was as popular as Linux it would have the exact same problems.

I wonder if you think other OSes are any different?

TempleOS is the only thing that comes to mind that doesn't fit your description and it's not practically useful.

Any sufficiently large codebase is a mix of ideas and concepts implemented by different people with different priorities over a large timespan and if you can fit the entire thing in your head it's not very interesting or complex.

Qubes OS, the Linux distribution aspiring to offer a reasonably secure operating system, pioneering a "every app runs in a virtual machine" approach in the Linux laptop/desktop space, tracks this at the following issue:

https://github.com/QubesOS/qubes-issues/issues/2890

Windows for ages did not really keep all the code in one repo. There were like a dozen parallel repos for e.g. the shell, kernel, IE, etc. Also every feature was developed on team-level branches; integrating all those branches often caused unexpected bugs.

In my experience any software system (not just operating system) after crossing a certain limit on complexity and age looks exactly as hodgepodge of code pieces thrown together, sometimes from different sources even if developed by one org. All major OSs have long crossed those limits, I believe.

"Mythos, find me a bug in LUKS. I know there is one in there".