A new mechanism to help thwart return-oriented programming (ROP) and similar attacks has recently been added to the OpenBSD kernel. It will block system calls that are not made via the C library (libc) system-call wrappers. Instead of being able to string together some “gadgets” that make a system call directly, an attacker would need to be able to call the wrapper, which is normally at a randomized location.
I understood some of these words.
My first impression is that I don’t actually like this very much, why should all applications and languages be dependent upon libc? We’re talking about userspace here and while it’s true userspace can be exploited through return oriented attacks in buggy code, among other things, this security mechanism implicitly hurts applications and low level functionality that is legitimately happening in userspace. They even acknowledge this, but instead of conceding that this harms legitimate use of system calls from user space, they would like to force all languages to use libc….
Also there are many versions of libc, they’re all totally legit. So even then I don’t know how they plan to authenticate them without forcing applications to use a specific one. I’m sure this is something they’ve thought about, but the article doesn’t make it clear if/how they solved it.
Out of curiosity, I’d like to see what impact this has on performance/latency.
Also, there’s something hugely ironic here that may not be immediately obvious, but this kernel restriction only protects from the kind of stack corruption that languages like go are impervious to. C programs are historically among the worst offenders for these kinds of vulnerabilities, yet it’s alternative languages like go that will be forced to embrace C’s libraries. Go figure.
There is only one “legit” libc on OpenBSD and you’re expected to use it.
even when I’m not writing in C?
Yes. Even linked article talks about Golang, which will have to switch to libc syscall wrappers after discussed change is implemented.
Lazarus,
I disagree semantically, all the libcs are legit.
https://www.etalabs.net/compare_libcs.html
However you could say there’s only one “official” libc. There could be differing opinions about whether it’s better to force us to use only the official libc, however ruling by authority kind of makes openbsd less “open” IMHO.
zdzichu,
That’s why I quoted it 🙂
There’s no question that’s the rule De Raadt wants to impose, however the question is whether it’s a good rule to impose on languages that don’t necessarily suffer from C’s chronic security problems. It’s not just golang, but any language that naturally doesn’t require libc to be used as a wrapper.
Perhaps a compromise would be to only enable the libc syscall enforcement for processes that include libc. Or give software a means of explicitly enabling or disabling these checks before they get locked in. The rational behind making this check elective is that this security mechanism does not protect from malicious processes and it must not be mistaken for something that can. The mechanism only helps legitimate processes protect from their own memory bugs. The authors are in the best position to know whether direct syscalls are intentional and helpful or unintentional and detrimental. So the best policy may be to give authors and users a way to override the checks when they’re not desired.
Given that it’s OpenBSD, they would probably tell you they aren’t particularly interested in being everything to everyone, and point out other projects which maybe better aligned with your goals. They don’t waste a lot of time worrying about what other people think, and they are very focused on their mission.
OpenBSD is a security research OS. Nothing more nothing less. It’s useful for lots of things, but ultimately, it’s a giant experiment in securing a C based operating system.
I can see this as closing off attack vectors. They would have to audit languages in ports versus just auditing libc which is in the base OS and already heavily audited by the OpenBSD team. Securing ports is one of the problems they face, and probably the reason for the change.
This is the conclusion that lead to things like pledge and veil instead of OpenBSD implementing a MAC on the FS. The devs decided guessing what a program was supposed to do was like wack-a-mole, so they went a different direction and pushed the effort back on the software devs.
I have no clue what their long game is, but I can see this being deprecated in favor of something else like eBPF which filters syscalls in a few years. Or it’s the first step in allowing other languages, like Rust, into the base. I’m not sure. We’ll see how it plays out.
Flatland_Spider,
Yes of course De Raadt’s entitled to think and do whatever he wants, but it doesn’t make opposing opinions any less valid.
Auditing libc is insufficient though. All this change does it prevent software from invoking syscalls directly. I said it already, but at best this helps to mitigate memory bugs in non-mallicious programs, but it does NOT help to mitigate malicious programs in general. Same goes for ASLR. These are mechanisms that help prevent hackers from successfully exploiting software bugs, but it does not protect a system from a malicious developer. Insofar as these mechanisms do actually help mitigate software bugs in the wild, then that’s great, however a counterpoint would be to point out that there are sometimes better ways to mitigate these bugs like safer languages and better software practices. Applying arbitrary restrictions in lieu of going for safer practices is somewhat of a “one step forwards, two steps backwards” maneuver. I think it’s reasonable to say these proposed restrictions are merely addressing the symptoms of the problem rather than the root cause. Obviously C is the elephant in the room. Given it’s ubiquity for systems development maybe it is unrealistic to expect change. In this context of C being the defacto underbelly for nearly everything. then perhaps it makes sense to resign ourselves to these kind of awkward software restrictions, but I still maintain that in a more ideal world, where C is just another language rather than the lowest common denominator, mandatory policies like enforcing software to use libc is not really a good approach.
Hmm, if their goal was to promote alternative/safer languages, then enforcing libc dependencies seems contrary to that goal. If anything it looks like C favoritism to me. I’ll admit that I’m partial to evolving operating systems beyond C, which is part of why I wince at this measure. Although I’m aware that evolving beyond C may be wishful thinking for systems development. We’ll see how it plays out for sure, but I predict C’s incumbency to outlive myself.
Indeed, which is why there are other projects. 🙂
Yes, which forces all software through approved channels which have been hardened.
There was an interesting comment on Lobste.rs about how AIX and Windows disallow raw syscalls, so this is not entirely unprecedented.
https://lobste.rs/s/6a6zne/some_reasons_for_go_not_make_system_calls#c_tlstk8
Yes, yes, and yes. This is all about mitigations rather then being preventative.
Right, when the bad actor is in a position of trust it’s hard to prevent abuse.
I can see this as being the first step in something more. Cut off all access and measure the damage. After that, you figure out the cases where it doesn’t work, and see if something can be designed which is a better all around fit.
I don’t know. I’m not a security researcher, or part of the team, and the best I can do is speculate.
To my knowledge, they do try to figure out the most secure software practices.
As for safer languages… They get asked that a lot, and I don’t really have an answer for that. They’re working with a worst case scenario in C, and they have lots of working C code.
People have speculated it’s a resource thing. If they wanted to use Rust, if would have to be in the base system. They would have to bring the Rust toolchain up to par with the C toolchain, and they would have to audit and maintain both of the toolchains until the C toolchain could be retired.
If the LLVM toolchain sticks around, this might not be a problem. We’ll see.
Yeah, most things degrade into C. Perl, Python, Ruby, PHP, and many more are just sugar on top of C, so it’s relevant.
It’s really only a recent thing that people have been interested in really pushing new system and service languages, which coincidentally coincides with x86 single-thread performance stagnating. We may have finally reached a point where we can push beyond our current computing model out of necessity.
LLVM makes a lot of things possible. It will ingest anything it has a frontend for and turn it into machine code.
LLVM might not be sticking around in the base system. The OpenBSD team might not be happy about LLVM being relicensed to Apache v2, and they might be considering switching to a compiler which only supports C and C++.
We might be getting there with LLVM, and it might finally be time for microkernels! 😀
I’m really been digging the idea of unikernel VMs for services lately. Stuff just stripped down to only the essentials.
Anyway, we’re probably reading more into this there is. OpenBSD is experimenting with ROP mitigations, and they don’t have enough data to engineer a way to make direct syscalls safe that doesn’t look like a MAC.
Fwiw, somebody penned a good write up as to why Go would prefer to not use a libc:
https://utcc.utoronto.ca/~cks/space/blog/programming/GoCLibraryAPIIssues
malxau,
Thanks for that link!
He mentions libc’s use of global variables like errno, which doesn’t play well with multithreading without thread local storage workarounds.
I hadn’t thought about stack size, but that’s an apt point in languages that support many threads & coroutines. In my own async library I faced a similar problem since my threads only needed a tiny stack by themselves, however stack requirements blew up when I needed to call gethostbyname. A second problem is that libc’s resolver is not asynchronous. I ended up creating a new thread pool for those calls. I noticed that software like nginx skip libc’s resolver altogether and just wrote their own instead, which feels like “not invented here syndrome” until you encounter those issues yourself, haha.
The stack size point seems very relevant. It’s interesting to observe that the whole point of an abstraction is to reserve the right to make changes, but making changes impacts the stack size that will be needed, so the abstraction is leaky. I’ve never seen a platform make any kind of statement about the stack needed when calling into libc. And a more academic observation about why libc is different to kernel is because the kernel will execute its work on a kernel stack, but libc is executing its work on the caller’s stack. There’s a hypothetical alternate universe out there where stacks are switched at a different layer and/or multiple layers, so libc’s stack requirements become libc’s problem.
Since I’m already in the philosophical realm, it’s worth noting that the OpenBSD change is really creating a security distinction between the user program and libc, which exists in the same process address space. It naturally begs the question about how long until more constraints are needed across the program/libc boundary, where things like protecting libc’s stack from the program become important.
malxau,
As you can guess, I don’t like the idea of libc having privileges over other libraries, but to the extent that it will, you pose a good question. To consider libc secure, all of libc’s internal state would need to be protected.
I don’t see a great way to do this without costly page table swapping. Also, libc would likely require substantial re-engineering/vetting to use private memory and IMHO it would not be worth it.
There’s potentially an old-school solution to this by giving libc it’s own descriptor table such that libc could use a special x86 segment/selector for private data, but obviously AMD deprecated selectors in 64bit, so this approach would no longer be possible.
I’ve always thought that it would be a good idea to cleanly separate the system call library from libc, but I haven’t seen any Unix-like OS that does that. AFAIK UX/RT, the OS I’m writing, will be the first to cleanly separate the two. Significant parts of the filesystem API will be implemented in the system call library due to the microkernel architecture, meaning that using raw system calls would mean reimplementing significant parts of the system call library. The seL4 microkernel that I’m using doesn’t even have stable system calls, so the system call library will be tied to the particular kernel version and will only be available in dynamic form. Because of this, I was planning to include some kind of mechanism to prevent the use of raw system calls.
Basically all other libraries including libc will be statically linkable though. All NSS functionality will be implemented in daemons with filesystem interfaces, rather than libraries (this means that asynchronous DNS resolution will basically be available for free).
andreww591,
Yeah, a daemon would be better than userspace libraries for DNS resolution. I feel services in userspace libraries is a weak design for both robustness as well as security. For example, the DNS resolver can’t have it’s own permissions over standard processes. It’s part of the reason why, on unix, so much of the system configuration has to be world readable. It really should not be. If you want to log DNS, that log would have to be world-writable. If you want to have a DNS cache, that cache has to be world readable and writable. There’s no way to protect those resources unless you use a separate daemon for them! It’s not just DNS either, some of the email agents have the same issue where the configuration including credentials have to world readable. It’s a poor design and IMHO security necessitates the use of separate daemons. The shared library should be little more than a front end for the daemon.
Flatland_Spider,
Well, I don’t necessarily disagree with that, but IMHO the biggest challenges aren’t as technical as they are existential. Alternatives face a perpetual struggle for critical mass, which is significantly impeded by mass consolidation around incumbent technologies. I don’t really have an answer for this…maybe a big player could break with it’s own traditions, but after several decades of watching I only expect more of the same with the main exception being new markets where legacy considerations are a non-issue.
It sort of feels like the opposite side of the spectrum, removing all restrictions and containing inside a VM. Personally I’m a proponent of using managed languages even for the kernel itself, like with singularity:
https://en.wikipedia.org/wiki/Singularity_(operating_system)
We discussed these ideas on osnews years ago even before I had heard of singularity!
I hear what you are saying, they can do what they want, but I would hope users can disable mitigations where they don’t add protection and get in the way of legitimate userspace calls.
Yeah, I’m not optimistic about replacing monolithic kernels overnight, especially since the dominant Unix-like OS couldn’t care less. Slowly, things could change. Like how everyone migrated to a world where normal machines have 4+ CPUs from a world where 2 CPUs was exotic.
If anything is going to push people to move it’s the pursuit of performance. This doesn’t mean much to a desktop or mobile user, but it means a lot to servers and the computing fringe. Physics has hard limits. Hardware advances are no longer free, electricity is a finite resource, and it’s back to optimizing the software to get the most out of COTS hardware.
Redox is the one I’m most interested in. Unix-like microkernel written in Rust. 🙂 https://www.redox-os.org/
Unikernals are hyper optimization. 🙂 Strip everything down to the bare minimum since something that isn’t there can’t be exploited. Next, weld the programming language into the kernel and communicate with everything via network calls.
My take on the whole thing is that we’re clinging to an outdated model that assumes interactive computing is the dominant paradigm. We shouldn’t be designing systems where human interaction is a requirement. Systems should be programmed and controlled remotely. Scrap all of the junk that assumes a human is on the other side of the connection, and take Infrastructure as Code to it’s logical conclusion. This would allow us maximum freedom to change the underlying OS as needed since people shouldn’t get twisted about specifics since the data would get formatted however they like.
I will admit my view is heavily skewed towards servers and deploying services. All the provisions for interactive computing is legacy cruft I can’t get rid of.
The only reason I ever wanted to use direct syscalls on Linux is for portability when using new calls not necessarily in every libc version.
Carewolf,
I too have faced that requirement. Also sometimes in low level code like assembly it’s actually easier to call syscalls directly than going through undesirable layers, although I’m more interested in the overhead differences. I’ll concede this does not necessarily reflect on openbsd, but just now I benchmarked a few trivial programs on linux.
100M iterations of each of the following, average of 5 runs…
Granted synthetic benchmarks don’t necessarily measure real world bottlenecks, but it could be worthwhile to test additional calls. I didn’t dig any deeper to see what’s happening with poll, nevertheless this shows that there can be a benefit going directly to syscalls and avoiding the standard C wrapper library. Also, linux does not have code to enforce all syscalls originate from libc code, adding that code can only make the performance worse. I’m curious how much additional overhead this check on every syscall will cost openbsd. Hypothetically if it were to add another 1% overhead, does anyone think that’s worthwhile to enforce the use of libc over syscalls?
Ultimately, I have no say so nothing I think matters, but if I were writing my own language standard library, say for alflang, I’d be inclined to call syscalls directly from my library as in program->alflib->syscall rather than program->alflib->clib->syscall. Or possibly even inlining the low level calls transparently for the programmer to avoid all of the wrapper overhead program->syscall.
Doh, my parameters to poll were incorrect, I fixed it and the results are more inline with what’s expected.
It’s not that much overhead, but if you’re writing your own standard library, these are relatively low hanging fruit and optimizing it could benefit all users of your library. In particular though, I think one would be justified in avoiding libc especially for non-C development.
Wouldn’t you get like 99.99% of the benefit of this change and none of the drawbacks just by verifying the return address from the syscall was word aligned, at least on platforms where that’s a sensible assumption (and not already enforced by the CPU.)? Or at least make it an ELF section flag that this section may make syscalls so the loader can enforce the prohibition on the rest. It seems like they had the grand idea “Only libc may make syscalls!” and now they’re going through a protracted process of “weeellll…. what if….” that’s going to make the code that enforces this more of a problem than the problem they’re trying to fix.