Thanks for the post, Lars! I'm loving the “Unpacking Elixir” series so far.
I really wish I had been exposed to Erlang/Elixir much sooner in my career. The inherent pragmatism and the overall desire to consolidate as much as possible within the BEAM makes for a very pleasant developer experience.
lawik [3 hidden]5 mins ago
Glad you are enjoying it. I have a pretty solid list of topics to hit still. Hope I maintain the momentum.
Thankfully writing them is kind of straightforward. Topics I've been interested in before and already know enough to write a bit about.
Thaxll [3 hidden]5 mins ago
I don't believe by modern standard that Elixir is aiming at low latency.
59nadir [3 hidden]5 mins ago
You're getting a lot of downvotes but I think it's mostly because of a misunderstanding of the bigger picture. Erlang can provide a low median latency and won't be very sensitive to outliers where many other languages will have schedulers that are much more sensitive to outliers (because they'll steal too many resources and can potentially steal an entire core for too long) while they can still have lower minimum latency.
It's not that hard to accidentally exhaust the thread pool of some runtimes whereas you have to end up in pretty deep pathological areas in Erlang in order to do so. Notably `term_to_binary` on massive structures has historically been mis-bookkept (if you can call it that) by the runtime and will only count as one function call despite its high cost. This means the scheduler can't preempt it and wouldn't be able to nicely schedule N processes (where N is the amount of schedulers you have currently running, i.e. usually your core count) that were executing it.
But again, these are much deeper pathological cases than you'd find causing issues in most runtimes.
keep_reading [3 hidden]5 mins ago
> Goldman Sach uses Erlang in its hedge-fund trading platform for its low-latency (microseconds) event-driven order-submission engine.
if it's good enough for a hedge fund HFT platform, it's good enough for anyone aiming for low latency
It's used for monitoring, not the actual trading. The Erlang scheduler is fantastic, but it targets a context switch roughly every 1ms, it's obviously not suitable for HFT where a process must respond to a network event within microseconds.
> Implemented a real-time monitoring solution for the distributed trading system using a combination of technologies (SNMP, Erlang/OTP, boost, ACE, TibcoRV, real-time distributed replicated database, etc) to monitor load and health of trading processes in the mother-ship and co-located sites so that trading decisions can be prioritized based on congestion and queuing delays.
vgatherps [3 hidden]5 mins ago
This is almost certainly incorrect, unless it’s using a very stretched definition of “hft” and using microseconds to mean 99th% <1ms instead of say <5us (reasonably competitive software range depending on your trade).
They might be forwarding non latency sensitive client/internal orders through a system written in erlang, or using erlang for an orchestration layer, but they’re not competing with other hft traders in one.
Source: I work in the field and have built <5us systems.
Was it HFT or just processing large client orders? I'm having a hard time finding this out
keep_reading [3 hidden]5 mins ago
It could be the C++ for executing trades and BEAM for the client orders. Obviously you'd want something like C++ there when you're racing. Though if they did the trading as a C++ NIF it could all still run on the BEAM avoiding a memory copy. Interesting thought exercise anyway.
Thaxll [3 hidden]5 mins ago
It's hard to find details though.
impulser_ [3 hidden]5 mins ago
Erlang, which Elixir compiles to, is literally built for low latency. It prioritizes latency while almost every other language prioritizes throughput.
vgatherps [3 hidden]5 mins ago
Erlang is built for easily achieving fairly low and reliable latencies, with easy distribution/concurrency primitives, and building highly reliable systems.
It’s not very good for “make packet in to packet out as fast as possible at the cost of all else”. All of the abstraction layers come at a performance cost and aren’t all useful in the first place for low latency trading systems.
lawik [3 hidden]5 mins ago
Yeah, I think I adressed this in the article and I hope it came across. If latency is the one singular goal the design should be quite different.
edvinbesic [3 hidden]5 mins ago
Is it fairer to say that Elixir compiles to Erlang or that Erlang & Elixir compile to Beam byte code?
Genuine question.
di4na [3 hidden]5 mins ago
Neither but the former is closer to it.
swixmix [3 hidden]5 mins ago
"Elixir compiles into BEAM byte code (via Erlang Abstract Format). This means that Elixir code can be called from Erlang and vice versa, without the need to write any bindings. All Elixir modules start with the Elixir. prefix followed by the regular Elixir name."
Yes, the Erlang Abstract Format is at least half a dozen IR above the bytecode, and lowering this is done by the erlang compiler.
lawik [3 hidden]5 mins ago
I think I address what it aims for quite thoroughly in the text itself.
tiffanyh [3 hidden]5 mins ago
Would you mind elaborating in more detail why.
jnsaff2 [3 hidden]5 mins ago
I guess this is similar to me going to a wine store. I am readily persuaded that a good 20 euro wine is actually much more enjoyable for me than a 5 euro wine. But I get no additional enjoyment from 500 euro wine over that 20 euro wine.
So it depends where you are looking from.
If you are a ruby shop and your regular out of the box latency is in the hundreds of milliseconds which under load falls apart into at first some and then all taking seconds or tens of seconds then the BEAM is fantastic, you go down to single-double digit responses and under load the latencies start to grow but very gradually. Every request gets more or less same latency.
When you come from microsecond latencies then it looks like the BEAM is doing a lot but is mostly in your way in achieving the lowest latency possible.
Completely different problem sets which gets confused by the writer not giving the necessary context for their claim.
In the Nerves/IoT/HW space one pattern used is to use Elixir for all the soft-realtime stuff and enjoy the QoL and other benefits of the BEAM and use other tools more suitable for hard-realtime tasks.
HN.zip
I really wish I had been exposed to Erlang/Elixir much sooner in my career. The inherent pragmatism and the overall desire to consolidate as much as possible within the BEAM makes for a very pleasant developer experience.
Thankfully writing them is kind of straightforward. Topics I've been interested in before and already know enough to write a bit about.
It's not that hard to accidentally exhaust the thread pool of some runtimes whereas you have to end up in pretty deep pathological areas in Erlang in order to do so. Notably `term_to_binary` on massive structures has historically been mis-bookkept (if you can call it that) by the runtime and will only count as one function call despite its high cost. This means the scheduler can't preempt it and wouldn't be able to nicely schedule N processes (where N is the amount of schedulers you have currently running, i.e. usually your core count) that were executing it.
But again, these are much deeper pathological cases than you'd find causing issues in most runtimes.
if it's good enough for a hedge fund HFT platform, it's good enough for anyone aiming for low latency
https://medium.com/hybrid-cloud-engineering/beam-otp-on-ocp-...
https://hackernoon.com/successful-companies-use-erlang-and-e...
> Erlang is used as part of the real-time monitoring solution for this distributed trading system.
http://zerohedge.blogspot.com/2009/07/is-case-of-quant-tradi...
> Implemented a real-time monitoring solution for the distributed trading system using a combination of technologies (SNMP, Erlang/OTP, boost, ACE, TibcoRV, real-time distributed replicated database, etc) to monitor load and health of trading processes in the mother-ship and co-located sites so that trading decisions can be prioritized based on congestion and queuing delays.
They might be forwarding non latency sensitive client/internal orders through a system written in erlang, or using erlang for an orchestration layer, but they’re not competing with other hft traders in one.
Source: I work in the field and have built <5us systems.
https://m.youtube.com/watch?v=NH1Tta7purM Is a free great watch about the things you have to do to very reliably have such low latencies
It’s not very good for “make packet in to packet out as fast as possible at the cost of all else”. All of the abstraction layers come at a performance cost and aren’t all useful in the first place for low latency trading systems.
Genuine question.
From https://elixir-lang.org/crash-course.html "Erlang/Elixir Syntax: A Crash Course"
So it depends where you are looking from.
If you are a ruby shop and your regular out of the box latency is in the hundreds of milliseconds which under load falls apart into at first some and then all taking seconds or tens of seconds then the BEAM is fantastic, you go down to single-double digit responses and under load the latencies start to grow but very gradually. Every request gets more or less same latency.
When you come from microsecond latencies then it looks like the BEAM is doing a lot but is mostly in your way in achieving the lowest latency possible.
Completely different problem sets which gets confused by the writer not giving the necessary context for their claim.
In the Nerves/IoT/HW space one pattern used is to use Elixir for all the soft-realtime stuff and enjoy the QoL and other benefits of the BEAM and use other tools more suitable for hard-realtime tasks.