Why?
It really depends on what they look like. The problem to be solved by Private Cloud is inference. My assumption is that what you would really want for inference is a chip that’s a tiny bit CPU and mostly NPU. That, by itself - while it’s incredibly efficient and fast for inference - isn’t a great chip for training a model. So they’re not going to be monetizing Private Cloud for model training.
Perhaps the weak sauce MacPro update last year was just a prelude to whatever Apple has up its sleeve for the next one. Perhaps Afterburner'esq daughter cards with NPUs for AI and or VR development.
Their on device models are going to be growing over time that means that scenarios that currently need the cloud will migrate to on device over time. That gets a lot easier when you’re using essentially the same code both locally and on the server. That means Metal as the backbone that orchestrates the neural engines and likely means that anything other than Apple Silicon was never in serious consideration.
I'm not sure I understand why? Isn't the model fundamentally the same when being trained vs when used for inference? And isn't the necessary tensor matrix HW units necessary for both training or inference, with the difference being you need to run it twice or more per pass during training as opposed to only needing to run through the model in the forward direction during inference? Obviously the training data is also going to need multiple passes itself during training, while using the model is a single pass per use. In other words you need more tensor HW to perform more calculations per pass, and per multiple passes, during inference compared to training.
Looking at a die shot for an M3 Max and you see that the chip is like 50% GPU, with as much cache as CPU, and I/O taking up as much space as the CPU + cache, with the NPU being barely larger than the efficiency cores + cache (this assumes the annotation is correct, obviously):
Given how tiny the NPU is (and how relatively inefficient it is to use the GPU), it would make sense to have a larger NPU for training. Take a look at an NVIDIA TU102 chip (similar to the TU104 used in the 2080 RTX, where the TU102 has 12 SMs and the TU104 only has 8):
No. My understanding is that the math for a forward inference pass is different from the matrix math needed for back propagation (training). The Neural Engine isn’t really suited for training whereas the AMX unit (SME now apparently with the M4) is great for training. Would love others to weigh in here.
That’s interesting. Maybe that’s why NVIDIA GPUs work so well since they have so much parallel compute as well as matrix units.
I believe training also wants higher precision than inference, and client inference accelerators are going extremely low precision (int4 etc).
My guess would be there's significant compatibility challenges combining ARMH-licensed cores and custom cores in the same SoC, stuff that's not apparent at the ISA level like internal cache coherency protocols. Not that it's impossible (I think Samsung did it?) but likely a significant headache that they can get away without addressing in a PC chip. Will be interesting to see if they make the attempt for a mobile SoC (or try to market the Nuvia core in mobile at all).
I assume it’s lack of time and resources. I figure in a year or two they will have developed Oryon lite to pair with Oryon v2; a 2.5GHz part, with 4-6 width, smaller buffers and cache, and a quarter the size. It just makes no sense to skip it, as a smaller and lower performance core can take on many tasks and save a lot of power.
But what they’re releasing isn’t a server, it’s a laptop chip. And it’s not like they would need to design a compatible E-core from scratch - Qualcomm has been integrating ARM designs forever.
I’m not talking about redesigning the Oryon core. I’m talking about pairing it with an E core. Or is your contention that the team spent so much time redesigning Oryon that they didn’t have time to integrate it with an E core?
Thanks for that. I haven’t followed this case closely and only know the major details so didn’t know that part.
I suspect this initial release was doing the bare minimum needed to get the work Nuvia had already done into a shipping product ASAP. Since the original design was for servers I’m guessing they didn’t have a system in place for managing heterogeneous cores.
What would suggest is a suitable eCore? An ARM A715?
Yes. The compatibility issues can be incredibly touchy. One bug I read about had to do with different cache line sizes between E-cores and P-cores, and in that case both the cores were compatible implementations of the same ISA. Very, very subtle differences become problems when you need to migrate running code seamlessly vs just having code that can run on different devices.
Seems like they can probably do something like what AMD did and do a synthesizable RTL with different optimizations and libraries. There might also be reasonably modular stuff they can take out, like SIMD units and cache. Though I question the efficiency of E-cores made that way, it's been pointed out that ARMH's E-cores aren't actually all that efficient so potentially more of a die area cost than an efficiency cost.
Not a Qualcomm folk, but everything I've read around those points out that there was no boosting enabled on the poor results. Which is probably just immature firmware, and if they can hit the speeds they're reputed to hit then the performance lines up and is quite pretty.
Slower also tends to be more efficient, so if they’re running artificially slow because of a bug their performance tests look poorer than they should but their battery tests may look better than they should. If/when they update firmware I’ll be interested to see what happens to the battery life tests.
Tom’s makes it sound like a solid first release:
Of course, we’ll wait for the official benchmark results from our lab and the full suite of Copilot+ PC features to give it a score. But all signs point to this being a milestone. This is the start of something big.I just tested Asus’ first Snapdragon X Elite laptop and it (mostly) blows away the MacBook Air M3
This thing toasts the M3 MacBook Air!www.tomsguide.com
Now we wait for Lunar Lake. Exciting times.
Hmmm...I just tested Asus’ first Snapdragon X Elite laptop and it (mostly) blows away the MacBook Air M3
This thing toasts the M3 MacBook Air!
