We are dealing with a hype, but the reality is that AI would change everything we do. Local models will start being helpful in [more] unobtrusive ways. Machines with decent local NPUs would be usable for longer before they feel too slow.
> the reality is that AI would change everything we do
Your true believer convictions don't matter here. Those AI accelerators are merely just marketing stunts. They won't help your local inference because they are not general purpose enough for that, they are too weak to be impactful, most people won't ever run local inference because it sucks and is a resource hog most can't afford, and it goes against the interests of those scammy unprofitable corporations who are selling us LLMs as AI as the silver bullet to every problem and got us there in the first place (they are already successful in that, by making computing unaffordable). There's little to no economical and functional meaning to those NPUs.
> most people won't ever run local inference because it sucks and is a resource hog most can't afford
a) Local inference for chats sucks. Using LLMs for chatting is stupid though.
b) Local inference is cheap if you're not selling a general-purpose chatbot.
There's lots of fun stuff you can get with a local LLM that previously wasn't economically possible.
Two big ones are gaming (for example, text adventure games or complex board games like Magic the Gathering) and office automation (word processors, excel tables).
Your comment is almost completely irrelevant to what the parent is saying. "AI would change everything we do" has nothing to do with "This new chip along with bloat from Windows enables new workflows for you". If you have been paying attention, you'd know that NPUs from these new CPUs barely made any difference from a consumer's perspective.
For some people maybe. I don't want to use local AI and NPU will be dead weight for me. Can't imagine a single task in my workflow that would benefit from AI.
It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction.
> Can't imagine a single task in my workflow that would benefit from AI.
You don't do anything involving realtime image, video, or sound processing? You don't want ML-powered denoising and other enhancements for your webcam, live captions/transcription for video, OCR allowing you to select and copy text out of any image, object and face recognition for your photo library enabling semantic search? I can agree that local LLMs aren't for everybody—especially the kind of models you can fit on a consumer machine that isn't very high-end—but NPUs aren't really meant for LLMs, anyways, and there are still other kinds of ML tasks.
> It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction.
Do you insist that your CPU cores must be completely homogeneous? AMD, Intel, Qualcomm and Apple are all making at least some processors where the smaller CPU cores aren't optimized for power efficiency so much as maximizing total multi-core throughput with the available die area. It's a pretty straightforward consequence of Amdahl's Law that only a few of your CPU cores need the absolute highest single-thread performance, and if you have the option of replacing the rest with a significantly larger number of smaller cores that individually have most of the performance of the larger cores, you'll come out ahead.
None of what I listed was in any way specific to "content creators". They're not the only ones who participate in video calls or take photos.
And on the platforms that have a NPU with a usable programming model and good vendor support, the NPU absolutely does get used for those tasks. More fragmented platforms like Windows PCs are least likely to make good use of their NPUs, but it's still common to see laptop OEMs shipping the right software components to get some of those tasks running on the NPU. (And Microsoft does still seem to want to promote that; their AI PC branding efforts aren't pure marketing BS.)
The issue is that the consumer strongly associates "AI" with LLMs specifically. The fact that machine learning is used to blur your background in a video call, for example, is irrelevant to the consumer and isn't thought of as AI.
Never wanted to do high quality voice recognition? No need for face/object detection in near instant speed for your photos, embedding based indexing and RAG for your local documents with free text search where synonyms also work? All locally, real-time, with minimal energy use.
That is fine. Most ordinary users can benefit from these very basic use cases which can be accelerated.
Guess people also said this for video encoding acceleration, and now they use it on a daily basis for video conferencing, for example.
Those usecases are at least 5 years if not 10 years out. They require software support which won't come until a significant part of the pc market has the necessary hardware for it. Until then, paying extra for the hardware is foolish.
This will only come if Windows 12 requires a TPU and most of the old hardware is decommissioned.
Also similar to GPU + CPU on the same die, yet here we are. In a sense, AI is already in every x86 CPU for many years, and you already benefit from using it locally (branch prediction in modern processors is ML-based).
> Also similar to GPU + CPU on the same die, yet here we are.
I think the overall trend is now moving somewhat away from having the CPU and GPU on one die. Intel's been splitting things up into several chiplets for most of their recent generations of processors, AMD's desktop processors have been putting the iGPU on a different die than the CPU cores for both of the generations that have an iGPU, their high-end mobile part does the same, even NVIDIA has done it that way.
Where we still see monolithic SoCs as a single die is mostly smaller, low-power parts used in devices that wouldn't have the power budget for a discrete GPU. But as this article shows, sometimes those mobile parts get packaged for a desktop socket to fill a hole in the product line without designing an entirely new piece of silicon.
So I’ve got a lot warmer to believing that AI can be a better programmer than most programmers these days. That is a low bar :). The current approach to AI can definitely change how effective a programmer is: but then it is up to the market to decide if we need so many programmers. The talk about how each company is going to keep all the existing programmers and just expect productivity multipliers is just what execs are currently telling programmers; that might change when the same is execs are talking to shareholders etc.
But does this extrapolate to the current way of doing AI being in normal life in a good way that ends up being popular? The way Microsoft etc is trying to put AI in everything is kinda saying no it isn’t actually what users want.
I’d like voice control in my PC or phone. That’s a use for these NPUs. But I imagine it is like AR- what we all want until it arrives and it’s meh.
In my team, I've been working to help everyone do a task that's necessary, but because it was too difficult, people bypassed it. Over time, I made it simpler, others are joining to make it even simpler, but in the process, I'm not doing as many "feature tasks" as I could. I joke that people are mad at me every day, but grateful every week and month.
I had to stop trying to prove myself to the company. I have already done that when y'all interviewed me. Now I do what's best for everyone, and I want the company to prove to me that it deserves people who do the right thing despite the processes not valuing it. If it does not, I have enough resources to spend some time on the projects I cared about most.
This mentality gave me peace of mind and helped many people in partner teams go faster and with higher quality.
Management still does not openly appreciate it, but it shows in how they interact with me. Like when you learn to talk to your parents as equals. It's unexpected for them, but they quickly embrace the new interaction and they love it much more than the one before.
My iPad wins by a large margin the lowest bang for bucks of any competing device I ever purchased
A locked down device with no multitasking, even the browser download is interrupted if I switch away.
The true embodiment of paternalism, it protects you from all the scam in the world, except the noble ones who are willing to pay the 30% share.
It has some multitasking on the latest versions of iOS. You can drag a YouTube window up by the bottom right corner and open a Reddit window on another part or the screen if you truly need to be distracted from distraction by distraction.
From your comment I don't know you have tried this or not, but get some sessions with the best trainer you can. Singing and even speaking with a good voice is incredibly counterintuitive for some people, speaking from experience.
You might have everything needed for a great voice expect the skills, or you might be trying for a voice that's far from what works well for you.
You gave up some convenience to avoid voting for a bad practice with your wallet.
I admire this, try to consistently do this when reasonably feasible.
Problem is, most people don't do this, choosing convenience at any given moment without thinking about longer-term impact. This hurts us collectively by letting governments/companies, etc tighten their grip over time. This comes from my lived experience.
Society is lacking people that stand up for something. My efforts to consume less is seen as being cheap by my family, which I find so sad. I much prefer donating my money than exchanging superfluous gifts on Christmas.
As I get older I more and more view convenience as the enemy of good. Luckily (or unluckily for some) a lot of the tradeoffs we are asked to make in the name of convenience are increasingly absurd. I have an easier and easier time going without these Faustian bargains.
IMHO The question is: who is in control? The user, or the profit-seeking company/control-seeking government?
There is nothing we can do to prevent companies from seeking profit. What we can do is to prefer tools that we control, if that choice is not available, then tools that we can abandon when we want, over tools that remove our control AND abandoning them would be prohibitively difficult.
It's simple: Lawyers creating market for themselves and other lawyers. A head of legal department at Netflix would have a better job and pay if ge has 50x more employees. Hence, the incentive to find ways to get involved in everything, even if it arguably hurts the company's revenue, let alone the rest of the market.
reply