Relevant Smarter Every Day video: https://www.youtube.com/watch?v=VPSm9gJkPxU

Stated Clearly also has a great deep dive that I've really enjoyed https://youtube.com/playlist?list=PLInNVsmlBUlSjLSj9yGEKphF0... He actually makes it as a reply to Smarter Every Day.

Holy crap this is a rabbit hole and a half. I know what I’m watching while the robot writes my code tomorrow.

Really nice 3D animations in that video. Something like this, I find quite difficult to comrehend jsut off of a text description.

I hope the other sizes are coming too(9B for me). Can't fit much context with this on a 36GB mac.

It's a MoE model and the A3B stands for 3 Billion active parameters, like the recent Gemma 4.

You can try to offload the experts on CPU with llama.cpp (--cpu-moe) and that should give you quite the extra context space, at a lower token generation speed.

Mac has unified memory, so 36GB is 36GB for everything- gpu,cpu.

CPU-MoE still helps with mmap. Should not overly hurt token-gen speed on the Mac since the CPU has access to most (though not all) of the unified memory bandwidth, which is the bottleneck.

I'll try to use that, but llama-server has mmap on by default and the model still takes up the size of the model in RAM, not sure what's going on.

Try running CPU-only inference to troubleshoot that. GPU layers will likely just ignore mmap.

For sure I was running on autopilot with that reply. Though in Q4 I would expect it to fit, as 24B-A4B Gemma model without CPU offloading got up to 18GB of VRAM usage

Do I expect the same memory footprint from an N active parameters as from simply N total parameters?

No - this model has the weights memory footprint of a 35B model (you do save a little bit on the KV cache, which will be smaller than the total size suggests). The lower number of active parameters gives you faster inference, including lower memory bandwidth utilization, which makes it viable to offload the weights for the experts onto slower memory. On a Mac, with unified memory, this doesn't really help you. (Unless you want to offload to nonvolatile storage, but it would still be painfully slow.)

All that said you could probably squeeze it onto a 36GB Mac. A lot of people run this size model on 24GB GPUs, at 4-5 bits per weight quantization and maybe with reduced context size.

i dont get it, mac has unified memory how would offloading experts to cpu help?

I bet the poster just didn’t remember that important detail about Macs, it is kind of unusual from a normal computer point of view.

I wonder though, do Macs have swap, coupled unused experts be offloaded to swap?

Of course the swap is there for fallback but I hate using it lol as I don't want to degrade SSD longevity.

can you elaborate? you can use quantized version, would context still be an issue with it?

A usable quant, Q5_KM imo, takes up ~26GB[0], which leaves around ~6-7GB for context and running other programs which is not much.

[0] https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF?show_fil...

context is always an issue with local models and consumer hardware.

correct but it should be some ratio of model size like if model size is x GB, max context would occupy x * some constant of RAM. For quantized version assuming its 18GB for Q4 it should be able to support 64-128k with this mac

For the 9B model, I can use the full context with Q8_0 KV. This uses around ~16GB, while still leaving a comfortable headroom.

Output after I exit the llama-server command:

  llama_memory_breakdown_print: | memory breakdown [MiB]  | total    free     self   model   context   compute    unaccounted |
  llama_memory_breakdown_print: |   - MTL0 (Apple M3 Pro) | 28753 = 14607 + (14145 =  6262 +    4553 +    3329) +           0 |
  llama_memory_breakdown_print: |   - Host                |                   2779 =   666 +       0 +    2112                |

Have you ever tried going to the model registry and seeing that the model was recently updated? What updated? What changed? Should I re-download this 20GB file?

I guess if you're not frustrated with things like this then sure, no need to stop using it.

Real Science video on the slug: https://www.youtube.com/watch?v=IH_uv4h2xYM

I thought github was being wonky but yeah, getting 401 Unauthorized error.

Edit: Can't view discussions as well.

You're right, discussions are affected as well. Been checking their status page[0] for an hour now but it seems there's no issue.

[0]: https://www.githubstatus.com/

And they say: https://www.youtube.com/watch?v=IfpMknlL-pg

>Scientists believe people with aphantasia use words or concepts to recall what they've seen.

Like the text-encoder assisted Diffusion models?

When the app only shows posts that are more than 10 hours old even when sorting by "hot" and shoving down the algorithmic feed on the app's home page, how are people still using the app?

Lately I've only been visiting a few subs that I'm interested in and keeping them open in safari with ublock; it's been a far better experience. This has drastically cut my reddit time now and if I do want to mindlessly scroll, I just use redlib(hosted in docker or one of their public instances)[0]. It has the same "sort" that's used on the desktop site.

[0] https://github.com/redlib-org/redlib

Worked with Retiming using DC Compiler in an ASIC implementation. Remember a lot of back & forth, sometimes the tool just doesn't add enough registers to meet the constraint, had to test variable register depths; this was a design that used Synopsys DesignWare for FP ops lol.