Can You Fine-Tune Llama 3.1 8B on a 4GB GPU? Soup’s Layer Streaming Explained

Yes. An open-source project called Soup has demonstrated LoRA fine-tuning of Meta’s Llama 3.1 8B Instruct model on an RTX 3050 Laptop GPU with only 4GB of VRAM.

But the important part is what that sentence does not mean.

Soup is not squeezing the entire 8-billion-parameter model into 4GB of GPU memory. It is not training Llama 3.1 from scratch. It is not updating all 8 billion parameters. And the result does not establish that every 4GB GPU can train every 8B model.

Instead, Soup exploits a useful property of LoRA fine-tuning: the enormous base model stays frozen while a much smaller set of adapter parameters is trained. Soup’s layer-streaming system keeps most of that frozen base outside VRAM, in system RAM or on NVMe storage, and moves the weights through the GPU a layer at a time.

That distinction is the story.

A conventional optimized 8B QLoRA setup still commonly assumes that the quantized base model remains resident in GPU memory. For comparison, Unsloth currently lists 6GB as its absolute minimum VRAM figure for 8B QLoRA. Soup attacks a different bottleneck: what if the frozen base does not need to stay in VRAM at all?

The answer, at least for a specific class of parameter-efficient fine-tuning workloads, appears to be that it does not.

The 4GB claim, in plain English

Here is the easiest way to understand what Soup has and has not shown.

So the useful version of the claim is narrower but still significant:

A frozen 8B base model does not necessarily have to remain in GPU memory while its LoRA adapters are being trained.

That is what Soup has demonstrated.

What is Soup?

Soup is an Apache-2.0 open-source LLM training project built around a relatively simple idea: describe a fine-tuning job in one YAML configuration and let the tool handle much of the setup around model loading, datasets, quantization, training and export.

Its broader feature set includes ordinary local training, cloud execution, evaluation, serving and multiple backends. Layer streaming is one specific part of the project, and the project still labels that feature BETA in its current layer-streaming documentation.

As of October 1, 2026, PyPI lists Soup 0.75.1 as the latest published package release.

The attraction is obvious. Local AI has already made it possible to run increasingly capable models on consumer hardware. Projects such as Project NOMAD show how far local and offline AI infrastructure has moved. Fine-tuning remains harder because training requires more memory than inference and has to preserve activations, gradients, optimizer state and other working data.

Soup is trying to push that boundary downward.

Why an 8B model normally wants more than 4GB of VRAM

The first misconception comes from simple parameter arithmetic.

Eight billion parameters stored at four bits each sounds like roughly 4GB. That can make it seem as though an 8B model should naturally fit on a 4GB card.

Training is not that simple.

Even with QLoRA, which combines a quantized frozen base model with small trainable LoRA adapters, GPU memory can also be consumed by:

• quantization metadata;
• LoRA adapter weights;
• adapter gradients;
• optimizer state;
• activations;
• temporary working buffers;
• attention and vocabulary-dependent allocations;
• framework and CUDA overhead.

That is why a mature optimization project such as Unsloth currently publishes 6GB as the absolute minimum for an 8B QLoRA configuration, with the warning that some models require more.

Soup changes the memory equation by moving the largest mostly static object out of VRAM: the frozen base model itself.

How Soup’s layer streaming actually works

The core mechanism is conceptually simple.

1. The base model is frozen

With LoRA, the original model weights are not being trained. Small trainable low-rank adapters are attached to selected layers instead.

That is critical because Soup does not have to preserve optimizer state and gradients for billions of frozen parameters.

2. The frozen base lives outside the GPU

According to Soup’s technical documentation, the checkpoint is prepared so the frozen decoder weights can live in CPU memory, page-locked when possible, or spill to an NVMe-backed disk tier when the host cannot hold the full store in RAM.

The GPU therefore does not need to hold the entire frozen model at once.

3. One decoder layer is brought into reusable GPU memory

Instead of loading every layer and leaving it resident, Soup copies the weights required for the current decoder layer into a small pool of VRAM buffers.

While one layer is computing, the next layer can be prefetched into another buffer. When a layer is finished, the buffer can be reused.

A simplified memory picture looks like this:

The parameters have not disappeared. Their storage location changed.

4. Backpropagation means the weights have to come back

Streaming is not free.

During gradient-checkpointed training, the model has to recompute portions of the forward pass during the backward pass. Soup’s documentation notes that decoder layers therefore have to be read again when their backward computation occurs.

That creates a fundamental trade:

VRAM requirement goes down. Data movement goes up.

That is why Soup itself says not to enable layer streaming if the model already fits comfortably in GPU memory. In that situation, streaming buys little and adds overhead.

What the original 4GB benchmark actually measured

The benchmark most people are likely to encounter is the project’s original Llama 3.1 8B result:

• GPU: RTX 3050 Laptop, 4GB
• Model: Llama 3.1 8B Instruct
• Method: LoRA with NF4-quantized streamed base
• Sequence length: 512
• Batch: 1
• Peak VRAM: 3.32GB
• Throughput: 119.6 tokens per second

Those figures appear in the Soup README and the project’s layer-streaming documentation.

They are real project measurements, not estimates.

They are also not the end of the story.

There is now a newer 4GB measurement, but Soup’s public pages are out of sync

This is where a simple reading of the README becomes misleading.

The current README and layer-streaming documentation still describe the 119.6 tok/s result as a pre-repair measurement and say a new 4GB run is pending.

But Soup’s more detailed benchmark ledger now records a post-repair Llama 3.1 8B measurement from September 19, 2026, and the project’s tracking issue for the rerun was subsequently closed.

The two recorded runs are:

The later result is particularly useful because it confirms that repaired code still ran the 8B workload within the 4GB physical VRAM limit.

What it does not prove is that Soup became 74% faster.

Why 208.6 tok/s does not mean a 1.74x software speedup

The two measurements were taken on different laptops using the same RTX 3050 Laptop GPU model, and their recorded clock rates were dramatically different: 952 MHz versus 1935 MHz.

The underlying benchmark record explicitly warns readers not to divide 208.6 by 119.6 and call the difference a software improvement.

The project’s own normalized analysis reaches almost the opposite conclusion: once the very different clocks are considered, the repaired path consumed a smaller fraction of its same-session compute ceiling. The benchmark authors estimate it was roughly 14% slower per unit of clock, although other implementation changes occurred between the two runs as well.

So the responsible interpretation is:

• The 4GB feasibility claim survived the repair.
• The later run reached a lower 2.40GB peak.
• The raw throughput numbers are not an apples-to-apples speed comparison.

That last point matters because many technology summaries turn benchmark numbers into performance rankings without checking whether the hardware was operating under comparable conditions.

Did a correctness bug invalidate the Llama 3.1 8B result?

No, but the bug is important enough that anyone evaluating Soup should understand it.

In August 2026, the Soup project reported a silent wrong-gradient defect in NF4 layer streaming.

The disturbing part was not merely that a bug existed. It was how invisible the bug could be.

For sufficiently large NF4 decoder layers, the streamed model could produce:

• the correct forward output;
• apparently healthy loss values;
• and yet the wrong gradients during backpropagation.

A user looking only at the training loss could therefore believe everything was working.

Soup’s testing placed the failure boundary between roughly 163.8 and 171.5 MiB per decoder layer in the investigated configuration. Qwen2.5 32B and 72B crossed that boundary and were affected. Llama 3.1 8B was measured at about 105 MiB per layer, while Qwen2.5 14B was around 132 MiB, and both remained below the observed boundary.

The issue was repaired in v0.73.0.

That means the original 8B benchmark was measured before the repair, but the specific 8B configuration was not one of the configurations found to have wrong gradients.

This distinction is important.

Saying “Soup had a gradient bug, therefore the 8B benchmark was fake” would be wrong.

Saying “the original benchmark predates a correctness repair, so current-code performance needed to be remeasured” is fair. The later September measurement now supplies that missing evidence.

The gradient bug also reveals something important about AI benchmarks

The episode is arguably more instructive than a clean launch would have been.

A training system can produce reasonable outputs and a smooth-looking loss curve while still computing incorrect gradients.

Soup’s validation work therefore separates two questions that are often casually collapsed into one:

1. Does the streamed model produce the same forward result as the resident model?
2. Does it produce the same LoRA gradients during backward propagation?

That is the right standard for a technique that changes where and when weights exist in memory.

The project has published the failed gates, discarded benchmark attempts and correction history in its measurement records, which makes the technical claims unusually inspectable for a young open-source project.

That does not make every Soup claim independently verified. Most of the detailed performance evidence still comes from the project and its contributors. But it gives outside researchers something much better than a marketing graph: reproducible harnesses, configuration details and a record of failures alongside successes.

RTX 50-series users need another caveat: Blackwell NF4 exactness is currently unresolved

There is a newer limitation that deserves more attention than it is likely to receive in quick tutorials.

Soup’s current documentation reports that, beginning with testing around v0.75.0, streamed NF4 did not remain bit-exact against resident NF4 on an RTX 5070 Blackwell GPU.

The project reports differences of approximately:

• 4.9e-4 in fp16 testing;
• 3.9e-3 in bf16 testing.

Meanwhile, the equivalent tests with quantization: none passed. Soup therefore attributes the observed split to the 4-bit bitsandbytes path on that tested Blackwell architecture rather than to layer streaming generally, while also stating that the root cause is not yet established.

That does not prove every RTX 50-series card will behave identically.

It does mean the broad claim that Soup’s NF4 streaming is bit-exact on any modern NVIDIA GPU would currently be too strong. The headline 4GB measurements were performed on Ampere, not Blackwell.

For anyone using recent hardware, this is exactly the sort of version-and-architecture caveat worth checking before committing hours to a run.

Another reason to use a current release: an adapter could appear to save successfully and contain nothing

Soup has also had serialization bugs around streamed adapters.

The v0.75.1 release notes describe a compatibility problem with PEFT 0.21.0 in which a layer-streamed LoRA adapter could be saved with zero tensors. The resulting adapter_model.safetensors could be essentially empty even though the preceding training job appeared to run.

The project says affected adapters cannot be recovered and must be retrained.

That issue is fixed in 0.75.1.

The practical lesson is simple: do not follow an old Soup tutorial without checking the version it targets. For a fast-moving training stack, Soup, PyTorch, PEFT, Transformers and bitsandbytes versions are part of the experiment.

What Soup layer streaming currently supports

Layer streaming is not a universal switch for every training method Soup supports elsewhere.

According to the current support and refusal rules, the streamed path is deliberately narrower:

The reason GRPO and PPO are excluded is particularly revealing. Those methods require generation rollouts, and autoregressive generation would repeatedly reread model layers for each generated token. That destroys the amortization that makes this style of streaming useful.

In other words, this is not merely an unfinished checkbox. Some workloads fundamentally fit the architecture better than others.

Soup is not simply a replacement for QLoRA or Unsloth

The terminology gets confusing because several optimization techniques can appear to solve the same problem.

They do not.

Soup can use Unsloth for ordinary non-streamed training, but the layer-streaming path itself rejects the Unsloth backend because both systems need to control how the model is loaded.

That is why “Soup vs. Unsloth” is not a clean winner-and-loser comparison.

If an 8B model already fits comfortably on your GPU, a fast resident training system is usually the more natural problem to solve. Soup’s layer streaming becomes interesting when the model does not fit resident in the first place.

Did Soup invent layer-by-layer weight streaming?

No.

The general idea of moving model weights between slower host storage and GPU memory has substantial prior art.

For example, Microsoft’s DeepSpeed ZeRO-Inference publicly described keeping model weights in CPU memory or NVMe and streaming them into GPU memory layer by layer for inference years before Soup existed. DeepSpeed’s training-oriented ZeRO work likewise uses CPU and storage offloading to reduce accelerator-memory requirements.

Soup’s contribution is more specific.

It applies layer streaming to a consumer-oriented LoRA training path, combines it with quantized frozen weights, packages it into a relatively accessible training workflow, and publishes a detailed correctness protocol intended to show that the streamed run matches the resident reference.

That is a meaningful engineering contribution without pretending the underlying concept appeared from nowhere.

Why NF4 matters so much

Moving the frozen model into system RAM solves one memory bottleneck only to create another.

A full 8B model stored in bf16 would still require roughly 16GB just for two-byte parameter values before other overhead. That is uncomfortable on a normal laptop with 16GB of total system RAM.

Soup therefore combines streaming with NF4, the four-bit quantization format associated with QLoRA.

The project reports that NF4 reduces the host-side frozen store to roughly one quarter of the bf16 size. The benefit is not only capacity. A smaller host store is also easier to keep in page-locked memory, allowing asynchronous CPU-to-GPU transfers to overlap with computation.

This is one reason Soup’s 4GB result is better understood as the combination of two techniques:

1. quantize the frozen base aggressively enough that host storage is manageable;
2. do not require that frozen base to stay in VRAM.

Neither idea by itself explains the result as well as the combination.

So where did the missing memory go?

Mostly to a different tier of the memory hierarchy.

The post-repair 8B benchmark is especially illustrative. The project’s benchmark record lists a 5.70GB pinned host store while peak allocated GPU memory remained 2.40GB.

That means the right question is not:

How did Soup make an 8B model only 2.4GB?

It did not.

The better question is:

How much of the model has to exist in expensive, capacity-constrained VRAM at the same moment?

Layer streaming makes that number much smaller.

This is the same broader hardware problem now confronting the entire AI industry at a radically different scale. As we have discussed in our explainer on why AI companies are running into hardware, memory and networking constraints, modern AI is increasingly constrained not only by raw arithmetic but by where data is stored and how quickly it can be moved to the compute that needs it.

Soup is a consumer-scale example of that same principle.

Will Soup work on any 4GB GPU?

No responsible reading of the evidence supports that conclusion.

The demonstrated physical 4GB result is an RTX 3050 Laptop GPU, an Ampere-generation NVIDIA card. Soup also provides a proof notebook that can impose a 4GB process-memory cap on a larger Tesla T4, but that is not the same thing as benchmarking every physically 4GB GPU.

Whether a particular run fits depends on factors including:

• GPU architecture and supported numeric formats;
• model architecture and decoder-layer size;
• vocabulary size;
• sequence length;
• batch size;
• LoRA configuration;
• quantization mode;
• available system RAM;
• whether host memory can be pinned;
• PCIe and memory-transfer behavior;
• NVMe performance when the disk tier is used;
• PyTorch, PEFT, Transformers and bitsandbytes versions.

Soup now performs a streaming-specific preflight estimate before training, which is much more useful than assuming “4GB” is a universal compatibility badge.

Can Soup fine-tune 14B, 32B or 70B models on the same 4GB laptop?

The layer-streaming design can address models larger than 8B, and the project has validated larger streamed models on more capable hardware.

That is not the same as demonstrating those models on the same 4GB laptop.

The project’s own documentation is explicit that nothing above 8B has been measured on its reference 4GB machine. Larger models also require much larger host-side stores. At some point, system RAM and storage bandwidth become the limiting resources even if VRAM is no longer holding the whole model.

So “8B on 4GB” should not be casually extrapolated into “70B on 4GB.”

The mechanism may scale. The specific hardware claim does not automatically scale with it.

Is layer streaming faster?

Usually, speed is not the reason to use it.

The purpose is to make a training job possible when a resident configuration does not fit.

Soup’s documentation reports an apples-to-apples 0.5B comparison in which streaming was slower than resident training. At larger sizes on the 4GB reference card, a fair resident comparison may not exist because the resident model simply does not fit.

That creates an important benchmarking principle:

A slower training method that completes the workload can be more useful than a faster method that cannot start it.

But if the model already fits in VRAM, Soup’s own recommendation is straightforward: do not turn streaming on merely because it exists.

Who should actually care about this?

Layer streaming is most interesting for users who:

• have a low-VRAM NVIDIA GPU;
• want to perform LoRA-based local fine-tuning;
• cannot fit the desired frozen base model resident in VRAM;
• have enough system RAM or fast NVMe storage to hold the offloaded weights;
• accept a beta feature and are willing to validate outputs;
• care more about making the run possible than maximizing raw training speed.

It is much less compelling when:

• the model already fits easily;
• full-model fine-tuning is required;
• the workload depends on GRPO or PPO generation rollouts;
• a production workflow cannot tolerate beta infrastructure;
• host memory and storage are themselves severely constrained.

The bigger significance is not that GPUs stopped mattering

It would be easy to turn Soup into a story about expensive AI hardware suddenly becoming unnecessary.

The evidence does not support that.

Frontier pretraining remains an industrial-scale computing problem. Even ordinary local fine-tuning becomes faster and more flexible as more VRAM is available. Moving weights out of VRAM also makes performance depend more heavily on system memory, storage and data-transfer behavior.

What Soup demonstrates is subtler and more useful:

the amount of GPU memory required for a workload is partly an architectural choice.

A model can be too large to reside in VRAM and still be trainable if the parts that do not need to remain resident can be moved through a hierarchy of slower memory without breaking the mathematics of training.

That idea already exists at data-center scale in systems such as DeepSpeed.

Soup’s contribution is showing how far the same principle can be pushed toward ordinary consumer hardware.

So, can you really fine-tune Llama 3.1 8B with 4GB of VRAM?

Yes, with important qualifications.

Soup has published a real Llama 3.1 8B LoRA fine-tuning run on a physical 4GB RTX 3050 Laptop GPU, and a later post-repair measurement confirms that the workload still fits within that VRAM limit on post-repair code.

The reason it works is not that Llama 3.1 8B has somehow become a 4GB model.

It works because Soup changes where the frozen model lives.

The base weights spend most of their time in system RAM or NVMe. Only the weights needed for the current part of the computation are staged through reusable GPU memory, while the small trainable LoRA state stays resident.

That lowers the VRAM floor enough to make a previously impossible class of local fine-tuning runs possible.

The tradeoff is more data movement, a narrower supported training path, dependence on host memory and storage, and a beta implementation that has already uncovered several subtle correctness and serialization problems.

So the headline survives scrutiny, but the precise version is better than the viral one:

Soup has not made serious AI training hardware irrelevant. It has shown that an 8B frozen base model does not have to live in GPU memory for LoRA fine-tuning to work. On an RTX 3050 Laptop GPU, that was enough to push Llama 3.1 8B below a 4GB VRAM ceiling.

That is a real technical result.

Frequently Asked Questions

Can I fine-tune Llama 3.1 8B with 4GB VRAM?

Yes, Soup has demonstrated LoRA fine-tuning of Llama 3.1 8B Instruct on a physical 4GB RTX 3050 Laptop GPU using NF4 layer streaming. This does not mean every 4GB GPU or every training configuration will fit.

Does Soup fit the entire Llama 3.1 8B model in 4GB?

No. The frozen model weights are primarily stored in system RAM or NVMe and streamed through GPU buffers layer by layer. Peak VRAM can therefore stay below the size required to hold the entire quantized base resident.

Is Soup doing full fine-tuning?

Not in the low-VRAM layer-streaming configuration discussed here. The streamed path requires LoRA adapters and keeps the base model frozen.

Is Soup the same as QLoRA?

No. QLoRA quantizes a frozen model and trains LoRA adapters. Soup can use NF4 quantization too, but its additional trick is moving the frozen base out of persistent VRAM and streaming it through the GPU as required.

Is Soup better than Unsloth?

They are not direct substitutes for every workload. Unsloth focuses heavily on making resident fine-tuning faster and more memory-efficient. Soup’s layer streaming targets cases where the frozen model does not fit resident in VRAM. Soup’s streamed path currently cannot use the Unsloth backend.

How much system RAM does Soup need for Llama 3.1 8B?

There is no universal minimum established by the 4GB result. The original 8B NF4 run used a 3.60GB pinned host store, while the later repaired-code benchmark lists a 5.70GB pinned store after changes to how large layers were handled. Both test laptops had roughly 17GB of system RAM. Other allocations and the operating system also need memory, so the store size alone should not be treated as the machine’s minimum RAM requirement.

Can Soup use an SSD instead of enough system RAM?

Yes. Current Soup supports an NVMe-backed disk tier for streamed weights when the base store does not fit safely in RAM. That introduces another bandwidth bottleneck, so storage performance matters.

Can Soup fine-tune 70B on a 4GB GPU?

The project has validated much larger streamed models on larger hardware, but it has not demonstrated a 70B fine-tune on the same 4GB reference laptop. Larger models also require far more host memory or storage. The 8B-on-4GB benchmark should not be extrapolated into a universal 70B claim.

Is Soup layer streaming safe to use on RTX 50-series GPUs?

Use caution with NF4. Soup currently reports that its streamed NF4 exactness tests diverge from the resident NF4 reference on an RTX 5070 Blackwell GPU, while the equivalent non-quantized streaming tests pass. The root cause remains unresolved in the project’s current documentation. That result should not automatically be generalized to every Blackwell card, but it is enough to justify verifying your own configuration rather than assuming bit-exact behavior.

Is the 208.6 tok/s benchmark faster than the older 119.6 tok/s result?

The raw number is higher, but the two measurements are not a valid direct speed comparison. They were recorded on different RTX 3050 laptops at very different GPU clock rates. The later measurement is most useful as confirmation that repaired code still fits the 8B workload inside 4GB VRAM.

References and Further Reading

Soup Primary Sources

• Soup GitHub repository — Main source repository, installation instructions and project overview.
• Layer streaming: train past your VRAM — Current project documentation describing the streaming mechanism, measured 4GB result, supported tasks, restrictions and known correctness caveats.
• Soup measurement and benchmark ledger — Index of the project’s published gate records, including the original and September 2026 post-repair 8B measurements.
• NF4 layer-streaming gate record — Detailed measurement record behind the 8B benchmark, including discarded runs, methodology, GPU clocks and the later repaired-code rerun.
• Issue #331: silent wrong gradients above the NF4 layer-size boundary — Primary issue record documenting the defect in larger NF4 layers, the measured boundary and affected versus unaffected models.
• Issue #361: post-repair 8B NF4 throughput rerun — Tracking record for repeating the physical 4GB benchmark after the correctness repair.
• Soup v0.75.1 release — Latest published release notes as of publication, including the PEFT 0.21 streamed-adapter serialization fix.
• Soup 0.75.1 on PyPI — Package release metadata and provenance.
• Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU, v3 — Alpamys Makazhan’s project preprint describing the method, validation protocol and revisions to earlier interpretations.

Background and Comparison

• LoRA: Low-Rank Adaptation of Large Language Models — Original LoRA paper explaining parameter-efficient fine-tuning with a frozen base model and trainable low-rank adapters.
• QLoRA: Efficient Finetuning of Quantized LLMs — Original QLoRA research introducing 4-bit quantized base-model fine-tuning and NF4.
• Unsloth VRAM requirements — Current comparison point for resident QLoRA memory requirements; lists 6GB as the absolute minimum for 8B QLoRA.
• DeepSpeed ZeRO-Inference — Earlier example of keeping model weights in CPU or NVMe storage and streaming layers into GPU memory, useful context for understanding the prior art behind weight streaming.

Editorial currency note: Soup is developing quickly. Version numbers, supported architectures, benchmark records and known limitations were checked against the project’s public repository, documentation and PyPI release history on October 1, 2026. Readers planning a training run should verify the current release notes and layer-streaming documentation before relying on version-specific behavior described here.