How I Customized Llama 3.1 8B on a Budget

I was impressed by how fine-tuned large language models outperform retrieval-augmented systems, especially at inference time. So I set out to fine-tune an open-source model like Meta's Llama 3.1 (8B parameters). But most sources stated you needed giant, expensive GPUs and tons of storage resources that I simply didn't have.

Determined to find another way, I tried it on a budget, and discovered you don't need a fancy setup to customize a state-of-the-art model.

My Secret Weapon: LoRA (Low-Rank Adaptation)

Instead of retraining 8 billion weights, LoRA tacks on small "adapter" matrices. Think of it as fine-tuning a car's suspension, rather than redesigning the entire engine.

My Setup on Northeastern's Cluster

I tapped into the Discovery cluster, which has:

Jobs are managed by SLURM, with a hard 4-hour limit. To avoid eating disk quota, I redirected all caches to scratch space:

export CACHE_DIR="${SCRATCH}/llama3_finetune/cache/huggingface"
export HF_HOME="$CACHE_DIR"
export TRANSFORMERS_CACHE="$CACHE_DIR"

Organizing the Project

I called it LlamaVox, with a simple structure:

LlamaVox/
├── config/          # LoRA settings
├── data/            # Training files
├── models/          # Saved adapters
├── slurm/           # Job scripts
└── src/             # Training code

My Tech Stack

Prepping the Data

I built three datasets:

Each example looked like this:

{
  "conversations": [
    {"role": "user", "content": "What are the main challenges of urban planning?"},
    {"role": "assistant", "content": "Urban planning faces several key challenges..."}
  ]
}

Running the Training Jobs

Here's a snippet of my SLURM script for an H200 GPU:

#!/bin/bash
#SBATCH --job-name=llama3_h200
#SBATCH --time=4:00:00
#SBATCH --gres=gpu:h200:1
#SBATCH --mem=96G
#SBATCH --cpus-per-task=16

python src/train.py \
  --model llama-3.1 \
  --dataset data/mini.json \
  --lora_rank 8 \
  --output_dir models/mini_h200

Mini Dataset Results on H200

GPU Performance Comparison

Performance Optimization Tricks

Troubleshooting: Issues & Fixes

Quick Start Guide

# Grab a GPU node
srun --gres=gpu:1 --mem=32G --time=1:00:00 --pty bash

# Run the launcher
./llama3_finetune/run_model.sh

What I Learned

The key insight is that you don't need expensive hardware to fine-tune large language models. With LoRA, smart optimization, and university cluster resources, you can achieve impressive results on a budget. The 4-hour time limit actually forced me to be more efficient and creative with my approach.

This project showed me that democratizing AI isn't just about open-source models—it's about making the fine-tuning process accessible to students and researchers with limited resources. By sharing these techniques, I hope to inspire others to experiment with large language models without breaking the bank.