Free & open source · Advanced setup

Installing PrivateGPT on WSL with GPU Support

Ask questions about your own documents using an LLM that never sends anything to the cloud accelerated by your NVIDIA GPU through WSL.

Get it from GitHub

Opens the official zylon-ai/private-gpt repository

Maintainer

zylon-ai

Cost

Free

Runs on

WSL Ubuntu

Difficulty

Advanced

Set expectations first: This is not a one-click install. It involves compiling a llama.cpp backend against CUDA inside WSL, and mismatched Python/CUDA/driver versions are the most common source of errors. Budget a couple of hours, and expect to troubleshoot.

Before you start

  1. Install WSL with Ubuntu

    From an admin PowerShell: wsl --install -d Ubuntu-22.04. Restart when prompted.

  2. Update your NVIDIA driver on Windows (not inside WSL)

    Get the latest driver from nvidia.com/drivers — it's what exposes your GPU to WSL, and you should NOT install a separate Linux driver inside WSL itself.

  3. Update Ubuntu and install build tools

    sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential

Install steps

  1. Clone the repository

    git clone https://github.com/zylon-ai/private-gpt
cd private-gpt

  2. Set up pyenv for a clean Python version

    sudo apt-get install git gcc make openssl libssl-dev \
libbz2-dev libreadline-dev libsqlite3-dev zlib1g-dev \
libncursesw5-dev libgdbm-dev libc6-dev tk-dev libffi-dev
curl https://pyenv.run | bash

    Add pyenv to your PATH per its install output, then restart your shell.

  3. Install Poetry and project dependencies

    curl -sSL https://install.python-poetry.org | python3 -
poetry install --extras "ui llms-llama-cpp vector-stores-qdrant embeddings-huggingface"

  4. Build llama-cpp-python with CUDA enabled

    CMAKE_ARGS="-DLLAMA_CUBLAS=on" poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python

    This is the step that actually wires GPU acceleration in — skipping it means CPU-only inference.

  5. Download the default model and launch

    poetry run python scripts/setup
make run

    Watch the startup log for BLAS = 1 — that confirms the GPU is actually being used.

  6. Open the UI

    Visit http://127.0.0.1:8001 in your Windows browser — WSL forwards the port automatically.

If it says BLAS = 0 (no GPU)

  1. Re-run the CMAKE_ARGS install step

    It's the single most-skipped step — if llama-cpp-python was already installed without it, a plain reinstall won't fix it; you need --force-reinstall --no-cache-dir.

  2. Confirm WSL sees your GPU

    Run nvidia-smi inside WSL — if it errors, the Windows-side driver isn't exposing the GPU correctly; reinstall it and restart WSL with wsl --shutdown.

  3. Check your Poetry extras match your use case

    Newer PrivateGPT versions replaced the old --with local flag with explicit --extras — make sure llms-llama-cpp is included.

Good to know: This is an independent guide, not affiliated with zylon-ai. The link above goes to the real official repository — this page never hosts any installer or model file itself. Documents you ingest and questions you ask stay local; nothing is sent externally.