Preparing a GPU Environment for Machine Learning (ML) and Data Science (DS)
After reading this guide, you will learn how to prepare an environment for performing tasks related to Machine Learning (ML) and Data Science (DS), as well as run the Llama-2-7B Fine-tuning with LoRA LLM configuration in a conda environment.
-
Prepare a new GPU virtual machine, if this has not been done earlier.
-
Connect the Cloud GPU service, if this has not been done earlier.
-
Request quotas for the VM configuration template based on GPU from technical support.
-
Create a virtual machine with the following parameters:
-
A configuration template with GPU.
-
An OS with support for the required driver version. Find the available versions for your preferred distribution in the NVIDIA CUDA repository.
If a specific driver version is not required, choose one of the recommended distributions:
- Ubuntu version 22.04 or 24.04.
- Debian version 11 or 12.
- Red Hat Enterprise Linux/AlmaLinux/Oracle Linux/Rocky Linux version 8 or 9.
-
Access to the external network using an IP address from the public IP address pool or a Floating IP address.
Set the remaining parameters at your discretion.
-
-
-
Connect to the VM.
You can prepare the ML environment in one of the following ways:
- by installing CUDA components in the OS;
- by running a Docker container from a ready-made image using NVIDIA Container Toolkit.
The environment is prepared directly in the OS of your virtual machine. Use this method if you need access to system components and want to install and configure the required CUDA component versions yourself.
-
Install the CUDA components.
-
Install Python, if this has not been done earlier:
Ubuntu/DebianRed Hat Enterprise Linuxsudo apt install -y python3 python3-venv python3-pip -
Install Miniconda:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh &&bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda &&eval "$($HOME/miniconda/bin/conda shell.bash hook)" &&conda init bash && source ~/.bashrc -
Review the terms of use. To confirm your agreement, enter the commands:
conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/mainconda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r -
Create and activate a conda environment:
conda create -n ml-env python -yconda activate ml-env -
Install PyTorch with CUDA 12.4 support:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124 -
Check that PyTorch detects the GPU, CUDA, and cuDNN:
python -c "import torch; a=torch.cuda.is_available(); print('cuda',a,'cudnn',torch.backends.cudnn.is_available()); print('gpu', torch.cuda.get_device_name(0) if a and torch.cuda.device_count()>0 else 'none')" -
Install the required ML libraries:
pip install \transformers \accelerate \bitsandbytes \peft \datasets \sentencepiece \einops \ -
(Optional) Save the environment and pip dependencies:
conda env export > environment.ymlpip freeze > requirements.txtLater, use these files to:
-
quickly create an identical conda environment:
conda env create -f environment.yml -
restore the pip dependencies:
pip install -r requirements.txt
-
-
Create the
llama.pyPython script that will prepare the LoRA model for training:from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, BitsAndBytesConfigfrom peft import LoraConfig, get_peft_model, TaskTypefrom datasets import load_datasetimport torchMODEL_NAME = "meta-llama/Llama-2-7b-hf" # Requires accepting the license on HF
4-bit quantization (fits in 16 GB VRAM instead of 28 GB)
bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16 )
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) model = AutoModelForCausalLM.from_pretrained( MODEL_NAME, quantization_config=bnb_config, device_map="auto" # automatically distributes across GPUs )
LoRA configuration
lora_config = LoraConfig( task_type=TaskType.CAUSAL_LM, r=16, # adapter rank lora_alpha=32, target_modules=["q_proj", "v_proj"], # only attention layers lora_dropout=0.05, bias="none" )
model = get_peft_model(model, lora_config) model.print_trainable_parameters()
trainable params: 4,194,304 || all params: 3,504,607,232 || trainable%: 0.1197
Loading the dataset (example — alpaca)
dataset = load_dataset("tatsu-lab/alpaca", split="train[:1000]")
Next — the standard Trainer from transformers
... (add your TrainingArguments and Trainer.train())
print("LoRA модель готова к обучению!")
1. Run the script:<tabs><tablist><tab>System installation of CUDA components</tab><tab>NVIDIA Container Toolkit + Docker</tab></tablist><tabpanel>```consoleexport HF_TOKEN="<HF_TOKEN>"python llama.py
Here <HF_TOKEN> — a Hugging Face token. The account using this token must have access to the meta-llama/Llama-2-7b-hf repository.
docker run --rm --gpus all \-v "$PWD":/workspace \-v "$HOME/.cache/huggingface":/root/.cache/huggingface \-e HF_TOKEN="<HF_TOKEN>" \-w /workspace \<DOCKER_IMAGE> \python3 train_lora.py
Here:
<HF_TOKEN>— a Hugging Face token. The account using this token must have access to the meta-llama/Llama-2-7b-hf repository.<DOCKER_IMAGE>— the name of the image used in the Dockerfile.
If executed successfully, the command output will contain the message:
LoRA модель готова к обучению!