VK Cloud logo

Preparing a GPU Environment for Machine Learning (ML) and Data Science (DS)

After reading this guide, you will learn how to prepare an environment for performing tasks related to Machine Learning (ML) and Data Science (DS), as well as run the Llama-2-7B Fine-tuning with LoRA LLM configuration in a conda environment.

Preparation steps

  1. Prepare a new GPU virtual machine, if this has not been done earlier.

    1. Connect the Cloud GPU service, if this has not been done earlier.

    2. Request quotas for the VM configuration template based on GPU from technical support.

    3. Create a virtual machine with the following parameters:

      • A configuration template with GPU.

      • An OS with support for the required driver version. Find the available versions for your preferred distribution in the NVIDIA CUDA repository.

        If a specific driver version is not required, choose one of the recommended distributions:

        • Ubuntu version 22.04 or 24.04.
        • Debian version 11 or 12.
        • Red Hat Enterprise Linux/AlmaLinux/Oracle Linux/Rocky Linux version 8 or 9.
      • Access to the external network using an IP address from the public IP address pool or a Floating IP address.

      Set the remaining parameters at your discretion.

  2. Connect to the VM.

  3. Install or update the NVIDIA driver.

1. Prepare the ML environment

You can prepare the ML environment in one of the following ways:

  • by installing CUDA components in the OS;
  • by running a Docker container from a ready-made image using NVIDIA Container Toolkit.

The environment is prepared directly in the OS of your virtual machine. Use this method if you need access to system components and want to install and configure the required CUDA component versions yourself.

  1. Install the CUDA components.

  2. Install Python, if this has not been done earlier:

    sudo apt install -y python3 python3-venv python3-pip
  3. Install Miniconda:

    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh &&bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda &&eval "$($HOME/miniconda/bin/conda shell.bash hook)" &&conda init bash && source ~/.bashrc
  4. Review the terms of use. To confirm your agreement, enter the commands:

    conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/mainconda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r
  5. Create and activate a conda environment:

    conda create -n ml-env python -yconda activate ml-env
  6. Install PyTorch with CUDA 12.4 support:

    pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
  7. Check that PyTorch detects the GPU, CUDA, and cuDNN:

    python -c "import torch; a=torch.cuda.is_available(); print('cuda',a,'cudnn',torch.backends.cudnn.is_available()); print('gpu', torch.cuda.get_device_name(0) if a and torch.cuda.device_count()>0 else 'none')"
  8. Install the required ML libraries:

    pip install \transformers \accelerate \bitsandbytes \peft \datasets \sentencepiece \einops \
  9. (Optional) Save the environment and pip dependencies:

    conda env export > environment.ymlpip freeze > requirements.txt

    Later, use these files to:

    • quickly create an identical conda environment:

      conda env create -f environment.yml
    • restore the pip dependencies:

      pip install -r requirements.txt

2. Run the Llama-2-7B Fine-tuning with LoRA LLM configuration

  1. Create the llama.py Python script that will prepare the LoRA model for training:

    from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, BitsAndBytesConfigfrom peft import LoraConfig, get_peft_model, TaskTypefrom datasets import load_datasetimport torchMODEL_NAME = "meta-llama/Llama-2-7b-hf"  # Requires accepting the license on HF

4-bit quantization (fits in 16 GB VRAM instead of 28 GB)

bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16 )

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) model = AutoModelForCausalLM.from_pretrained( MODEL_NAME, quantization_config=bnb_config, device_map="auto" # automatically distributes across GPUs )

LoRA configuration

lora_config = LoraConfig( task_type=TaskType.CAUSAL_LM, r=16, # adapter rank lora_alpha=32, target_modules=["q_proj", "v_proj"], # only attention layers lora_dropout=0.05, bias="none" )

model = get_peft_model(model, lora_config) model.print_trainable_parameters()

trainable params: 4,194,304 || all params: 3,504,607,232 || trainable%: 0.1197

Loading the dataset (example — alpaca)

dataset = load_dataset("tatsu-lab/alpaca", split="train[:1000]")

Next — the standard Trainer from transformers

... (add your TrainingArguments and Trainer.train())

print("LoRA модель готова к обучению!")

1. Run the script:<tabs><tablist><tab>System installation of CUDA components</tab><tab>NVIDIA Container Toolkit + Docker</tab></tablist><tabpanel>```consoleexport HF_TOKEN="<HF_TOKEN>"python llama.py

Here <HF_TOKEN> — a Hugging Face token. The account using this token must have access to the meta-llama/Llama-2-7b-hf repository.

docker run --rm --gpus all \   -v "$PWD":/workspace \   -v "$HOME/.cache/huggingface":/root/.cache/huggingface \   -e HF_TOKEN="<HF_TOKEN>" \   -w /workspace \   <DOCKER_IMAGE> \   python3 train_lora.py

Here:

If executed successfully, the command output will contain the message:

LoRA модель готова к обучению!
Was this article helpful?