EMZETT.
Login

Emzett Wiki

CUDA and GPU programming: how AI computes on graphics cards

Properties1
tagsglossar, gpu, ki

2 min read

CUDA and GPU programming: how AI computes on graphics cards

In short: GPUs execute thousands of simple calculations in parallel; with CUDA (NVIDIA) or ROCm (AMD) you write kernel functions, but mostly you use frameworks such as PyTorch that already bring them.

In more detail: A CPU has few fast cores for complex sequences; a GPU has thousands of small cores for massively parallel calculations with the same operation on lots of data (SIMT). Matrix multiplications in neural networks fit perfectly.

In Depth

The model

  • Kernel: function running thousands of times in parallel on the GPU.
  • Thread → block → grid: threads are grouped in blocks (sharing shared memory), blocks form a grid; hardware groups 32 threads into a warp (run in lockstep, branches cost).
  • Memory hierarchy: registers → shared memory/L1 → L2 → VRAM (HBM/GDDR); host memory (RAM) separate: copies over PCIe/NVLink are expensive.
  • Tensor cores: special units for matrix multiplication (FP16/BF16/FP8).
  • Bottlenecks: compute-bound vs. memory-bandwidth-bound; occupancy, coherence.

CUDA example

__global__ void add(const float* a, const float* b, float* c, int n) {
  int i = blockIdx.x * blockDim.x + threadIdx.x;
  if (i < n) c[i] = a[i] + b[i];
}
// call: add<<<(n + 255) / 256, 256>>>(d_a, d_b, d_c, n);

Procedure: allocate memory on the GPU (cudaMalloc), copy data (cudaMemcpy), launch the kernel, copy the result back. From Python: Numba, CuPy, Triton (OpenAI) and PyTorch (tensor.to('cuda')) encapsulate this.

Ecosystem

  • CUDA (NVIDIA): market leader, libraries cuBLAS, cuDNN, NCCL (multi-GPU communication), TensorRT.
  • ROCm/HIP (AMD), oneAPI/SYCL (Intel), Metal (Apple), Vulkan Compute, OpenCL, WebGPU (browser).
  • Frameworks: PyTorch, JAX, TensorFlow use these backends; keep driver/toolkit versions matching (frequent source of errors: CUDA out of memory, wrong version).
  • Multiple GPUs: data/model/pipeline parallelism (see training at scale).
  • Rent cloud GPUs instead of buying; cost per hour, mind utilisation.

Practice and troubleshooting

  • nvidia-smi: utilisation, VRAM, temperature.
  • CUDA out of memory: reduce batch size, mixed precision (bf16), gradient checkpointing, quantise the model, torch.cuda.empty_cache().
  • Slower than CPU? Tasks too small, many host↔GPU copies, Python loops: batch/vectorise.
  • Profiling: PyTorch Profiler, Nsight.
  • Energy/cooling: GPUs draw 200–700 W: mind power supply and airflow.
  • Learning path: first use PyTorch on GPU, then Numba/Triton, then CUDA C++.
  • Related: why GPUs for AI, AI chips (TPU/NPU), inference servers.

See also: Why GPUs for AI, AI chips compared, AI frameworks, Graphics cards

Backlinks