LLM - Sampling Temperature, Top K, and Top P, What & How?

In the LLM, using sampling

When we set the temperature, top K, and top P in the LLM, do you know what they are and what the order for applying them is?

• LLM Samplers Explained
• How to sample from the language model
• What is the actual order of execution when Top-K, Top-P, and Temperature are used together for NLP decoding?
• Token selection strategies: Top-K, Top-P, and Temperature
• Can someone explain what Top K and Top P are and what they do and how to use them?
• Mastering LLM Parameters: A Deep Dive into Temperature, Top-K, and Top-P

actual order: Temperature -> Top K -> Top P

arXiv

• The Curious Case of Neural Text Degeneration (Nucleus Sampling)

 

PyTorch - get the total number of model parameter

Total number of model parameters

• https://stackoverflow.com/questions/49201236/check-the-total-number-of-parameters-in-a-pytorch-model
• https://discuss.pytorch.org/t/how-do-i-check-the-number-of-parameters-of-a-model/4325

1. simple version

pytorch_total_params = sum(p.numel() for p in model.parameters())

2. listed version

def count_parameters(model):

  str_name = "name"

  str_parameter = "parameter"

  print(f"{str_name:50s}: {str_parameter:10s}")

  total_params = 0

  for name, parameter in model.named_parameters():

    if not parameter.requires_grad:

      continue

    params = parameter.numel()

    print(f"{name:50s}: {params:10s}")

    total_params += params

  print(f"Total Trainable Params: {total_params}")

  return total_params

PIL - image resize() from the basic code & idea

PIL: Python Image Library: https://github.com/python-pillow/

  Written by Fredrik Lundh(https://groups.google.com/g/dev-python/c/fbv5gWgaGpM?pli=1)

image resize algorithm

• https://en.wikipedia.org/wiki/Image_scaling
• https://en.wikipedia.org/wiki/Affine_transformation#Examples
• affine transformation scaling
• Please refer to "scale about image" image
• https://en.wikipedia.org/wiki/Bicubic_interpolation
• https://en.wikipedia.org/wiki/Bicubic_interpolation#/media/File:Comparison_of_1D_and_2D_interpolation.svg
• Pillow(PIL) Image resize NEAREST behavior differs depending on the version
• Resizing Images With Bicubic Interpolation

On Device AI - sLLM (or SLM)

sLLM (small Large Language Model) or SLM (Small Language Model)

What is the scheduler in the Stable Diffusion?

1. what is the scheduler?

• Choosing the Best SDXL Samplers for Image Generation
• Schedulers in AI Image Generation
• The ML developers guide to Schedulers in Stable Diffusion

2. sampler or scheduler?

• Samplers and schedulers summary by Bing, thoughts?
• Schedulers - What is a scheduler?

Transformer step by step

Original Paper: Attention Is All You Need

• The Illustrated Transformer by Jay Alammar
• https://www.linkedin.com/posts/eordax_transformers-ai-genai-activity-7221423755506421760-4fyh?utm_source=share&utm_medium=member_desktop
• https://www.linkedin.com/posts/eordax_transformers-encoder-ai-activity-7221063158302425088-kGLD?utm_source=share&utm_medium=member_desktop

About Quantization

Quantization 

https://paperswithcode.com/task/quantization

Vector Quantization 

• https://ieeexplore.ieee.org/document/1162229
• GPTVQ
• Example: https://scikit-learn.org/stable/auto_examples/cluster/plot_face_compress.html

& Maarten Greentendorst

• A Visual Guide to Quantization
• Which Quantization Method is Right for You

- Quantization FP16 FP8, and INT8