Sampler - Stable Diffusion

What are Diffusion Models? - Lil'Log (Lilian Weng)

The Illustrated Stable Diffusion - Jay Alammar

1. DDPM: Denoising Diffusion Probabilistic Model
1. basic sampler -> Stable Diffusion's basic sampler 
2. 1000 steps 
2. DDIM: Denoising Diffusion Implicit Model
1. DDPM -> DDIM 
2. 10 ~ 50 steps 
3. Euler Discrete
1. Ordinary Differential Equation (OED) 
2. simple & fast 
3. same of DDIM or more steps
4. OLSS: Optimal Linear Subspace Search
1. fast, but using the old model 
2. 5 ~ 10 steps 
5. CM: Consistency Model
1. 1 step
2. pixel space
3. complex calculation
4. limited high resolution image 
6. LCM: Latent Consistency Model
1. 1 ~ 4 steps 
2. latent space
3. low dimension calculation
4. support high resolution image 
5. need LoRA or need re-training model (cannot use old model) 

Image quality metric (Generative AI)

Image Quality Metrics for Generative AI 

 

1. PSNR (Peak Signal to Noise Ratio)
1.  Image Quality Metrics: PSNR vs. SSIM
2. SSIM (Structural SIMilarity)
1.  Image quality assessment: from error visibility to structural similarity
2.  Understanding SSIM
3. The SSIM Index for Image Quality Assessment 
3. FID (Frechet Inception Distance)
1.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
2. Frechet distance 
1. How long should your dog leash be? 
4. LPIPS
1.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

 

Code Review - C++

- Papers

• Challenges of Refactoring C Programs 
• https://refactoring.guru/design-patterns/cpp 
• Discipline Matters: Refactoring of Preprocessor Directives in the #ifdef Hell 
• Embracing the C preprocessor during refactoring 

 

- Articles

• Why Google Stores Billions of Lines of Code in a single Repository
• Software Engineering at Google  
• C to C++: 5 Tips for Refactoring C Code into C++
• Tips for C++ Refactoring 
• Refactoring for C/C++ and C#: Let the Machine do the Work
• Refactoring in C++: Top Techniques and Best Practices

 

- Books

• Clean Code
• Clean Coder
• Software Engineering at Google 
  

LVM - DDPM, DDIM

1. Evaluation
1. Inception Score
2. FID (Freche Inception Distance)
2. LVM - Large Visual Model
1. GAN
2. DDPM: denoising diffusion probabilistic models (https://arxiv.org/abs/2006.11239)
3. DDIM: denoising diffusion models (https://arxiv.org/abs/2010.02502)

LLM - Sampling Temperature, Top K, and Top P, What & How?

In the LLM, using sampling

When we set the temperature, top K, and top P in the LLM, do you know what they are and what the order for applying them is?

• LLM Samplers Explained
• How to sample from the language model
• What is the actual order of execution when Top-K, Top-P, and Temperature are used together for NLP decoding?
• Token selection strategies: Top-K, Top-P, and Temperature
• Can someone explain what Top K and Top P are and what they do and how to use them?
• Mastering LLM Parameters: A Deep Dive into Temperature, Top-K, and Top-P

actual order: Temperature -> Top K -> Top P

arXiv

• The Curious Case of Neural Text Degeneration (Nucleus Sampling)

Repetition penalties are terribly implemented - A short explanation and solution

https://www.reddit.com/r/LocalLLaMA/comments/1g383mq/repetition_penalties_are_terribly_implemented_a/?rdt=41955

 

PyTorch - get the total number of model parameter

Total number of model parameters

• https://stackoverflow.com/questions/49201236/check-the-total-number-of-parameters-in-a-pytorch-model
• https://discuss.pytorch.org/t/how-do-i-check-the-number-of-parameters-of-a-model/4325

1. simple version

pytorch_total_params = sum(p.numel() for p in model.parameters())

2. listed version

def count_parameters(model):

  str_name = "name"

  str_parameter = "parameter"

  print(f"{str_name:50s}: {str_parameter:10s}")

  total_params = 0

  for name, parameter in model.named_parameters():

    if not parameter.requires_grad:

      continue

    params = parameter.numel()

    print(f"{name:50s}: {params:10s}")

    total_params += params

  print(f"Total Trainable Params: {total_params}")

  return total_params

PIL - image resize() from the basic code & idea

PIL: Python Image Library: https://github.com/python-pillow/

  Written by Fredrik Lundh(https://groups.google.com/g/dev-python/c/fbv5gWgaGpM?pli=1)

image resize algorithm

• https://en.wikipedia.org/wiki/Image_scaling
• https://en.wikipedia.org/wiki/Affine_transformation#Examples
• affine transformation scaling
• Please refer to "scale about image" image
• https://en.wikipedia.org/wiki/Bicubic_interpolation
• https://en.wikipedia.org/wiki/Bicubic_interpolation#/media/File:Comparison_of_1D_and_2D_interpolation.svg
• Pillow(PIL) Image resize NEAREST behavior differs depending on the version
• Resizing Images With Bicubic Interpolation

On Device AI - sLLM (or SLM)

sLLM (small Large Language Model) or SLM (Small Language Model)

What is the scheduler in the Stable Diffusion?

1. what is the scheduler?

• Choosing the Best SDXL Samplers for Image Generation
• Schedulers in AI Image Generation
• The ML developers guide to Schedulers in Stable Diffusion

2. sampler or scheduler?

• Samplers and schedulers summary by Bing, thoughts?
• Schedulers - What is a scheduler?

Transformer step by step

Original Paper: Attention Is All You Need

• The Illustrated Transformer by Jay Alammar
• https://www.linkedin.com/posts/eordax_transformers-ai-genai-activity-7221423755506421760-4fyh?utm_source=share&utm_medium=member_desktop
• https://www.linkedin.com/posts/eordax_transformers-encoder-ai-activity-7221063158302425088-kGLD?utm_source=share&utm_medium=member_desktop

About Quantization

Quantization 

https://paperswithcode.com/task/quantization

Vector Quantization 

• https://ieeexplore.ieee.org/document/1162229
• GPTVQ
• Example: https://scikit-learn.org/stable/auto_examples/cluster/plot_face_compress.html

& Maarten Greentendorst

• A Visual Guide to Quantization
• Which Quantization Method is Right for You

- Quantization FP16 FP8, and INT8

Apple, WWDC

• https://developer.apple.com/videos/play/wwdc2023/10047/
• https://apple.github.io/coremltools/docs-guides/source/opt-overview.html (website)

how to convert array to vector in c++ fast way

array to vector

#include <vector>

constexpr int vec_size = 5;

float a[vec_size] = {0, 1, 2, 3, 4};

std::vector<float> vec_a(a, a + vec_size); // good

1. https://sites.google.com/site/hashemian/home/tips-and-tricks/copy-array-cpp
2. https://stackoverflow.com/questions/8777603/what-is-the-simplest-way-to-convert-array-to-vector
3. https://www.freecodecamp.org/news/cpp-vector-how-to-initialize-a-vector-in-a-constructor/ <- how to initialize a vector from an array in C++

And vector to array

#include <vector>

constexpr int vec_size = 5;

float a[vec_size] = {0, 1, 2, 3, 4};

std::vector<float> vec_a(a, a + vec_size); // good

#include <algorithm>

float b[vec_size] = {}

std::copy(vec_a.begin(), vec_a.end(), b); // good

1. https://stackoverflow.com/questions/2923272/how-to-convert-vector-to-array
2. https://iq.opengenus.org/convert-vector-to-array-in-cpp/

Python float to hexadecimal & hexadecimal to float, and default flaot vs. fp32

 import struct

def float_to_hex(f):

    return hex(struct.unpack('<I', struct.pack('<f', f))[0])

def hex_to_float(h):

    return struct.unpack('!f', bytes.fromhex(h))[0]

hex_val = "0xbf557ca4"

float_val = hex_to_float(hex_val.replace("0x", ""))

print(f"-0.8339 -> 0xbf557ca4 -> {float_val} <- -0.8339331150054932")

----

-0.8339 -> 0xbf557ca4 -> -0.8339331150054932 <- -0.8339331150054932

import numpy as np

fp32_value = np.float32(-0.8339)

print(f"-0.8339 -> fp32: {fp32_value}")

----

output: -0.8339 -> fp32: -0.833899974822998

fp64_value = -0.8339

print(f"-0.8339 -> fp64: {fp64_value}")

----

output: -0.8339 -> fp64: -0.8339

Quantization FP16, FP8, or INT8

In this article, you can download many papers.

Floating-point arithmetic for AI inference — hit or miss?

• FP8 versus INT8 for efficient deep learning inference
• With Shared Microexponents, A Little Shifting Goes a Long Way
• The case for 4-bit precision: k-bit Inference Scaling Laws

About LoRA

LoRA (Low-Rank Adaptation)

- Paper

[Current]

[Before]

• Measuring the Intrinsic Dimension of Objective Landscapes, 2018
• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, 2020 

[After]

Towards A Unified View of Parameter-Efficient Transfer Learning, 2022

---- 

- YouTube

 https://www.youtube.com/watch?v=dA-NhCtrrVE

 https://www.youtube.com/watch?v=BJqwmDpa0wM

https://www.youtube.com/watch?v=t509sv5MT0w

----

The case for 4-bit precision: k-bit Interence Scaling Laws

Parameter-Efficient Transfer Learning for NLP

Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

QLoRA: Efficient Finetuning of Quantized LLMs

https://www.youtube.com/watch?v=X4VvO3G6_vw

----

Intrinsic Dimensionality Ezplanins the Effectiveness of Language Model Fine-Tuning

VLA: variable length array

VLA: variable length array Variable-length array - Wikipedia

and 

c - What's the difference between variable length array and dynamic memory allocation? - Stack Overflow

VLA: allocate memory in stack area 

malloc(): allocate memory in heap area

Turbo encoder / decoder

 From the Web

- PSI00073.pdf (ucsc.edu)

- Microsoft Word - From BCJR to turbo_v3 (up.pt)

- AWCNS.dvi (southampton.ac.uk)

---

(Korean)

3세대 이동통신에 적합한 Log-MAP 터보 디코더 설계 (catholic.ac.kr)

Implementation of Turbo Decoder using SOVA Algorithm - 대한전자공학회 ISOCC - 대한전자공학회 : 논문 - DBpia

[논문]3세대 이동통신에 적합한 슬라이딩 윈도우 로그 맵 터보 디코더 설계 (kisti.re.kr)

An efficient method for Turbo Decoder design using Block Combining -Proceedings of the IEEK Conference | Korea Science

[논문]다양한 Design Issue에 대한 터보 디코더의 성능분석 (kisti.re.kr)

매체보기 | 서강대학교 로욜라도서관 (sogang.ac.kr)

매체보기 | 서강대학교 로욜라도서관 (sogang.ac.kr)

LTE turbo N_ir

 

Please refer to TS 136 212 - V14.2.0 - LTE; Evolved Universal Terrestrial Radio Access (E-UTRA); Multiplexing and channel coding (3GPP TS 36.212 version 14.2.0 Release 14) (etsi.org)

AVX - ternary operation

This site is very useful for me.

 AVX512: ternary functions evaluation (0x80.pl)

SIMD - transpose

 Intel SIMD (AVX) transpose intrinsics

Intel® Intrinsics Guide

- _shuffle_* (mask)

- permute (mask)

- _unpacklo_, _unpackhi_

- _MM_TRANSPOSE4_PS & c++ - Fastest way to transpose 4x4 byte matrix - Stack Overflow

Example

https://godbolt.org/z/xzfPqnrdo