Hi everyone! For those who do not know me yet, my name is Francois, I am a Research Scientist at Meta. I have a passion for explaining advanced AI concepts and making them more accessible.
Today, let’s dive into one of the most significant contribution in the field of Computer Vision: the Vision Transformer (ViT).
This post focuses on the state-of-the-art implementation of the Vision Transformer since its release. To fully understand how a ViT works, I strongly recommend reading my other post on the theoretical foundations: The Ultimate Guide to Vision Transformers
Let’s start with the most well-known building block of the Transformer Encoder: the Attention Layer.
class Attention(nn.Module):
def __init__(self, dim, heads=8, dim_head=64, dropout=0.):
super().__init__()
inner_dim = dim_head * heads # Calculate the total inner…