AiVibeTransformer (Neural Network Architecture)
The transformer is a neural network architecture based on self-attention, which lets every element of a sequence weigh its relationship to every other element. Introduced in 2017, it underpins today's large language models and many vision models.
Text, image patches or other data are converted into tokens and embedded as vectors, with positional information added. Each transformer layer applies multi-head self-attention, which computes how strongly each token should attend to every other token, followed by a feed-forward network, with residual connections and normalisation. The original design, published in 2017 by Vaswani and colleagues at Google, had an encoder and a decoder; BERT-style models use only the encoder, and GPT-style generative models use only the decoder.
Because attention processes all tokens in parallel during training, transformers scale efficiently on GPUs, which enabled large language models, vision transformers, speech models such as Whisper, and multimodal and vision-language-action models. In industry they power chat assistants, document understanding, code generation and a growing range of time-series and inspection models.
Standard self-attention cost grows quadratically with sequence length, which limits context windows and drives research into more efficient attention variants. Transformers usually need large datasets or pre-training to perform well. The term is unrelated to electrical transformers.
Key points
- Built on self-attention between all tokens in a sequence
- Introduced in a 2017 paper by Vaswani and colleagues at Google
- Encoder-only, decoder-only and encoder-decoder variants exist
- Attention cost grows quadratically with sequence length
Where AiVibe comes in
AiVibe delivers AI and machine learning services, chatbots and virtual assistants with RAG, MCP tools and voice, AI quality management including bias detection and model validation, and the AIMURUGA AI agent, and builds Intel-based edge AI devices using the Intel Distribution of OpenVINO toolkit.
Related terms
- Large Language Model (LLM)AI & Machine Learning
- Tokens and Tokenisation (LLMs)AI & Machine Learning
- Context WindowAI & Machine Learning
- Vision-Language Model (VLM)AI & Machine Learning
- Deep LearningAI & Machine Learning
- Embeddings (Vector Embeddings)AI & Machine Learning