Quiz: Attention Heatmap

1. What does an attention heatmap visualize in NLP models?

Relationships and focus between words based on attention scores

Word embeddings

Model accuracy over time

Token frequency

2. What do the rows and columns represent in an attention heatmap?

Words attending and words being attended to

Document length and punctuation

Layer count and neuron ID

Input and output token embeddings

3. What does a brighter color in an attention heatmap cell typically indicate?

More padding in the sequence

A higher attention score between two tokens

A longer input sentence

The presence of a special token

4. Why do Transformer models use multi-head attention?

To improve dropout rate

To reduce the embedding size

To speed up tokenization

To capture different relationships and features across multiple perspectives

5. What is the role of attention heads in a heatmap visualization?

They mark punctuation importance

They replace position embeddings

Each head focuses on a different aspect of word relationships

They determine the sentence length

6. How does training affect the attention heatmap over time?

It removes less frequent words

It fades to zero values

It shows sharper and more meaningful focus areas

It becomes more random

7. What is typically used to compute attention scores in BERT?

RNN cell outputs

Dot products between queries and keys

One-hot vector comparisons

Convolution operations

8. How are attention head outputs typically combined in a Transformer?

By summing

By concatenation followed by a linear transformation

By averaging

By multiplying with position embeddings

9. Which tokens in a sentence are often used to evaluate attention patterns?

Masked or content-heavy tokens

Stop words only

Output tokens

Only [PAD] tokens

10. Why is the attention heatmap a useful interpretability tool in NLP?

It visualizes how models distribute focus across input tokens

It predicts masked tokens

It increases training speed

It measures GPU usage

Quiz 30: Attention Heatmap