Quiz: Attention Score

1. What does the attention mechanism allow a Transformer model to do?

Generate random outputs

Forget earlier inputs

Reduce sequence length

Focus on specific parts of the input sequence

2. What are the three main components used in scaled dot-product attention?

Tokens, Layers, Embeddings

LSTM, GRU, CNN

Convolutions, Pooling, Padding

Keys, Queries, Values

3. Why is the dot product between Query and Key scaled?

To prevent extremely large values that impact softmax

To ensure all values are integers

To reduce training time

To normalize dimensions for positional encoding

4. What is the role of the softmax function in the attention mechanism?

It drops random elements from vectors

It transforms scores into probabilities

It multiplies all attention scores

It stores attention vectors for reuse

5. What happens after applying softmax to the attention scores?

The inputs are encoded again

The weights are multiplied by the value vectors

The sequence is reversed

The output is one-hot encoded

6. What does the final output of the attention mechanism represent?

A compressed audio signal

A raw copy of the input sequence

A weighted combination of values based on relevance

A frequency distribution

7. In self-attention, how many other tokens can each token attend to?

Only one

Only the previous token

None

All other tokens in the sequence

8. What mathematical operation is used to compare queries with keys?

Logarithmic sum

Cross product

Concatenation

Dot product

9. Why is attention useful in tasks like machine translation?

It removes punctuation

It compresses long documents

It segments speech input

It enables the model to align and focus on relevant source words

10. What does a higher attention score between two tokens indicate?

They appear at different sentence boundaries

They are semantically unrelated

They should be ignored

The model considers them more relevant to each other

Quiz 29: Attention Score