ICASSP Short Course Education Form

Short Course SC-6 Quiz

Transformer Architectures for Multimodal Signal Processing and Decision Making

Date of Completion*

1. The self attention operation used by Transformers aggregates information from the entire input sequence. However, there are certain scenarios where future positions in a sequence are not available or should not be used. To avoid information leak, attention masking can be applied. A subset of the following algorithms applies this attention masking strategy; please select all algorithms that apply:*

A) BERT

B) Decision Transformers

C) Vision Transformers (ViT)

D) GPT-2

2. To apply transformers to machine translation and image classification, we discussed the concept of positional encodings and positional embeddings in the lectures. However, positional information is not always needed. Please select one of the following scenarios when the positional information does not need to be provided:*

A) Object detection with visual transformers

B) Document classification based only on citation information such as Cora

C) Automatic speech recognition with an audio spectrum transformer

D) Document classification based on its content with BERT

3. Many recent variants of Transformer aim to make it more efficient and handle longer input sequences. The vanilla implementation of Transformer scales quadratically with respect to the input sequence length for. Please select all that apply:*

A) Number of model parameters per Transformer block

B) Total model size

C) Memory usage

D) FLOPs

4. Transformers nowadays are scaled up to have enormous model sizes. For example, GPT-3 has around 175 billion parameters! What are some approaches to successfully training a model of such a big size? Select all that apply:*

A) Collect big datasets

B) Stronger data augmentation and model regularization

C) Self-supervised pre-training

D) Multi-modal and cross-modal pre-training

5. There are many attempts to understand how transformers learn effective representations for the downstream tasks. Please select all methods that can be applied to understand transformers:*

A) Visualize the attention heatmaps

B) Build probing classifiers on the intermediate activations

C) Measure representation similarity across layers

D) Replace the learned self-attention operation with random attention

6. Why is the dot product divided by the square root of the dimension of the keys and queries in the following equation copied and pasted from the Transformer paper, "Attention Is All You Need"?*

A) Because the authors noticed that a large query/key dimension allows for large dot products, which end up in a flat part of the softmax. The flat areas cause optimization problems as the gradients are extremely small.

B) Because the square root of the query/key dimension normalizes the dot products between different query-key pairs, making them more comparable than otherwise.

7. In Transformer, a self-attention head outputs a sequence of the same length as the input sequence.*

A) True

B) False

8. Which of the following neural architectures can model sequence data? Select all that apply.*

A) Recurrent neural networks, including LSTMs, GRUs, etc.

B) MLPs.

C) CNNs.

D) Transformers.

9. Transformer is a*

A) Feedforward neural network

B) Recurrent neural network

10. Which of the following statements are correct? Please select all that apply.*

A) Sharpness-aware minimization (SAM) and surrogate-gap guided SAM (GSAM) are two optimization approaches

B) SAM and GSAM improve a neural network’s generalization by avoiding the network’s convergence at sharp local minima

C) SAM and GSAM incurs backpropagation twice in each iteration of the training

D) ViT benefits from SAM and GSAM more than ResNet does

Speaker and Course

Please indicate the extent to which you disagree or agree with the following statements:

1- Strongly Disagree

2- Disagree

3- Neutral

4- Agree

5- Strongly Agree

Additional Comments:

Full Name (as you would like it to appear on CEU/PDH certificate)*

Email Address*

PE License Number (Mandatory if licensed in the state of Florida or New York)

Can we follow up with you in the future regarding how this course impacted your work and career?*

Yes

No

Send me a copy of my responses