PyTorch image captioning system trained on the VizWiz dataset (7,750 images from people who are blind). Two architectures designed and compared: MobileNetV3 + LSTM baseline and GoogLeNet + LSTM with Luong attention.
A deep learning image captioning project completed as part of the Deep Learning subject in the Master of Data Science and Innovation at UTS. The dataset is VizWiz-Captions (7,750 images captured by people who are blind, frequently blurry or poorly lit), which presents a harder domain shift from ImageNet than standard captioning benchmarks. I designed and trained two architectures. Model 1 uses a MobileNetV3-Small encoder with partial unfreezing (features[9:12] trainable) projecting to 256 dimensions, paired with a single-layer LSTM decoder without attention following the Show and Tell baseline. Model 2 replaces the encoder with GoogLeNet exposing a 7x7 spatial grid (49 patches at 512 dims) and adds Luong General attention over those patches, allowing the decoder to focus on spatially relevant image regions at each decoding step. Both use Adam with differential learning rates, beam search (k=3), and ReduceLROnPlateau scheduling. Model 1 achieved stronger overall BLEU and ROUGE-L scores due to its leaner encoder converging more reliably within the 10-epoch budget; Model 2's attention heatmaps showed spatial grounding and narrowed the gap at BLEU-4, indicating the mechanism worked but needed more training time to fully show. Both models exceeded the expected VizWiz baseline ranges.
