Feedback LSTM Network Based on Attention for Image Description Generator

• 作者: Bingyu Cao Fu Li Luhan Zhang Peirong Xu Xiaoru Wang Zhaowei Qu
• 来源期刊: 计算机、材料和连续体(英文)
• 学科: 工学
• 关键词: Image DESCRIPTION GENERATOR FEEDBACK LSTM NETWORK ATTENTION CBAM
• 年，卷（期）: 2019,（5）
• 页码范围: 575-589
• 页数: 15页
• 分类号: TP3

摘要

Images are complex multimedia data which contain rich semantic information.Most of current image description generator algorithms only generate plain description,with the lack of distinction between primary and secondary object,leading to insufficient high-level semantic and accuracy under public evaluation criteria.The major issue is the lack of effective network on high-level semantic sentences generation,which contains detailed description for motion and state of the principal object.To address the issue,this paper proposes the Attention-based Feedback Long Short-Term Memory Network(AFLN).Based on existing codec framework,there are two independent sub tasks in our method:attention-based feedback LSTM network during decoding and the Convolutional Block Attention Module(CBAM)in the coding phase.First,we propose an attentionbased network to feedback the features corresponding to the generated word from the previous LSTM decoding unit.We implement feedback guidance through the related field mapping algorithm,which quantifies the correlation between previous word and latter word,so that the main object can be tracked with highlighted detailed description.Second,we exploit the attention idea and apply a lightweight and general module called CBAM after the last layer of VGG 16 pretraining network,which can enhance the expression of image coding features by combining channel and spatial dimension attention maps with negligible overheads.Extensive experiments on COCO dataset validate the superiority of our network over the state-of-the-art algorithms.Both scores and actual effects are proved.The BLEU 4 score increases from 0.291 to 0.301 while the CIDEr score rising from 0.912 to 0.952.