Brain-inspired multimodal learning based on neural networks

• 作者: Bo Zhang Chang Liu Fuchun Sun
• 来源期刊: 临床转化神经医学(英文版)
• 学科: 医学
• 年，卷（期）: 2018,（1）
• 页码范围: 61-72
• 页数: 12页
• 分类号: R338

摘要

Modern computational models have leveraged biological advances in human brain research. This study addresses the problem of multimodal learning with the help of brain-inspired models. Specifically, a unified multimodal learning architecture is proposed based on deep neural networks, which are inspired by the biology of the visual cortex of the human brain. This unified framework is validated by two practical multimodal learning tasks: image captioning, involving visual and natural language signals, and visual-haptic fusion, involving haptic and visual signals. Extensive experiments are conducted under the framework, and competitive results are achieved.