Oregon State University



Event Details

MS Final Examination – Dapeng Li

Wednesday, November 1, 2017 2:00 PM - 4:00 PM

Multimodal Machine Translation
We study the task of “multimodal translation” where the input is an image paired with its source-language caption, and the output is the target-language caption. This task has real applications in social networks such as Facebook where users post photos with comments in various languages. The key difference between this task and conventional machine translation is that we have the corresponding images as additional information for each input sentence. We develop a simple but effective system which takes an image and runs it through a convolutional neural network (CNN), which results in an image representation as a vector. This image vector is fed it into both the encoding of source-language sentence and target-language generation. We report our system’s performance for English-to-French and English-to-German with Flickr30K (in-domain) and MSCOCO (out-of-domain) datasets. Our system achieves the best performance in TER metric for English-German for the MSCOCO dataset.

Major Advisor: Lizhong Chen
Committee: Fuxin Li
Committee: Yue Zhang

Kelley Engineering Center (campus map)
Calvin Hughes
1 541 737 3168
Calvin.Hughes at oregonstate.edu
Sch Elect Engr/Comp Sci
This event appears on the following calendars: