This event has ended. Visit the official site or create your own event on Sched.
Tom Tom champions civic innovation, creativity, and entrepreneurship in America’s hometowns.

[Back to Tom Tom Festival]
Back To Schedule
Thursday, April 11 • 10:50am - 11:05am
A Deep Learning Approach to Meme Generation

Log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
Natural language is used in day to day life to evoke and convey thoughts, feelings, and emotions. One of the emotions commonly expressed through language is humor. It is a universal phenomenon that occurs in all languages and has a wide appeal.With the advent of social media platforms, new forms of creative expression such as memes and gifs have become really popular. 
Generating humor is challenging because it is not always direct. It can have different styles such as sarcasm, satire, irony etc. In the context of memes, the joke has to be expressed in a few words, usually 20 or less and support the background image. Hence, the syntax of the language used in memes is different and does not follow the conventional language structure. 
In this project, we introduce a model using encoder-decoder framework with attention mechanism to generate memes where the attention mechanism is applied on an abstract vector space representing the image and thematic linguistic representation of the meme. 
An attention based encoder decoder framework was implemented where the encoder extracts an abstract representation of the image from a pre-trained Resnet101 model while the decoder was composed a an LSTM network that generated the output word by word. The attention mechanism was incorporated to inspect and take into account the presence of certain objects that guide the process of generating any given word by focusing on a certain area of the image. 
To provide more context to the memes, in addition to the above architecture pre-trained GloVe embedded vectors were used as the initial weights for the input text. Unconventional words like 'bae' were pre-assigned with a random vector. All of these vectors were again trained to fine-tune their weights. This initialization approach assisted in faster convergence of embedding weight matrix. 
A descriptive representation of the meme was introduced in the encoder in order to better capture the sentiment and thematic structure of the meme. Keywords representing the sentiment of the meme were converted to GloVe embeddings and concatenated with the image vector. This encoding was passed to the attention mechanism so that it could generate words based on an abstract combination of the pictorial and thematic properties of the meme. 

You need this ticket from Eventbrite to sign up: Applied Machine Learning Conference.

avatar for Murugesan Ramakrishnan

Murugesan Ramakrishnan

Graduate Student, Data Science Institute, University of Virginia
Murugesan is a current Graduate student at the Data Science Institute, University of Virginia. His research interest lies in solving problems in Natural Language Processing and Computer Vision using Deep Learning techniques, He also has a considerable experience in consulting solving... Read More →
avatar for Sri Vaishnavi Vemulapalli

Sri Vaishnavi Vemulapalli

Graduate Student, Data Science Institute, University of Virginia
Sri is a Graduate Student pursuing her Masters in Data Science at the Data Science Institute, University of Virginia. She has a background in Information Systems and experience as a Software Development Engineer. Her areas of interest are Natural Language Processing and Visual Re... Read More →


Thursday April 11, 2019 10:50am - 11:05am EDT
Violet Crown: Theater 5 200 W Main St, Charlottesville, VA 22902, USA