Computer Vision News - November 2018

Back in 2014, we all fell in love with RNN and LSTM networks and their various implementations: their potential seemed unbounded. What does their future look like? Is this the beginning of their decline? In 2014 RNN and LSTM came back from the dead, achieving impressive results on various tasks, thanks in large part to LSTM, as described in our overview of the field . For several years they were the go-to solution for difficult problems like seq2seq translation, and achieved impressive results in text-to-speech tasks. However, two years later, technologies like ResNet and attention were proposed. The research community realized that LSTM is in fact a type of residual connection. The attention mechanism became one of the most crucial components of neural networks when dealing with sequences, whether they are sequences of video, images, audio, text or any other type of data. Let’s try and understand the attention mechanism in a nutshell (using a simple Keras code sample). The attention mechanism differentially weighs different parts of the input to the network in order to direct the network element, or nudge it, towards the most relevant parts of the input for classification, translation etc. (the attention weights will differ depending on the task). To get a better idea of what we’re talking about, let’s take a look at the following code snippet, which demonstrates an attention mechanism for a very simple network . The simple attention mechanism is implemented using a Dense layer (first line of code), the input is multiplied by the output of the Dense layer -- and the product of this multiplication is the attention mechanism. 20 Focus on: RNN and LSTM networks Focus on by Assaf Spanier “What does the future hold for RNN and LSTM networks?” Computer Vision News