Industry Use-Case of Neural Networks

Welcome to my Blog, Today we will see “what is the neural networks” in brief and industrial use - Case.

🔰Task Description📄

📌Research for industry use cases of Neural Networks and create a blog, Article or Video elaborating how it works.

The blog commences with a brief introduction on the working of neural networks. We have tried to keep it very simple yet effective.

An Introduction : Neural Network

Neural networks represent deep learning using artificial intelligence. Certain application scenarios are too heavy or out of scope for traditional machine learning algorithms to handle. As they are commonly known, Neural Network pitches in such scenarios and fills the gap.

Artificial neural networks are inspired from the biological neurons within the human body which activate under certain circumstances resulting in a related action performed by the body in response. Artificial neural nets consist of various layers of interconnected artificial neurons powered by activation functions which help in switching them ON/OFF. Like traditional machine algorithms, here too, there are certain values that neural nets learn in the training phase.

Briefly, each neuron receives a multiplied version of inputs and random weights which is then added with static bias value (unique to each neuron layer), this is then passed to an appropriate activation function which decides the final value to be given out of the neuron. There are various activation functions available as per the nature of input values. Once the output is generated from the final neural net layer, loss function (input vs output)is calculated and backpropagation is performed where the weights are adjusted to make the loss minimum. Finding optimal values of weights is what the overall operation is focusing around. Please refer to the following for better understanding-

Weights are numeric values which are multiplied with inputs. In backpropagation, they are modified to reduce the loss. In simple words, weights are machine learnt values from Neural Networks. They self-adjust depending on the difference between predicted outputs vs training inputs.
Activation Function is a mathematical formula which helps the neuron to switch ON/OFF.

• Input layer represents dimensions of the input vector.
• Hidden layer represents the intermediary nodes that divide the input space into regions with (soft) boundaries. It takes in a set of weighted input and produces output through an activation function.
• Output layer represents the output of the neural network.

Types of Neural Networks

There are many types of neural networks available or that might be in the development stage. They can be classified depending on their: Structure, Data flow, Neurons used and their density, Layers and their depth activation filters etc.

A. Perceptron

Advantages of Perceptron
Perceptrons can implement Logic Gates like AND, OR, or NAND

Disadvantages of Perceptron
Perceptrons can only learn linearly separable problems such as boolean AND problem. For non-linear problems such as boolean XOR problem, it does not work.

B. Feed Forward Neural Networks

Applications on Feed Forward Neural Networks:

• Simple classification (where traditional Machine-learning based classification algorithms have limitations)
• Face recognition [Simple straight forward image processing]
• Computer vision [Where target classes are difficult to classify]
• Speech Recognition

Advantages of Feed Forward Neural Networks

1. Less complex, easy to design & maintain
2. Fast and speedy [One-way propagation]
3. Highly responsive to noisy data

Disadvantages of Feed Forward Neural Networks:

1. Cannot be used for deep learning [due to absence of dense layers and back propagation.

C. Multilayer Perceptron

Applications on Multi-Layer Perceptron

• Speech Recognition
• Machine Translation
• Complex Classification

Advantages on Multi-Layer Perceptron

1. Used for deep learning [due to the presence of dense fully connected layers and back propagation]

Disadvantages on Multi-Layer Perceptron:

1. Comparatively complex to design and maintain

Comparatively slow (depends on number of hidden layers)

D. Convolutional Neural Network

Applications on Convolution Neural Network

• Image processing
• Computer Vision
• Speech Recognition
• Machine translation

Advantages of Convolution Neural Network:

1. Used for deep learning with few parameters
2. Less parameters to learn as compared to fully connected layer

Disadvantages of Convolution Neural Network:

• Comparatively complex to design and maintain
• Comparatively slow [depends on the number of hidden layers]

E. Radial Basis Function Neural Networks

Radial Basis Function Network consists of an input vector followed by a layer of RBF neurons and an output layer with one node per category. Classification is performed by measuring the input’s similarity to data points from the training set where each neuron stores a prototype. This will be one of the examples from the training set.

Application: Power Restoration
a. Powercut P1 needs to be restored first
b. Powercut P3 needs to be restored next, as it impacts more houses
c. Powercut P2 should be fixed last as it impacts only one house

F. Recurrent Neural Networks

Applications of Recurrent Neural Networks

• Text processing like auto suggest, grammar checks, etc.
• Text to speech processing
• Image tagger
• Sentiment Analysis
• Translation
• Designed to save the output of a layer, Recurrent Neural Network is fed back to the input to help in predicting the outcome of the layer. The first layer is typically a feed forward neural network followed by recurrent neural network layer where some information it had in the previous time-step is remembered by a memory function. Forward propagation is implemented in this case. It stores information required for it’s future use. If the prediction is wrong, the learning rate is employed to make small changes. Hence, making it gradually increase towards making the right prediction during the backpropagation.

Advantages of Recurrent Neural Networks

1. Model sequential data where each sample can be assumed to be dependent on historical ones is one of the advantage.
2. Used with convolution layers to extend the pixel effectiveness.

Disadvantages of Recurrent Neural Networks

1. Gradient vanishing and exploding problems
2. Training recurrent neural nets could be a difficult task
3. Difficult to process long sequential data using ReLU as an activation function.

Improvement over RNN: LSTM (Long Short-Term Memory) Networks

LSTM networks are a type of RNN that uses special units in addition to standard units. LSTM units include a ‘memory cell’ that can maintain information in memory for long periods of time. A set of gates is used to control when information enters the memory when it’s output, and when it’s forgotten. There are three types of gates viz, Input gate, output gate and forget gate. Input gate decides how many information from the last sample will be kept in memory; the output gate regulates the amount of data passed to the next layer, and forget gates control the tearing rate of memory stored. This architecture lets them learn longer-term dependencies

G. Sequence to sequence models

A sequence to sequence model consists of two Recurrent Neural Networks. Here, there exists an encoder that processes the input and a decoder that processes the output. The encoder and decoder work simultaneously — either using the same parameter or different ones. This model, on contrary to the actual RNN, is particularly applicable in those cases where the length of the input data is equal to the length of the output data. While they possess similar benefits and limitations of the RNN, these models are usually applied mainly in chatbots, machine translations, and question answering systems.

H. Modular Neural Network

Applications of Modular Neural Network

1. Stock market prediction systems
2. Adaptive MNN for character recognitions
3. Compression of high level input data

A modular neural network has a number of different networks that function independently and perform sub-tasks. The different networks do not really interact with or signal each other during the computation process. They work independently towards achieving the output.

As a result, a large and complex computational process are done significantly faster by breaking it down into independent components. The computation speed increases because the networks are not interacting with or even connected to each other.

Advantages of Modular Neural Network

1. Efficient
2. Independent training
3. Robustness

Disadvantages of Modular Neural Network

1. Moving target Problems

Google — Neural Networks and ‘Machines That Dream’

These days, it’s probably easier to list areas of scientific R&D that Google — or, rather, parent company Alphabet — isn’t working on, rather than trying to summarize Google’s technological ambition.

Needless to say, Google has been very busy in recent years, having diversified into such fields as anti-aging technology, medical devices, and — perhaps most exciting for tech nerds — neural networks.

The most visible developments in Google’s neural network research has been the DeepMind network, the “machine that dreams.” It’s the same network that produced those psychedelic images everybody was talking about a while back.

According to Google, the company is researching “virtually all aspects of machine learning,” which will lead to exciting developments in what Google calls “classical algorithms” as well as other applications including natural language processing, speech translation, and search ranking and prediction systems.

Salesforce : New neural network building block allows faster and more accurate text understanding

In deep learning, there are two very different ways to process input (like an image or a document). Typically, images are processed all at once, with the same kind of computation happening for every part of the image simultaneously. But researchers have usually assumed that you can’t do this for text data: that you need to process it the way people read text: word by word, from the beginning to the end, taking context into account as you go. In this paper, we showed that this doesn’t have to be true: you can process it all at once like an image, then quickly take into account context and fix the parts that you got wrong. Since the conventional, one-word-at-a-time approach to deep learning for text uses what’s called a “recurrent neural network,” we called our system a “quasi-recurrent neural network,” or QRNN. Because most of the computation happens all at once, in parallel, it’s up to 16 times faster than the old approach. Amazingly, it also gives better results than conventional deep-learning models for all three tasks that we tried (sentiment analysis, next-word prediction, and translation).

The diagram above shows the basic structure of our model. A continuous block of red means that complicated, slow computations can proceed in parallel (i.e., much faster). Blue signifies fast, simple functions (in the case of the QRNN, the components that “fix” the parts the out-of-context red computations got wrong). Depending on the length of text inputs we need to process (whether they’re sentences, paragraphs, etc.) and other properties of the dataset, our architecture, shown on the right, runs anywhere between 30% faster and 16 times faster than the architecture usually used for text, shown on the left. The advantage is highest for long sequences like paragraphs.

Accuracy comparison

We compared our new building block with the traditional LSTM architecture (a special type of recurrent neural network) by building a pair of models for each of three tasks. In each case, the models are identical other than the fact that one uses the LSTM and one uses the QRNN. In all three cases, the one with the QRNN performs better:

Results on IMDb movie review sentiment analysis dataset.

Results on Penn Treebank language analysis (next-word prediction) task.

Results on IWSLT (TED talk transcript) German to English machine translation task.

The Future of Machine Learning

One of the main problems with rapid technological advancement is that, for whatever reason, we end up taking these leaps for granted. Some of the applications of machine learning listed above would have been almost unthinkable as recently as a decade ago, and yet the pace at which scientists and researchers are advancing is nothing short of amazing.

So, what’s next in machine learning trends?

I hope you like it !!!

Thank you

keep learning and keep sharing!!!