Recurrent Neural Network

import numpy as np
from keras.models import Sequential
from keras.layers import SimpleRNN, Dense

# Sample data: sequences of 5 numbers
X = np.array([
[1, 2, 3, 4, 5],
[2, 3, 4, 5, 6],
[3, 4, 5, 6, 7],
[4, 5, 6, 7, 8]
])

# Labels: the 6th number in the sequence
y = np.array([6, 7, 8, 9])

# Reshape X to fit the RNN input requirements: [samples, time steps, features]
X = X.reshape((X.shape[0], X.shape[1], 1))

# Build the model
model = Sequential([
SimpleRNN(50, input_shape=(5, 1)),
Dense(1)
])

model.compile(optimizer='adam', loss='mean_squared_error')

# Train the model
model.fit(X, y, epochs=1000, verbose=0)
print(model.summary())

output:
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param # 
=================================================================
simple_rnn_1 (SimpleRNN) (None, 50) 2600 

dense_1 (Dense) (None, 1) 51 

=================================================================
Total params: 2651 (10.36 KB)
Trainable params: 2651 (10.36 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
None

# Predicting a new sequence
test_input = np.array([5, 6, 7, 8, 9])
test_input = test_input.reshape((1, 5, 1))
predicted_number = model.predict(test_input, verbose=0)

print(f"Predicted number: {predicted_number.flatten()[0]}")

output:
Predicted number: 9.2970552444458

Explanation of the Example

1. Data Preparation:
   • We create some simple sequences of numbers to demonstrate the concept. Each sequence is a list of 5 consecutive numbers.
   • We reshape our input X to fit the RNN input shape which is [samples, time steps, features]. In our case, each sequence has 5 time steps and 1 feature per step.
2. Model Building:
   • We use the Sequential model from Keras, adding a SimpleRNN layer followed by a Dense layer. The SimpleRNN layer has 50 units (neurons).
   • The Dense layer has 1 unit and outputs the next number in the sequence.
3. Model Compilation and Training:
   • We compile the model using the Adam optimizer and mean squared error loss function, which is common for regression tasks.
   • The model is then trained on the data for 1000 epochs.
4. Prediction:
   • We predict the next number in a new sequence [5, 6, 7, 8, 9]. The model predicts the next number based on the learned sequences.
   • We reshape the input for the prediction to match the expected input shape for the RNN.

This example demonstrates how RNNs can predict the next step in a sequence based on learned patterns from previous data. The use of RNNs is not limited to numerical data; they are also highly effective in processing and generating text, handling speech data, and more.

RNN Applications

Recurrent Neural Networks (RNNs) have a wide array of applications in the real world, especially in domains where data is inherently sequential. Here are several prominent applications and examples of how RNNs are used in data analysis and beyond:

1. Natural Language Processing (NLP)

RNNs are extensively used in NLP for tasks that involve sequences of words or characters:

• Text Generation: RNNs can be trained on a corpus of text and then used to generate new text that mimics the style and content of the training set. This is useful for creative writing aids, automated story generation, and more.
• Machine Translation: RNNs are part of architectures that translate text from one language to another. They can model the sequence of words in both the source and target languages, helping maintain contextual meaning across translations.
• Speech Recognition: Translating spoken language into text is a classic RNN application. The network processes audio signals segmented into time frames and predicts words or phonemes sequence by sequence.

2. Time Series Prediction

In fields like finance, meteorology, and engineering, where data points are sequentially correlated:

• Stock Prices Prediction: RNNs can analyze the historical price data of stocks and predict future movements. They consider the sequence of price changes and other factors like trading volume over time.
• Weather Forecasting: Meteorological data (temperature, humidity, wind speed, etc.) is sequential and can be modeled using RNNs to predict future weather conditions.

3. Healthcare

RNNs analyze sequential data for various applications in healthcare:

• Medical Diagnosis: EHR (Electronic Health Records) contain time-stamped entries that can be analyzed using RNNs to predict disease progression or patient outcomes.
• ECG Analysis: RNNs can be used to predict cardiac abnormalities by analyzing the sequential data of heartbeats recorded in electrocardiograms.

4. Finance

In addition to stock prediction, RNNs find applications in other areas of finance:

• Algorithmic Trading: RNNs can be trained to make buy or sell decisions based on the sequence of market data.
• Credit Scoring: Analyzing the sequence of a person’s financial actions to predict creditworthiness or the likelihood of default.

5. Video Processing

Sequence processing capabilities of RNNs extend to video where frames are sequences in time:

• Activity Recognition: Predicting the type of activity being performed in a video by analyzing the sequence of video frames.
• Video Captioning: Generating descriptive text for the contents of a video over time.

6. Music Generation

RNNs can learn from sequences of musical notes and generate new music pieces:

• Automatic Music Composition: Composing music by learning styles from various pieces and generating new compositions that reflect learned patterns.

7. Anomaly Detection

In any time-dependent system, RNNs can help detect anomalies:

• Network Security: Detecting unusual sequences of network traffic that could indicate a cyber attack.
• Industrial Equipment Monitoring: Predicting failures in machinery by detecting deviations from normal operational patterns.

RNNs are powerful because of their ability to remember past information and use this context to make decisions about new data. This characteristic makes them ideal for any application where context and history significantly impact the current output.

A recurrent neural network (RNN) is a type of artificial neural network which uses 
sequential data or time series data. 

commonly used for ordinal temporal problems such as language translation, NLP, speech 
recognition, and image captioning. 

unque in that "memory" as RNN take information from prior inputs to influence the 
current input and output.

While traditional deep neural networks assume that inputs and outputs are independent 
of each other, the output of recurrent neural networks depend on the prior elements 
within the sequence. 

For example, "feeling under the weather" said by one. In order fro the idiom to make sense, 
it needs to be expressed in that specific order. RNN needs to accounts fot the position of 
each word in the idiom to predict the next word in the sequence. 
Another distinguishing characteristic of recurrent networks is that they share 
parameters across each layer of the network. While feedforward networks have 
different weights across each node, recurrent neural networks share the same 
weight parameter within each layer of the network. That said, these weights are 
still adjusted in the through the processes of backpropagation and gradient 
descent to facilitate reinforcement learning.

Through this process, RNNs tend to run into two problems, known as exploding gradients 
and vanishing gradients. These issues are defined by the size of the gradient, which 
is the slope of the loss function along the error curve. When the gradient is too small, 
it continues to become smaller, updating the weight parameters until they become 
insignificant—i.e. 0. When that occurs, the algorithm is no longer learning. 
Exploding gradients occur when the gradient is too large, creating an unstable model. 
In this case, the model weights will grow too large, and they will eventually 
be represented as NaN. One solution to these issues is to reduce the number of hidden 
layers within the neural network, eliminating some of the complexity in the RNN model.

Long short-term memory (LSTM): This is a popular RNN architecture, which was introduced 
by Sepp Hochreiter and Juergen Schmidhuber as a solution to vanishing gradient problem.
 In their paper (link resides outside ibm.com), they work to address the problem of 
long-term dependencies. That is, if the previous state that is influencing the current 
prediction is not in the recent past, the RNN model may not be able to accurately
 predict the current state. As an example, let’s say we wanted to predict the 
italicized words in following, “Alice is allergic to nuts. She can’t eat peanut butter.”
 The context of a nut allergy can help us anticipate that the food that cannot be eaten 
contains nuts. However, if that context was a few sentences prior, then it would make
 it difficult, or even impossible, for the RNN to connect the information. To remedy
 this, LSTMs have “cells” in the hidden layers of the neural network, which have
 three gates–an input gate, an output gate, and a forget gate. These gates control 
the flow of information which is needed to predict the output in the network. 
 For example, if gender pronouns, such as “she”, was repeated multiple times 
in prior sentences, you may exclude that from the cell state.

--------   from https://www.ibm.com/topics/recurrent-neural-networks

Sequence model
sequence data is everywhere like audio and text data
Sentiment classification is one example of seuential classification
Image captioning and machine translation are another example of sequence model.

Neurons with recurrence
Time step are passed on to next percetrons
A linkage of temporal memory
Recurrence relation

RNN
RNN have a state ht that is updated at each time step as a sequence is processed
The same function and set of parameters are used at every time step

embedding: transform indexes into a vector of fixed size
For example, this morning I took my cat for a walk.
cat can be mapped into [0, 1, 0, 0, 0, 0]

To model sequences, we need to 
to maintain the order
to keep long term depedence, 
to handle variable-length sequences
to share parameters across the sequence

Potentical issues
vanishing gradienst

Sequence model:
Attention based model focus on the most important features of the input data
Instead of looking at the whole. Feed forward neural network never have output 
fed back to the network, so it is not sufficient in certain types of data 
such as text data.

Sequences are data points which have special time relationship 
different parts of the input data occur in different time period

A tradition NN does not consider what happened in the past
predict the future values in a given sequence based on the past patterns in the sequence. 
have the ability to capture information about the past and store it in memory. It will 
then use this memory to predict future occurrences. Sequence models can predict multiple 
future values, if needed. 
There are also bidirectional models that can predict prior values in the sequence based 
on the values that happened after.

RNN architectures:
Gated recurrent units (GRU)
Long short-term memory (LTSM)
Bidirectional - name entity recognition

Many-to-many RNN - speech recognition
Many-to-one RNN - stock price prediction, sentiment analysis
Encoder-decoder RNN - trasnformer, machine translation, text summarization

# BASIC RNN Model

from keras.models import Sequential
from keras.layers import SimpleRNN,Dense
import tensorflow as tf

tf.random.set_seed(3)

#Create a Keras Model
price_model=Sequential()
#Add Simple RNN layer with 32 nodes
price_model.add(SimpleRNN(32, input_shape=(1,lookback)))
#Add a Dense layer at the end for output
price_model.add(Dense(1))

#Compile with Adam Optimizer. Optimize for minimum mean square error
price_model.compile(loss="mean_squared_error",
                 optimizer="adam",
                 metrics=["mse"])

#Print model summary
price_model.summary()

#Train the model
price_model.fit(train_req_x, train_req_y, 
             epochs=5, batch_size=1, verbose=1) 

#Evaluate the model
price_model.evaluate(test_req_x, test_req_y, verbose=1)

#Predict on the test dataset
predict_on_test = price_model.predict(test_req_x)

#Inverse the scaling to view results
predict_on_test = scaler.inverse_transform(predict_on_test)

# LTSM Model

from keras.models import Sequential
from keras.layers import LSTM,Dense
import tensorflow as tf

tf.random.set_seed(3)

#Create a Keras Model
ts_model=Sequential()
#Add LSTM
ts_model.add(LSTM(256, input_shape=(1,lookback)))
ts_model.add(Dense(1))

#Compile with Adam Optimizer. Optimize for minimum mean square error
ts_model.compile(loss="mean_squared_error",
                 optimizer="adam",
                 metrics=["mse"])

#Print model summary
ts_model.summary()

#Train the model
ts_model.fit(train_req_x, train_req_y, 
             epochs=5, batch_size=1, verbose=1)

#Evaluate the model
ts_model.evaluate(test_req_x, test_req_y, verbose=1)

#Predict for the training dataset
predict_on_train= ts_model.predict(train_req_x)
#Predict on the test dataset
predict_on_test = ts_model.predict(test_req_x)

#Inverse the scaling to view results
predict_on_train = scaler.inverse_transform(predict_on_train)
predict_on_test = scaler.inverse_transform(predict_on_test)

# RNN example- word embeddings

The traditional text models does not consider the semantics of word relationships
If the used words are not in the training sets, there is no way to know whether it is positive or negtive
Or, we need a huge corpora for all used cases
So new concept of word embeddings is presented to capute the associations of similar words

#Preprocess data for spam messages
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences


#Max words in the vocabulary for this dataset
VOCAB_WORDS=10000
#Max sequence length for word sequences
MAX_SEQUENCE_LENGTH=100

#Create a vocabulary with unique words and IDs
spam_tokenizer = Tokenizer(num_words=VOCAB_WORDS)
spam_tokenizer.fit_on_texts(spam_messages)


print("Total unique tokens found: ", len(spam_tokenizer.word_index))
print("Example token ID for word \"me\" :", spam_tokenizer.word_index.get("me"))

#Convert sentences to token-ID sequences
spam_sequences = spam_tokenizer.texts_to_sequences(spam_messages)

#Pad all sequences to fixed length
spam_padded = pad_sequences(spam_sequences, maxlen=MAX_SEQUENCE_LENGTH)

#Split into training and test data
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(
                                    spam_padded,spam_classes,test_size=0.2)


#Create a model
from tensorflow import keras
from tensorflow.keras import optimizers
from tensorflow.keras.regularizers import l2
from keras.layers import LSTM,Dense

#Setup Hyper Parameters for building the model
NB_CLASSES=2

model = tf.keras.models.Sequential()

model.add(keras.layers.Embedding(vocab_len,
                                 50, 
                                 name="Embedding-Layer",
                                 weights=[embedding_matrix],
                                 input_length=MAX_SEQUENCE_LENGTH,
                                 trainable=True))

#Add LSTM Layer
model.add(LSTM(256))
model.add(keras.layers.Flatten())

model.add(keras.layers.Dense(NB_CLASSES,
                             name='Output-Layer',
                             activation='softmax'))

model.compile(loss='categorical_crossentropy',
              metrics=['accuracy'])

model.summary()

#Make it verbose so we can see the progress
VERBOSE=1

#Setup Hyper Parameters for training
BATCH_SIZE=256
EPOCHS=10
VALIDATION_SPLIT=0.2

print("\nTraining Progress:\n------------------------------------")

history=model.fit(X_train,
          Y_train,
          batch_size=BATCH_SIZE,
          epochs=EPOCHS,
          verbose=VERBOSE,
          validation_split=VALIDATION_SPLIT)

print("\nEvaluation against Test Dataset :\n------------------------------------")
model.evaluate(X_test,Y_test)

Long Short-Term Memory (LSTM)

Sure, let’s break it down!

LSTM stands for Long Short-Term Memory. It’s a type of artificial intelligence (AI) algorithm, particularly useful for tasks where data comes in sequences, like sentences, time series, or music.

Think of LSTM as a smart unit that can remember important stuff for a long time and forget less important stuff quickly. It’s like having a super attentive memory system in a computer program.

Here’s a simple breakdown of how LSTM works:

1. Input Gate: This gate decides which information is important to keep from the current input.
2. Forget Gate: It decides what information to forget from the previous cell state (the long-term memory).
3. Output Gate: Determines what parts of the cell state to output as the prediction or the next short-term memory.
4. Cell State: This is like the memory of the LSTM. It runs through time and carries information from past inputs. It’s like your long-term memory in your brain.
5. Hidden State: This is the short-term memory or the output of the LSTM at a given time step.

LSTMs are great because they can learn long-term dependencies in data, which regular neural networks often struggle with. They’re commonly used in tasks like language translation, speech recognition, and even in controlling robots!

So, in a nutshell, LSTM is like a smart memory system within AI that helps it remember important things for a long time and forget less important stuff quickly, making it really handy for understanding sequences of data.

LSTM applications

LSTMs have found various real-world applications, including in finance and the stock market. Here are some examples:

1. Stock Price Prediction: LSTMs are used to predict stock prices based on historical data. They can analyze past stock prices, trading volumes, and other relevant factors to forecast future price movements. This helps traders and investors make informed decisions about buying, selling, or holding stocks.
2. Algorithmic Trading: LSTMs are employed in algorithmic trading systems to automate the process of buying and selling financial instruments, such as stocks, currencies, or commodities. These systems use LSTM models to analyze market data in real-time and execute trades based on predefined strategies.
3. Risk Management: LSTMs are utilized for risk management purposes in finance. They can analyze historical market data and identify potential risks, such as market volatility, credit default, or fraud. Financial institutions use LSTM-based models to assess and mitigate risks in their operations.
4. Credit Scoring: LSTMs are applied in credit scoring models to assess the creditworthiness of individuals or businesses. By analyzing past financial behavior and other relevant factors, LSTM models can predict the likelihood of default and help lenders make informed decisions about extending credit.
5. Fraud Detection: LSTMs are used in fraud detection systems to identify suspicious activities or transactions in real-time. By analyzing patterns in transaction data, LSTM models can flag potentially fraudulent behavior, such as unauthorized access, identity theft, or money laundering.
6. Portfolio Management: LSTMs are employed in portfolio management systems to optimize investment portfolios based on risk and return objectives. By analyzing historical market data and asset correlations, LSTM models can recommend optimal asset allocations to achieve desired investment goals.

These are just a few examples of how LSTMs are used in finance and the stock market. Overall, LSTMs are powerful tools for analyzing sequential data and making predictions, which are essential tasks in various financial applications.

Other Applications

LSTMs, or Long Short-Term Memory networks, have a wide range of real-world applications across various fields. Here are some examples:

1. Natural Language Processing (NLP):
   • Language Translation: LSTMs are used in machine translation systems to translate text from one language to another.
   • Sentiment Analysis: They help analyze sentiment in text data, such as social media posts or customer reviews, to understand public opinion or customer feedback.
   • Text Generation: LSTMs can generate human-like text, such as generating product descriptions, news articles, or creative writing.
2. Speech Recognition:
   • Automatic Speech Recognition (ASR): LSTMs are employed in ASR systems to transcribe spoken language into text, enabling applications like virtual assistants (e.g., Siri, Google Assistant) and voice-controlled devices.
3. Time Series Prediction:
   • Stock Market Prediction: LSTMs analyze historical stock prices, trading volumes, and other financial data to predict future price movements.
   • Weather Forecasting: They analyze historical weather data to forecast future weather conditions, such as temperature, precipitation, and wind patterns.
4. Gesture Recognition:
   • LSTMs can recognize gestures from video data, enabling applications like sign language recognition, human-computer interaction, and gesture-based control systems.
5. Healthcare:
   • Disease Diagnosis: LSTMs analyze electronic health records (EHRs), medical imaging data (e.g., MRI, CT scans), and genetic data to assist in disease diagnosis and prognosis.
   • Patient Monitoring: They analyze real-time patient data, such as vital signs and sensor data from wearable devices, to monitor patient health and detect abnormalities.
6. Autonomous Vehicles:
   • LSTMs process sensor data (e.g., LiDAR, radar, cameras) from autonomous vehicles to recognize objects, predict pedestrian movements, and navigate complex environments safely.
7. Robotics:
   • LSTMs enable robots to perform tasks requiring sequential decision-making, such as object manipulation, navigation, and human-robot interaction.
8. Finance:
   • Credit Risk Assessment: LSTMs analyze financial data to assess the creditworthiness of individuals and businesses, aiding in loan approval decisions.
   • Fraud Detection: They identify fraudulent activities in financial transactions by detecting patterns of suspicious behavior.

These are just a few examples, and the applications of LSTMs continue to expand as researchers and practitioners explore new ways to leverage their capabilities in solving real-world problems across diverse domains.

How LSTM and transformer are different

LSTMs (Long Short-Term Memory) networks and transformers are both architectures used for handling sequential data, but they have different underlying structures and mechanisms. Here’s how they differ:

1. Architecture:
   • LSTM: LSTM is a type of recurrent neural network (RNN) architecture. It consists of recurrent units with memory cells and gating mechanisms that control the flow of information through the network over time. LSTMs process sequences step by step, maintaining hidden states that carry information from previous time steps.
   • Transformer: The transformer architecture is based on self-attention mechanisms and does not rely on recurrent connections. It consists of multiple layers of self-attention and feedforward neural networks. Transformers process entire sequences in parallel, attending to all positions at each layer, allowing them to capture long-range dependencies more efficiently than traditional RNNs.
2. Handling Long-Term Dependencies:
   • LSTM: LSTMs are designed to capture long-term dependencies in sequential data by maintaining memory cells and gating mechanisms that control the flow of information over time. However, they can still struggle with capturing dependencies across very long sequences due to the vanishing gradient problem.
   • Transformer: Transformers are highly effective at capturing long-range dependencies in sequential data, thanks to the self-attention mechanism, which allows them to attend to all positions in the input sequence simultaneously. This makes transformers particularly well-suited for tasks that require modeling complex relationships across distant elements in the sequence.
3. Parallelization:
   • LSTM: LSTMs process sequences sequentially, one time step at a time, which limits their ability to parallelize computations across time steps. As a result, training and inference with LSTMs can be slower, especially for long sequences.
   • Transformer: Transformers process entire sequences in parallel at each layer, allowing for highly parallelized computations. This makes transformers more efficient for training and inference, especially with the availability of hardware accelerators like GPUs and TPUs.
4. Positional Information:
   • LSTM: LSTMs do not inherently encode positional information within the network architecture. Positional information may be implicitly captured through the sequence of input embeddings or explicitly incorporated as additional input features.
   • Transformer: Transformers explicitly incorporate positional encodings into the input embeddings to convey the position of each element in the sequence. This allows transformers to handle sequential data where the order of elements is important, such as natural language processing tasks.

In summary, LSTMs and transformers are different architectures for handling sequential data. LSTMs are recurrent neural networks with memory cells and gating mechanisms, suitable for capturing long-term dependencies in sequential data. Transformers, on the other hand, are based on self-attention mechanisms and process sequences in parallel, allowing them to efficiently capture long-range dependencies and handle tasks requiring complex relationships across distant elements in the sequence.

Traditional RNNs are not good at capturing long-range dependencies. This is mainly due 
to the vanishing gradient problem. When training very deep network gradients or the 
derivatives decreases exponentially as it propagates down the layers. This is known 
as Vanishing Gradient Problem. These gradients are used to update the weights of neural 
networks. When the gradients vanish then the weights will not be updated. Sometimes it 
will completely stop the neural network from training. This vanishing gradient problem
is a common issue in very deep neural networks.

To overcome this vanishing gradient problem in RNNs, LSTM is a modification to the 
RNN hidden layer. LSTM has enabled RNNs to remember its inputs over a long period of 
time. In LSTM in addition to the hidden state, a cell state is passed to the next time 
step.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import LSTM
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error

# create and fit the LSTM network
model = Sequential()
model.add(LSTM(4, input_shape=(1, look_back)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(trainX, trainY, epochs=100, batch_size=1, verbose=2)

Deep Generative Model – Autoencoder

An autoencoder is a type of artificial neural network used for unsupervised learning of efficient data representations. It works by compressing input data into a lower-dimensional representation (encoding) and then reconstructing the original data from this representation (decoding). The goal of an autoencoder is to learn a compact and meaningful representation of the input data, capturing its essential features in the process.

Here’s a detailed explanation of how autoencoders work:

1. Architecture:
   • Encoder: The encoder part of the autoencoder takes the input data and maps it to a lower-dimensional representation, also known as the latent space or encoding. It consists of one or more layers of neurons that apply nonlinear transformations to the input data, gradually reducing its dimensionality.
   • Latent Space: The latent space is a lower-dimensional representation of the input data learned by the encoder. It captures the essential features of the input data in a compressed form.
   • Decoder: The decoder part of the autoencoder takes the encoded representation from the latent space and reconstructs the original input data from it. Like the encoder, the decoder consists of one or more layers of neurons that apply transformations to the encoded representation to generate the reconstructed output.
2. Training:
   • Objective: The primary objective of training an autoencoder is to minimize the reconstruction error, i.e., the difference between the input data and its reconstructed output. Common loss functions used for this purpose include mean squared error (MSE) or binary cross-entropy, depending on the nature of the input data.
   • Backpropagation: Autoencoders are trained using backpropagation, a technique for updating the weights of the neural network to minimize the loss function. During training, the input data is fed through the encoder to obtain the encoded representation, and then the decoder reconstructs the input data from this representation. The reconstruction error is computed, and the gradients of the loss function with respect to the network parameters (weights and biases) are computed using backpropagation. The weights are then updated using an optimization algorithm like stochastic gradient descent (SGD) or Adam.

Overall, autoencoders are powerful neural network architectures that can learn compact and informative representations of input data in an unsupervised manner, making them useful for a wide range of applications in machine learning and data analysis.

https://youtu.be/3G5hWM6jqPk

https://www.ibm.com/topics/autoencoder

https://www.datacamp.com/tutorial/introduction-to-autoencoders

import keras
from keras import layers

# This is the size of our encoded representations
encoding_dim = 32  # 32 floats -> compression of factor 24.5, assuming the input is 784 floats

# This is our input image
input_img = keras.Input(shape=(784,))
# "encoded" is the encoded representation of the input
encoded = layers.Dense(encoding_dim, activation='relu')(input_img)
# "decoded" is the lossy reconstruction of the input
decoded = layers.Dense(784, activation='sigmoid')(encoded)

# This model maps an input to its reconstruction
autoencoder = keras.Model(input_img, decoded)

Generative Adversarial Networks (GANs) are a type of generative model consisting of two neural networks: a generator and a discriminator. GANs are designed to learn to generate new data samples that are similar to a given dataset. They do this by training the generator to produce realistic samples, while simultaneously training the discriminator to distinguish between real and fake samples. The competition between the generator and discriminator leads to the improvement of both networks over time.

Here’s a detailed explanation of GANs:

1. Architecture:
   • Generator: The generator takes random noise or a latent input vector as input and generates new data samples. It consists of one or more layers of neural networks that map the latent input to the output space. The goal of the generator is to produce realistic samples that are indistinguishable from real data.
   • Discriminator: The discriminator takes data samples as input and predicts whether they are real (from the true data distribution) or fake (generated by the generator). It consists of one or more layers of neural networks that output a probability score indicating the likelihood that the input sample is real.
   • Training: GANs are trained using a minimax game between the generator and discriminator. The generator tries to maximize the probability of fooling the discriminator (producing samples that are classified as real), while the discriminator tries to maximize the probability of correctly classifying real and fake samples.
2. Example:
   • Image Generation: One common application of GANs is generating realistic images. For example, GANs can be trained on a dataset of human faces and learn to generate new images of faces that look like real people. These generated images can be used for various applications in computer graphics, art generation, and face synthesis.
   • Finance and Stock Market: In finance, GANs can be used for generating synthetic financial data, such as stock prices, trading volumes, or economic indicators. For example, GANs can learn the underlying distribution of historical stock market data and generate new synthetic data samples that resemble real stock market behavior. These synthetic data samples can be used for backtesting trading strategies, simulating market scenarios, and generating training data for machine learning models.
3. Comparison with Autoencoders:
   • Purpose: GANs are primarily used for generating new data samples that resemble real data, while autoencoders are used for learning compact representations of input data.
   • Training: GANs are trained using adversarial training, where the generator and discriminator compete against each other. Autoencoders are trained using unsupervised learning, where the goal is to minimize the reconstruction error between input and output data.
   • Output: GANs generate new data samples from random noise or a latent input, while autoencoders reconstruct input data into lower-dimensional representations.

In summary, GANs are a powerful class of generative models that can learn to generate new data samples similar to a given dataset. They have various applications, including image generation, text generation, and financial data generation. GANs differ from autoencoders in terms of their purpose, training mechanism, and output.

How GAN and autoencorder are different

Generative Adversarial Networks (GANs) and autoencoders are both types of neural network architectures used for different purposes in the realm of generative modeling. Here’s a breakdown of the main differences between GANs and autoencoders:

1. Purpose:
   • GANs (Generative Adversarial Networks): GANs are primarily used for generating new data samples that are similar to a given dataset. The goal of a GAN is to learn the underlying data distribution and generate realistic samples from that distribution.
   • Autoencoders: Autoencoders, on the other hand, are used for learning compact representations of input data. They compress the input data into a lower-dimensional representation (encoding) and then reconstruct the original data from this representation (decoding).
2. Architecture:
   • GANs: GANs consist of two neural networks: a generator and a discriminator. The generator takes random noise or a latent input as input and generates new data samples, while the discriminator distinguishes between real and fake samples. The generator and discriminator are trained simultaneously in a competitive setting.
   • Autoencoders: Autoencoders typically consist of two main parts: an encoder and a decoder. The encoder compresses the input data into a lower-dimensional representation (encoding), and the decoder reconstructs the original data from this representation (decoding).
3. Training:
   • GANs: GANs are trained using adversarial training, where the generator and discriminator are trained simultaneously in a minimax game. The generator tries to generate realistic samples to fool the discriminator, while the discriminator tries to distinguish between real and fake samples.
   • Autoencoders: Autoencoders are trained using unsupervised learning, where the objective is to minimize the reconstruction error between the input and reconstructed output. The parameters of the autoencoder (encoder and decoder weights) are optimized using optimization algorithms like stochastic gradient descent.
4. Output:
   • GANs: GANs generate new data samples from random noise or a latent input. The output of a GAN is a generated sample that resembles the input data distribution.
   • Autoencoders: Autoencoders reconstruct input data into lower-dimensional representations. The output of an autoencoder is a reconstructed sample that closely matches the input data.

In summary, GANs and autoencoders serve different purposes and have distinct architectures and training mechanisms. GANs are used for generating new data samples, while autoencoders are used for learning compact representations of input data.

https://developers.google.com/machine-learning/gan/gan_structurehttps://aws.amazon.com/what-is/gan/

Generative = creating new images from the learned data

Adversarial = two models working together or against each other to improve
play the zero-sum games

Network = NN

Training  - training both generator and discriminator at the same time
both models are updated from the discriminator based on the loss function

generator - The objective of a generator is to generate plausible realistic data, 
and this data is then fed in as negative samples to a discriminator. 

discriminator -  identify the generator's fake data, it penalizes the generator for 
generating implausible data, thus forcing the generator to improve. 

First, the samples that a generator creates is of poor quality. 
The discriminator is easily able to identify fake data from the generator versus 
the real data that it receives from a training dataset. 

Second, as training progresses, the generator receives feedback from the discriminator 
and steadily improves the quality of the generated data till the generator 
is able to fool the discriminator. 

Third, the discriminator will then find it hard to tell what data is generated and 
what data is real. 

Process: the discriminator's objective is to distinguish the fake data generated by 
the generator from the real data instances that are also fed into the discriminator. 
If the discriminator is able to identify  the generator's fake data, it penalizes 
the generator for generating implausible data, thus forcing the generator to improve.

During the training process, as the generator improves, the discriminator's ability 
to distinguish real from fake steadily diminishes. During training, the generator 
will improve to such an extent till the discriminator is unable to tell fake and 
real data apart.

Convolutional neural networks, or  CNNs, 
are a neural network architecture that is primarily used for image recognition and 
processing tasks. The architecture of the layers in a CNN mimic the visual cortex of 
the brain, and how our eye and brain together perceive images. And this is why 
convolutional neural networks work very well with image data. 

The GAN that we'll build will essentially use the architecture of a convolutional neural 
network. This is going to be a deep convolutional GAN. 

You can think of the DCGAN as a class of CNNs that have certain architectural constraints,
and can learn a hierarchy of representations from input images. When you use deep 
convolutional GANs to construct the generator and discriminator network, this can greatly 
improve the quality of generated images.

# train a generative adversarial network on a one-dimensional function
from numpy import hstack
import numpy as np
from numpy import zeros
from numpy import ones
from numpy.random import rand
from numpy.random import randn
from keras.models import Sequential
from keras import Input
from keras.layers import Dense,LSTM
from matplotlib import pyplot
import matplotlib.pyplot as plt

LENGTH_INPUT = 300

# define the standalone discriminator model
def define_discriminator(n_inputs=LENGTH_INPUT):
model = Sequential()
model.add(Dense(LENGTH_INPUT, activation='relu', input_dim=n_inputs))
model.add(Dense(250, activation='relu', input_dim=n_inputs))
model.add(Dense(100, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
# compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
return model

# define the standalone generator model
def define_generator(latent_dim, n_outputs=LENGTH_INPUT):
model = Sequential()
model.add(Input(shape=(latent_dim, 1)))
model.add(LSTM(150))
model.add(Dense(LENGTH_INPUT, activation='linear'))
model.compile(loss='mean_absolute_error', optimizer='adam', metrics=['mean_absolute_error'])

return model

# define the combined generator and discriminator model, for updating the generator
def define_gan(generator, discriminator):
# make weights in the discriminator not trainable
discriminator.trainable = False
# connect them
model = Sequential()
model.add(generator)
model.add(discriminator)
model.compile(loss='binary_crossentropy', optimizer='adam')
return model

# generate n real samples with class labels
def generate_real_samples(n):
amps = np.arange(0.1,10,0.1)
bias = np.arange(0.1,10,0.1)
freqs = np.linspace(1,2,1000)
X2 = np.linspace(-5,5,LENGTH_INPUT)
X1 = []
for x in range(n):
noise = np.random.normal(size=len(X2))
X1.append(np.random.choice(amps)*np.sin(X2*np.random.choice(freqs))+np.random.choice(bias)+0.3*noise)
X1 = np.array(X1).reshape(n, LENGTH_INPUT)
# generate class labels
y = ones((n, 1))
return X1, y

# generate points in latent space as input for the generator
def generate_latent_points(latent_dim, n):
# generate points in the latent space
x_input = randn(latent_dim * n)
# reshape into a batch of inputs for the network
x_input = x_input.reshape(n, latent_dim)
return x_input

# use the generator to generate n fake examples, with class labels
def generate_fake_samples(generator, latent_dim, n):
# generate points in latent space
x_input = generate_latent_points(latent_dim, n)
# predict outputs
X = generator.predict(x_input, verbose=0)
# create class labels
y = zeros((n, 1))
#print(x_input)
return X, y


# train the generator and discriminator
def train(g_model, d_model, gan_model, latent_dim, n_epochs=10000, n_batch=128, n_eval=200):
# determine half the size of one batch, for updating the discriminator
half_batch = int(n_batch / 2)
# manually enumerate epochs
for i in range(n_epochs):
# prepare real samples
x_real, y_real = generate_real_samples(half_batch)
# prepare fake examples
x_fake, y_fake = generate_fake_samples(g_model, latent_dim, half_batch)
# update discriminator
d_model.train_on_batch(x_real, y_real)
d_model.train_on_batch(x_fake, y_fake)
# prepare points in latent space as input for the generator
x_gan = generate_latent_points(latent_dim, n_batch)
# create inverted labels for the fake samples
y_gan = ones((n_batch, 1))
# update the generator via the discriminator's error
gan_model.train_on_batch(x_gan, y_gan)
# evaluate the model every n_eval epochs
if (i+1) % n_eval == 0:
plt.title('Number of epochs = %i'%(i+1))
pred_data = generate_fake_samples(generator,latent_dim,latent_dim)[0]
real_data = generate_real_samples(latent_dim)[0]
plt.plot(pred_data[0],'.',label='Random Fake Sample',color='firebrick')
plt.plot(real_data[0],'.',label = 'Random Real Sample',color='navy')
plt.legend(fontsize=10)
plt.show()
# size of the latent space
latent_dim = 3
# create the discriminator
discriminator = define_discriminator()
# create the generator
generator = define_generator(latent_dim)
# create the gan
gan_model = define_gan(generator, discriminator)
# train model
train(generator, discriminator, gan_model, latent_dim)

Transformers

The transformer model is a type of neural network architecture introduced in the paper “Attention is All You Need” by Vaswani et al. It revolutionized the field of natural language processing (NLP) by providing a more efficient and effective alternative to recurrent neural networks (RNNs) and convolutional neural networks (CNNs) for sequence modeling tasks. The transformer architecture relies on self-attention mechanisms to capture dependencies between elements in a sequence, enabling it to model long-range dependencies more effectively than traditional sequential models.

Here’s a detailed explanation of the transformer model:

1. Architecture:
   • Encoder-Decoder Structure: The transformer architecture consists of an encoder and a decoder, each composed of multiple layers. The encoder processes the input sequence, while the decoder generates the output sequence.
   • Self-Attention Mechanism: The key innovation of the transformer model is the self-attention mechanism, which allows each element in the sequence to attend to all other elements in the sequence simultaneously. This enables the model to capture long-range dependencies and contextual information more effectively.
   • Positional Encoding: Since the transformer does not inherently understand the order of elements in a sequence like RNNs or CNNs, positional encoding is added to the input embeddings to convey positional information. This allows the model to learn representations that are sensitive to the order of elements in the sequence.
   • Feedforward Neural Networks: In addition to self-attention layers, the transformer also includes feedforward neural network layers within each encoder and decoder layer to capture complex patterns in the data.
2. Training:
   • Objective: The transformer is trained using a supervised learning framework, where the objective is to minimize a task-specific loss function such as cross-entropy loss for sequence classification or mean squared error for sequence regression.
   • Backpropagation: The parameters of the transformer (encoder and decoder weights) are optimized using backpropagation and gradient descent-based optimization algorithms like Adam.
3. Applications:
   • Machine Translation: One of the primary applications of the transformer model is in machine translation, where it has achieved state-of-the-art performance on benchmarks like the WMT translation tasks. The ability of the transformer to capture long-range dependencies and contextual information makes it well-suited for translating between languages.
   • Text Generation: The transformer can also be used for text generation tasks such as language modeling, text summarization, and dialogue generation. Its self-attention mechanism allows it to capture dependencies between words and generate coherent and contextually relevant text.
   • Finance and Stock Market: In finance, the transformer model can be applied to various tasks such as sentiment analysis of financial news, predicting stock prices, and analyzing market trends. By learning from historical financial data and textual information, transformers can generate insights and predictions that inform investment decisions and risk management strategies.
4. Benefits:
   • Parallelization: Unlike RNNs, which process sequences sequentially, the transformer can process entire sequences in parallel, making it more efficient for training and inference, especially with the availability of hardware accelerators like GPUs and TPUs.
   • Long-Range Dependencies: The self-attention mechanism allows the transformer to capture long-range dependencies in sequences more effectively than traditional sequential models like RNNs, making it suitable for tasks involving long sequences such as machine translation and text generation.

In summary, the transformer model is a powerful neural network architecture that has been successfully applied to various sequence modeling tasks, including machine translation, text generation, and finance. Its ability to capture long-range dependencies and contextual information has made it a popular choice for tasks involving sequential data.

Difference between RNN and transformer

RNN is weak in that 
Long range depedences
Grandient vanishing and explosion
large numner of training steps because you cannot parallelize


Transformers is more flexible because
allows to focus on attnetion on partuicular aspects of the input texts
can model long-rnge dependences
fewer training steps
No grandient vanishing and explosion
can arallelize


LLM is emergent meaning that an ability is emergent if it is present in larger but 
not smaller models