Codes / data / videos / readings

Introduction to texts 
https://aeturrell.github.io/coding-for-economists/text-intro.html#text-intro
https://kunststube.net/encoding/

sentiment analysis:
A simple sentiment analysis using various models and libraries
https://nbviewer.org/github/cyrus723/my-first-binder/blob/main/sentiment_analysis_1_all_models.ipynb
https://realpython.com/sentiment-analysis-python/

topic modeling
finance papers - topic modeling 
Analyst Information Discovery and Interpretation Roles: A Topic Modeling Approach | Management Science (informs.org) 
Measuring Corporate Culture Using Machine Learning | The Review of Financial Studies | Oxford Academic (oup.com) 

gensim example for topic modeling, LDA
https://nbviewer.org/github/cyrus723/my-first-binder/blob/main/nlp_lda_model_gensim1_spacy.ipynb#topic=4&lambda=1&term=

LDA from scikit site
Topic extraction with Non-negative Matrix Factorization and Latent Dirichlet Allocation — scikit-learn 1.4.2 documentation 

https://github.com/tatsath/fin-ml/blob/master/Chapter%2010%20-%20Natural%20Language%20Processing/NLP-MasterTemplate.ipynb

data
text data from scikit learn site 7.2. Real world datasets — scikit-learn 1.4.2 documentation

text classification
https://realpython.com/python-keras-text-classification/ 

word embedding
https://www.tensorflow.org/text/guide/word_embeddings 
https://en.wikipedia.org/wiki/Word_embedding 
Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space:  
http://arxiv.org/pdf/1504.06654v1 
Comparison Between BagofWords and Word2Vec — PyImageSearch:  
https://pyimagesearch.com/2022/07/18/comparison-between-bagofwords-and-word2vec/ 
https://radimrehurek.com/gensim/models/word2vec.html
The amazing power of word vectors:  
https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/ 
Understanding NLP Word Embeddings — Text Vectorization:  
https://towardsdatascience.com/understanding-nlp-word-embeddings-text-vectorization-1a23744f7223 
https://nlp.stanford.edu/projects/glove/ 
https://spacy.io/usage/models 
https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf 

Twitter Data Collection Tutorial with Python: 
https://towardsdatascience.com/twitter-data-collection-tutorial-using-python-3267d7cfa93Links to an external site. 
eLinks to an external site. 

Introduction to NLTK:
https://www.nltk.org/#:~:text=Natural%20Language%20Processing%20with%20Python,aLinks to an external site. 
nalyzing%20linguistic%20structure%2C%20and%20moreLinks to an external site. 

DataCamp Text Analytics Tutorial: 
https://www.datacamp.com/community/tutorials/text-analytics-beginners-nltkLinks to an external site. 

Natural Language Processing with Transformers Book:  
https://transformersbook.com 


tokenization
What is NLP (Natural Language Processing) Tokenization? — tokenex:  
https://www.tokenex.com/blog/ab-what-is-nlp-natural-language-processing-tokenization/ 

What is Tokenization | Tokenization In NLP:  
https://www.analyticsvidhya.com/blog/2020/05/what-is-tokenization-nlp/ 

CORPUS: 
https://hypersense.subex.com/aiglossary/corpus 

Text similarity search in Elasticsearch using vector fields | Elastic Blog:  
https://www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch 

Word, Subword, and Character-Based Tokenization: Know the Difference:  
https://towardsdatascience.com/word-subword-and-character-based-tokenization-know-the-difference-ea0976b64e17 

How to use [HuggingFace’s] Transformers Pre-Trained tokenizers?:  
ttps://nlpiation.medium.com/how-to-use-huggingfaces-transformers-pre-trained-tokenizers-e029e8d6d1fa 
https://dev.mrdbourke.com/tensorflow-deep-learning/04_transfer_learning_in_tensorflow_part_1_feature_extraction/

videos
https://youtu.be/zqVtHYFYQY8
https://youtu.be/zqVtHYFYQY8?t=1138  

  

Introduction

Everything in NLP is discrete, meaning there is no predictable relationship between the forms and the meanings. On the other hand, neural networks are best at dealing with something numerical and continuous, meaning everything in neural networks needs to be float numbers. How can we “bridge” between these two worlds—discrete and continuous? The key is the use
of word embeddings.  Technically, an embedding is a continuous vector representation of something that is usually discrete. A word embedding is a continuous vector representation of a word.

(from Hagiwara, 2021)

A neural network (also called an artificial neural network) is a generic mathematical model that transforms a vector to another vector. Neural networks are trainable. We usually group a number of instances together and feed them to a neural network, updating model parameters per each group, not per each instance. We call this group of instances a batch.

(from Hagiwara, 2021)

Pre-prosessing raw texts

Natural language process is a set of methods which map natural language units into a machine readable form. Once we can figure out how to represent language to a machine, we can then use statistical/mathematical tools to learn things about tests without having to read them by human.  (SPAM filters, speech recognition, machine translation, information retrieval like Research engines, and artificial intelligence.)

tokenization – splits the document into tokens which can be words or n-grams or phrases, The purpose of tokenization is to create a vocabulary from a corpus. A corpus is a collection of texts (such as the dataset used to train an NLP model), and a vocabulary is the set of unique tokens found within the corpus.

removing formatting – punctuation, numbers, cases, spacing

stop word removal – removal of “stop words” like “if,” “but,” “or,” and so on

Normalizing words by condensing all forms of a word into a single form. Stemming and lemmatization are techniques used to reduce words to their root forms.

Stemming involves removing the suffixes from words, such as “ing” or “ed,” to reduce them to their base form. For example, the word “jumping” would be stemmed to “jump.” With stemming, a word is cut off at its stem, the smallest unit of that word from which you can create the descendant words. Stemming simply truncates the string using common endings, so it will miss the relationship between “feel” and “felt,” for example.

Lemmatization, however, involves reducing words to their base form based on their part of speech. For example, the word “jumped” would be lemmatized to “jump,” but the word “jumping” would be lemmatized to “jumping” since it is a present participle. Lemmatization process uses a data structure that relates all forms of a word back to its simplest form, or lemma. Because lemmatization is generally more powerful than stemming, it’s the only normalization strategy offered by spaCy.

Sources:  from https://www.datacamp.com/tutorial/text-analytics-beginners-nltk
Stemming and Lemmatization in Python

Feature representation or vectorizing

Next feature representation – the preprocessed tokens need to be translated into predictive features, which means that a researcher needs to transform tokens to a digestible format by computer. Machine learning models take vectors (arrays of numbers) as input. When working with text, the first thing you must do is come up with a strategy to convert strings to numbers (or to “vectorize” the text) before feeding it to the model. We create vectors from tokens.

Representing text as numbers 

These vectors are numerical representations of the meaning of a token and how it relates to other tokens in our vocabulary. A model is used to embed raw text into a vector space where we can use the data science tool. Vectorizing text by turning the text into a numerical representation for consumption by your classifier, Vectorization is a process that transforms a token into a vector, or a numeric array that, in the context of NLP, is unique to and represents various features of a token. Vectors are used under the hood to find word similarities, classify text, and perform other NLP operations. This particular representation is a dense array, one in which there are defined values for every space in the array. This is in opposition to earlier methods that used sparse arrays, in which most spaces are empty.

One-hot encodings : As a first idea, you might “one-hot” encode each word in your vocabulary. Consider the sentence “The cat sat on the mat”. The vocabulary (or unique words) in this sentence is (cat, mat, on, sat, the). To represent each word, you will create a zero vector with length equal to the vocabulary, then place a one in the index that corresponds to the word. This approach is shown in the following diagram.

Encode each word with a unique number:  A second approach you might try is to encode each word using a unique number. Continuing the example above, you could assign 1 to “cat”, 2 to “mat”, and so on. You could then encode the sentence “The cat sat on the mat” as a dense vector like [5, 1, 4, 3, 5, 2]. This approach is efficient. Instead of a sparse vector, you now have a dense one (where all elements are full).

N-Grams is a sequence of n items in a sample of text. Depending on N, it’s called bigrams, trigrams, four-grams, and et cetera. 

For example, “This is a sentence”. 

Source: https://www.educative.io/answers/what-are-n-grams

Count vectorization: Creates a document-term matrix where the entry of each cell will be a count of the number of times that word occurred in that document.

TF-IDF is a technique to convert text to a numeric table representation. 

• In this table, each row represents a document in the corpus. 
• Each column represents a word in the corpus. 
• Each cell in the table provides a value that indicates the relative strength of the word with respect to the document. 
• A higher value indicates higher correlation between the word and the document. 
• The rarer the word is, the higher that this value’s going to be. 
• If a word occurs very frequently within a particular text message, so that’s TF, but very infrequently elsewhere, that’s going to be the second term. 
• Then a very large number will be assigned, and it’ll be assumed to be very important to differentiating that text message from others. 
• In summary, this method helps you pull out important but seldom-used words.

Example – vectorizing

sentences = ['John likes ice cream', 'John hates chocolate.']

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(min_df=0, lowercase=False)
vectorizer.fit(sentences)
vectorizer.vocabulary_

output:
{'John': 0, 'chocolate': 1, 'cream': 2, 'hates': 3, 'ice': 4, 'likes': 5}
 vectorizer.transform(sentences).toarray() output: array([[1, 0, 1, 0, 1, 1],
    [1, 1, 0, 1, 0, 0]])

Count vectorization: Creates a document-term matrix where the entry of each cell will be a count of the number of times that word occurred in that document.

from sklearn.feature_extraction.text import CountVectorizer

count_vect = CountVectorizer(analyzer=clean_text)

X_counts = count_vect.fit_transform(data['body_text'])

N-gram

from sklearn.feature_extraction.text import CountVectorizer

ngram_vect = CountVectorizer(ngram_range=(2,2)) #bi-gram

X_counts = ngram_vect.fit_transform(data['cleaned_text'])

TF-IDF

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vect = TfidfVectorizer(analyzer=clean_text)

X_tfidf = tfidf_vect.fit_transform(data['body_text'])

Word embeddings give us a way to use an efficient, dense representation in which similar words have a similar encoding. Importantly, you do not have to specify this encoding by hand. An embedding is a dense vector of floating point values (the length of the vector is a parameter you specify). Instead of specifying the values for the embedding manually, they are trainable parameters (weights learned by the model during training, in the same way a model learns weights for a dense layer). It is common to see word embeddings that are 8-dimensional (for small datasets), up to 1024-dimensions when working with large datasets. A higher dimensional embedding can capture fine-grained relationships between words, but takes more data to learn.

In natural language processing, “embedding” refers to the process of mapping non-vectorized data, such as tokens, into a vector space that has meaning for a machine learning model or neural network. By doing this, the model can learn the relationships between words and their meanings automatically rather than having to specify them manually. When you visualize word embeddings, you can think of the result as a map of the vocabulary that shows how one token is related to other tokens in terms of meaning. Each token is positioned near other tokens with similar meanings.

To give tokens meaning, the model must be trained on them. This allows the model to learn the meanings of words and how they relate to other words. To achieve this, the word vectors are “embedded” into an embedding space. As a result, similar words should have similar vectors after training.

Word2Vec essentially means expressing each word in your text corpus in an N-dimensional space (embedding space). The word’s weight in each dimension of that embedding space defines it for the model.

“Bag of words”

most text analysis methods treat documents as a big bunch of words or terms.​ Order is generally not taken into account, just word and term frequencies.​ There are ways to parse documents into ngrams or words

Drawbacks of BOW: the inclusion of many 0s in our matrix (i.e., a sparse matrix). A sparse matrix contains less information and wastes a lot of memory.

The biggest disadvantage in Bag-of-Words is the complete inability to learn grammar and semantics.

• Lack of meaningful relations and no consideration for order of words: Usually, a text is mainly represented by a BOW via a one-hot encoding strategy.
  • Many latent and deep semantics among words and particularly the order or the contextual information of words are usually ignored.
• Sparse matrix: In the BOW model, each document is represented as a word-count vector. These counts can be binary counts, a word may occur in the text or not or will have absolute counts. 
  • The size of the vector is equal to the number of elements in the vocabulary. If most of the elements are zero then the bag of words will be a sparse matrix. 
• Computationally intensive: In deep learning, sparse representations are harder to model for computational reasons.

• Huge amount of weights: Huge input vectors mean a huge number of weights for a neural network. More weights mean more computation required to train and predict.

• “Predicting word meaning based on its neighboring words (collocation), is generally one of the most common extensions beyond the simple bag of words approach.” (p. 27, LM, 2016)

Word Embedding is solution to these problems.

Sparse Matrix problem with BOW is solved by mapping high-dimensional data into a lower-dimensional space  that preserves semantic relationships.

Lack of meaningful relationship issue of BOW is solved by placing vectors of semantically similar items close to each other. This way words that have similar meaning have similar distances in the vector space.

Source: https://www.tensorflow.org/text/guide/word_embeddings

Source: Chakraborty, D. “Word2Vec: A Study of Embeddings in NLP,” PyImageSearch, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, R. Raha, and A. Thanki, eds., 2022, https://pyimg.co/2fb0z

Source: https://pyimagesearch.com/2022/07/11/word2vec-a-study-of-embeddings-in-nlp/

One-hot ending

As a first idea, you might "one-hot" encode each word in your vocabulary. Consider the 
sentence "The cat sat on the mat". The vocabulary (or unique words) in this sentence 
is (cat, mat, on, sat, the). To represent each word, you will create a zero vector 
with length equal to the vocabulary, then place a one in the index that corresponds
to the word. 




from https://www.tensorflow.org/text/guide/word_embeddings

Word embedding

Word embeddings give us a way to use an efficient, dense representation in which similar
 words have a similar encoding. Importantly, you do not have to specify this encoding by 
hand. An embedding is a dense vector of floating point values (the length of the vector 
is a parameter you specify). Instead of specifying the values for the embedding manually,
 they are trainable parameters (weights learned by the model during training, in the 
same way a model learns weights for a dense layer). It is common to see word embeddings 
that are 8-dimensional (for small datasets), up to 1024-dimensions when working with 
large datasets. A higher dimensional embedding can capture fine-grained relationships 
between words, but takes more data to learn.






from https://www.tensorflow.org/text/guide/word_embeddings

https://www.tensorflow.org/text/tutorials/word_embeddings
https://www.tensorflow.org/text/tutorials/warmstart_embedding_matrix
https://www.tensorflow.org/text/tutorials/word2vec

The Difference Between a Token, a Vector, and an Embedding

To get to a point where your model can understand text, you first have to tokenize it, vectorize it and create embeddings from these vectors.

• Tokenization: This is the process of dividing the original text into individual pieces called tokens. Each token is assigned a unique id to represent it as a number.
• Vectorization: The unique ids are then assigned to randomly initialized n-dimensional vectors.
• Embedding: To give tokens meaning, the model must be trained on them. This allows the model to learn the meanings of words and how they relate to other words. To achieve this, the word vectors are “embedded” into an embedding space. As a result, similar words should have similar vectors after training.

Source: https://medium.com/@saschametzger/what-are-tokens-vectors-and-embeddings-how-do-you-create-them-e2a3e698e037

Word embeddings

Help in the following use cases.

• Compute similar words
• Text classifications
• Document clustering/grouping
• Feature extraction for text classifications
• Natural language processing.

“WORD VECTORS are numerical vector representations of word semantics, or meaning, including literal and implied meaning. So word vectors can capture the connotation of words, like “peopleness,” “animalness,” “placeness,” “thingness,” and even “conceptness.” And they combine all that into a dense vector (no zeros) of floating point values. This dense vector
enables queries and logical reasoning.

Word2vec learns the meaning of words merely by processing a large corpus of unlabeled text. No one has to label the words in the Word2vec vocabulary. No one has to tell
the Word2vec algorithm that Marie Curie is a scientist, that the Timbers are a soccer
team, that Seattle is a city, or that Portland is a city in both Oregon and Maine. And no
one has to tell Word2vec that soccer is a sport, or that a team is a group of people, or
that cities are both places as well as communities. Word2vec can learn that and much
more, all on its own! All you need is a corpus large enough to mention Marie Curie and
Timbers and Portland near other words associated with science or soccer or cities.”

“Instead of trying to train a neural network to learn the target word meanings directly
(on the basis of labels for that meaning), you teach the network to predict words near
the target word in your sentences. So in this sense, you do have labels: the nearby
words you’re trying to predict. But because the labels are coming from the dataset
itself and require no hand-labeling”

The process of converting words into numbers are called Vectorization. Word Embeddings is one of vectorization and is a methodology in NLP to map words or phrases from vocabulary to a corresponding vector of real numbers which used to find word predictions, word similarities/semantics. After the words are converted as vectors, we need to use some techniques such as  Euclidean distance, Cosine Similarity to identify similar words.

Why Cosine Similarity Count the common words or Euclidean distance is the general approach used to match similar documents which are based on counting the number of common words between the documents. This approach will not work even if the number of common words increases but the document talks about different topics. To overcome this flaw, the “Cosine Similarity” approach is used to find the similarity between the documents. Mathematically, it measures the cosine of the angle between two vectors (item1, item2)
projected in an N-dimensional vector space. The advantageous of cosine similarity is,
it predicts the document similarity even Euclidean is distance.

Word embeddings coming from pre-trained methods such as,

Word2Vec — From Google
Fasttext — From Facebook
Glove — From Standford

To illustrate how word embeddings can be analyzed using Euclidean distance, we’ll use the same hypothetical word vectors I described earlier and compute the distances between a few pairs of words. This will show how similar or dissimilar words are to each other in the embedding space.

Here are the word vectors again:

• apples: [0.9, 0.1, 0.8]
• oranges: [0.9, 0.2, 0.8]
• eating: [0.3, 0.8, 0.2]
• fruits: [0.7, 0.3, 0.9]
• enjoy: [0.4, 0.9, 0.1]
• like: [0.5, 0.9, 0.1]

To compute the Euclidean distance between two vectors, you use the formula:

Interpretation

• The distance between “apples” and “oranges” is small (0.1), indicating high similarity, which makes sense as both are fruits.
• The distance between “enjoy” and “like” is also small (0.1), showing that these verbs are considered similar in meaning.
• The distance between “apples” and “eating” is larger (1.1), reflecting their different roles and meanings in language (one being a noun/fruit and the other a verb/action).

These distances provide a numeric measure of the “semantic distance” between words, according to the learned embeddings. Lower distances suggest closer semantic relationships. This method can be a powerful tool for many natural language processing applications.

Sources: https://arxiv.org/pdf/1301.3781.pdf
https://www.machinelearningplus.com/nlp/cosine-similarity/ 
https://medium.com/@adriensieg/text-similarities-da019229c894

Example from Chat GPT: Show an example of document term matrix and cosine similarity using the real-world example

Step 1: Define the Documents

Let’s consider three short text documents as our corpus:

1. Document 1: “Apple and orange are popular fruits.”
2. Document 2: “Microsoft and Apple are leading tech companies.”
3. Document 3: “I like to eat apple and orange.”

Step 2: Create the Document-Term Matrix

A Document-Term Matrix represents documents as rows and terms (or words) as columns. Each cell in the matrix represents the frequency of a term in a document. For simplicity, we will ignore common stop words like “and”, “are”, “to”, etc., and we won’t apply stemming or lemmatization:

This result shows a relatively low similarity between Document 1 and Document 2, which is expected given their different contexts (fruits vs tech companies).

Using a similar approach, you can calculate cosine similarity between other pairs of documents to determine how closely related they are in terms of their content. This type of analysis is fundamental in systems like search engines, recommendation systems, and more sophisticated natural language processing tasks.

Lexicon-based analysis

This type of analysis, such as the NLTK Vader sentiment analyzer, involves using a set of predefined rules and heuristics to determine the sentiment of a piece of text. These rules are typically based on lexical and syntactic features of the text, such as the presence of positive or negative words and phrases.

While lexicon-based analysis can be relatively simple to implement and interpret, it may not be as accurate as ML-based or transformed-based approaches, especially when dealing with complex or ambiguous text data.

Machine learning (ML)

This approach involves training a model to identify the sentiment of a piece of text based on a set of labeled training data. These models can be trained using a wide range of ML algorithms, including decision trees, support vector machines (SVMs), and neural networks.

ML-based approaches can be more accurate than rule-based analysis, especially when dealing with complex text data, but they require a larger amount of labeled training data and may be more computationally expensive.

Pre-trained transformer-based deep learning

A deep learning-based approach, as seen with BERT and GPT-4, involve using pre-trained models trained on massive amounts of text data. These models use complex neural networks to encode the context and meaning of the text, allowing them to achieve state-of-the-art accuracy on a wide range of NLP tasks, including sentiment analysis. However, these models require significant computational resources and may not be practical for all use cases.

• Lexicon-based analysis is a straightforward approach to sentiment analysis, but it may not be as accurate as more complex methods.
• Machine learning-based approaches can be more accurate, but they require labeled training data and may be more computationally expensive.
• Pre-trained transformer-based deep learning approaches can achieve state-of-the-art accuracy but require significant computational resources and may not be practical for all use cases.

The choice of approach will depend on the specific needs and constraints of the project at hand.

Source:  https://www.datacamp.com/tutorial/text-analytics-beginners-nltk

Word2vec

Word2Vec is a popular and influential model for producing word embeddings introduced by a team of researchers at Google led by Tomas Mikolov in 2013. Word embeddings are dense vector representations of words that capture semantic meanings based on their usage in context. Word2Vec represents a breakthrough in the way computers interpret the semantic relationships between words by mapping these words into a high-dimensional space where the distance and direction between vectors help to define their relationships.

How Word2Vec Works

Word2Vec offers two architectures for generating word embedding: Continuous Bag-of-Words (CBOW) and Skip-Gram:

1. Continuous Bag-of-Words (CBOW): This model predicts a target word from a set of context words surrounding it. For example, given the context words “the cat on the”, the model predicts the missing word “mat”. CBOW is faster and has better representations for more frequent words.
2. Skip-Gram: This model works in the opposite way of CBOW; it uses a target word to predict context words. For example, given the word “apple”, it might predict context words like “fruit”, “red”, or “eat”. Skip-Gram works well with small datasets and represents well even rare words or phrases.

Both models are trained using a technique known as negative sampling, a simplified form of Noise Contrastive Estimation (NCE) that improves both the speed and quality of learning by randomly sampling negative examples to modify the weights rather than using the entire vocabulary, which can be computationally expensive.

Applications of Word2Vec

Word2Vec’s ability to capture semantic relationships and syntactic patterns within a corpus of text has led to its widespread application in many areas of natural language processing (NLP) and beyond:

1. Semantic Analysis: Word2Vec embeddings can be used to assess the similarity between words or phrases, aiding in tasks like sentiment analysis, where the context of words around terms expressing sentiment is crucial.
2. Language Translation: By mapping words from multiple languages into the same space based on their meanings, Word2Vec can assist in creating baselines for machine translation systems.
3. Recommendation Systems: By understanding the relationships and similarities between different items (not just limited to words but also including any items that can be semantically represented), Word2Vec can improve the accuracy of recommendations in content-based filtering systems.
4. Information Retrieval: Search engines can utilize Word2Vec to enhance the relevance of search results by understanding the semantic similarities between the terms used in a search query and the content available in the search index.
5. Chatbots and Virtual Assistants: Word2Vec can help in understanding user queries by converting them into vectors and finding semantically relevant answers or actions.
6. Enhanced Feature Generation: For more traditional machine learning tasks, Word2Vec can be used to generate more meaningful features from text data, which can then be used to train more effective models.

The introduction of Word2Vec has significantly influenced the development of more complex models like BERT, GPT (Generative Pre-trained Transformer), and others, each building upon or complementing the foundational ideas introduced by Word2Vec. As such, it remains a cornerstone technique in NLP, demonstrating the power of neural networks in understanding and processing language.

Limitations of word embeddings

Word embeddings are a powerful concept that can give your application a boost in the
performance, although they are not without limitation. One obvious issue is that they
cannot take context into account. Words you see in natural language are often polysemous, meaning they may have more than one meaning, depending on their context. However, because word embeddings are trained per token type, all the different
meanings are compressed into a single vector. For example, training a single vector for “dog” or “apple” cannot deal with the fact that “hot dog” or “Big Apple” are not a
type of animal or fruit, respectively. As another example, consider what “play” means
in these sentences: “They played games,” “I play Chopin,” “We play baseball,” and
“Hamlet is a play by Shakespeare” (these sentences are all from Tatoeba.org). These
occurrences of “play” have different meanings, and assigning a single vector wouldn’t
help much in downstream NLP tasks (e.g., in classifying the topic into sports, music,
and art).  (from Hagiwara, 2021, ch9)

Is word2vec vector representation for each word?

Yes, exactly! Word2Vec creates a dense vector representation for each word in a given vocabulary. These vectors are of fixed dimensionality (e.g., 100, 200, 300 dimensions), and they capture the semantic meaning of words in a continuous vector space.

The key idea behind Word2Vec is that similar words have similar vector representations, meaning they are closer together in the vector space. For example, in a well-trained Word2Vec model, the vectors for “king” and “queen” would be closer together compared to the vectors for “king” and “dog” because “king” and “queen” are semantically related in terms of royalty, whereas “king” and “dog” are not.

This dense vector representation allows Word2Vec to capture various linguistic patterns and relationships between words, making it useful for a wide range of natural language processing tasks such as text classification, sentiment analysis, machine translation, and more.

Show me a vector representation of word2vec for the words, king, queen, dog, and cat.

Sure, I can’t provide the exact vectors as it would depend on the specific Word2Vec model used for training, including the dimensionality of the vectors and the specific embeddings learned during training. However, I can give you an example of what the vector representations might look like conceptually.

Let’s assume we’re working with a simplified Word2Vec model where each word is represented by a 3-dimensional vector. Here’s an example of what the vector representations might look like:

– “king”: [0.2, 0.5, 0.8]
– “queen”: [0.1, 0.6, 0.7]
– “dog”: [0.9, 0.3, 0.2]
– “cat”: [0.8, 0.4, 0.1]

In this hypothetical example, each word is represented by a 3-dimensional vector, where each dimension captures some aspect of the word’s meaning in the vector space. The values in each vector are arbitrary and chosen for illustrative purposes.

Keep in mind that in real Word2Vec models, the vectors would typically have much higher dimensionality (e.g., 100, 200, 300 dimensions), and the values would be learned during the training process based on the context in which the words appear in the training corpus.

Getting Started with KerasNLP 
https://keras.io/guides/keras_nlp/getting_started/

Text generation with an RNN
https://www.tensorflow.org/text/tutorials/text_generation


https://www.tensorflow.org/text/tutorials/nmt_with_attention

https://www.tensorflow.org/text/tutorials/transformer


text classification
https://www.tensorflow.org/text/tutorials/classify_text_with_bert
https://www.tensorflow.org/text/tutorials/text_classification_rnn
https://www.tensorflow.org/text/tutorials/text_similarity


NLP with BERT
https://www.tensorflow.org/text/tutorials/bert_glue
https://www.tensorflow.org/tfmodels/nlp/fine_tune_bert
https://www.tensorflow.org/text/tutorials/uncertainty_quantification_with_sngp_bert

Example from NLTK https://www.nltk.org/#:~:text=Natural%20Language%20Processing%20with%20Python,a%20nalyzing%20linguistic%20structure%2C%20and%20more

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens

['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

>>> tagged = nltk.pos_tag(tokens)
>>> tagged[0:6]

[('At', 'IN'), ('eight', 'CD'), ("o'clock", 'JJ'), ('on', 'IN'),
('Thursday', 'NNP'), ('morning', 'NN')]

Identify named entities

>>> entities = nltk.chunk.ne_chunk(tagged)
>>> entities

Tree('S', [('At', 'IN'), ('eight', 'CD'), ("o'clock", 'JJ'),
           ('on', 'IN'), ('Thursday', 'NNP'), ('morning', 'NN'),
       Tree('PERSON', [('Arthur', 'NNP')]),
           ('did', 'VBD'), ("n't", 'RB'), ('feel', 'VB'),
           ('very', 'RB'), ('good', 'JJ'), ('.', '.')])

Example

# to tokenize
text.split()

# to create counter
import collections
collections.Counter()

collections.Counter().most_common(10)

# to remove special characters
import re
re.sub(r'[^\w]',' ', text)

text.lower()

import nltk 
nltk.download('stopwords')
from nltk.corpus import stopwords
stop_words = stopwords.words('english')
for word in words:
    if word not in stop_words:
       words_no_stop.append(word)

from nltk.stem import PorterStemmer
for word in words_no_stop:
    words_clean.append(PorterStemer().stem(word))

from sklearn.feature_extraction.text import TfidfVectorizer
# Build a vocabulary from our training text and transform training text
training_dtm_tf = TfidfVectorizer(stop_words='english').fit_transform(training_text)

from https://spacy.io/usage/spacy-101

import spacy
text = """
Dave watched as the forest burned up on the hill,
only a few miles from his house. The car had
been hastily packed and Marta was inside trying to round
up the last of the pets. "Where could she be?" he wondered
as he continued to wait for Marta to appear with the pets.
"""

nlp = spacy.load("en_core_web_sm")
doc = nlp(text)
token_list = [token for token in doc]
token_list

[
, Dave, watched, as, the, forest, burned, up, on, the, hill, ,,
, only, a, few, miles, from, his, house, ., The, car, had,
, been, hastily, packed, and, Marta, was, inside, trying, to, round,
, up, the, last, of, the, pets, ., ", Where, could, she, be, ?, ", he, wondered,
, as, he, continued, to, wait, for, Marta, to, appear, with, the, pets, .,
]

filtered_tokens = [token for token in doc if not token.is_stop]
filtered_tokens

[
, Dave, watched, forest, burned, hill, ,,
, miles, house, ., car,
, hastily, packed, Marta, inside, trying, round,
, pets, ., ", ?, ", wondered,
, continued, wait, Marta, appear, pets, .,
]

lemmas = [
f"Token: {token}, lemma: {token.lemma_}"
for token in filtered_tokens
]
lemmas

['Token: \n, lemma: \n', 'Token: Dave, lemma: Dave',
 'Token: watched, lemma: watch', 'Token: forest, lemma: forest',
 # ...
]

filtered_tokens[1].vector

array([ 1.8371646 ,  1.4529226 , -1.6147211 ,  0.678362  , -0.6594443 ,
        1.6417935 ,  0.5796405 ,  2.3021278 , -0.13260496,  0.5750932 ,
        1.5654886 , -0.6938864 , -0.59607106, -1.5377437 ,  1.9425622 ,
       -2.4552505 ,  1.2321601 ,  1.0434952 , -1.5102385 , -0.5787632 ,
        0.12055647,  3.6501784 ,  2.6160972 , -0.5710199 , -1.5221789 ,
        0.00629176,  0.22760668, -1.922073  , -1.6252862 , -4.226225  ,
       -3.495663  , -3.312053  ,  0.81387717, -0.00677544, -0.11603224,
        1.4620426 ,  3.0751472 ,  0.35958546, -0.22527039, -2.743926  ,
        1.269633  ,  4.606786  ,  0.34034157, -2.1272311 ,  1.2619178 ,
       -4.209798  ,  5.452852  ,  1.6940253 , -2.5972986 ,  0.95049495,
       -1.910578  , -2.374927  , -1.4227567 , -2.2528825 , -1.799806  ,
        1.607501  ,  2.9914255 ,  2.8065152 , -1.2510269 , -0.54964066,
       -0.49980402, -1.3882618 , -0.470479  , -2.9670253 ,  1.7884955 ,
        4.5282774 , -1.2602427 , -0.14885521,  1.0419178 , -0.08892632,
       -1.138275  ,  2.242618  ,  1.5077229 , -1.5030195 ,  2.528098  ,
       -1.6761329 ,  0.16694719,  2.123961  ,  0.02546412,  0.38754445,
        0.8911977 , -0.07678384, -2.0690763 , -1.1211847 ,  1.4821006 ,
        1.1989193 ,  2.1933236 ,  0.5296372 ,  3.0646474 , -1.7223308 ,
       -1.3634219 , -0.47471118, -1.7648507 ,  3.565178  , -2.394205  ,
       -1.3800384 ], dtype=float32)

https://www.geeksforgeeks.org/nlp-gensim-tutorial-complete-guide-for-beginners/

https://www.machinelearningplus.com/nlp/gensim-tutorial/

https://radimrehurek.com/gensim/auto_examples/index.html

https://radimrehurek.com/gensim/models/word2vec.html

https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/
https://www.machinelearningplus.com/nlp/gensim-tutorial/ https://topic-modeling.pythonhumanities.com/intro.html

Topic extraction with Non-negative Matrix Factorization and Latent Dirichlet Allocation — scikit-learn 1.4.2 documentation

https://repub.eur.nl/pub/116043/annals.2017.0099.pdf

Measuring Corporate Culture Using Machine Learning | The Review of Financial Studies | Oxford Academic (oup.com)

https://repub.eur.nl/pub/116043/annals.2017.0099.pdf
Analyst Information Discovery and Interpretation Roles: A Topic Modeling Approach | Management Science (informs.org)

Measuring Corporate Culture Using Machine Learning | The Review of Financial Studies | Oxford Academic (oup.com)

https://nbviewer.org/github/cyrus723/my-first-binder/blob/main/LDA_Kochmar2022_NLP_book_CH10.ipynb

https://github.com/PacktPublishing/Machine-Learning-for-Algorithmic-Trading-Second-Edition_Original/blob/master/16_word_embeddings/05_financial_news_word2vec_gensim.ipynb

https://nbviewer.org/github/cyrus723/my-first-binder/blob/main/embedding_gensim_sentiment1.ipynb

https://nbviewer.org/github/cyrus723/my-first-binder/blob/main/nlp_lda_model_gensim1_spacy.ipynb#topic=4&lambda=1&term=


from gensim.utils import tokenize 
from gensim.parsing.preprocessing import preprocess_string,strip_tags,strip_punctuation,strip_numeric,remove_stopwords,strip_short 
from gensim.corpora.dictionary import Dictionary 
from gensim import models 
from gensim.models.ldamodel import LdaModel 

lda_bow = LdaModel(dataset_corpus_bow,num_topics=20,id2word=dataset_dictionary,random_state=0) 

output: 
(0, '0.013*"space" + 0.006*"nasa" + 0.006*"earth" + 0.006*"writes" + 0.005*"edu" + 0.005*"shuttle" + 0.004*"gamma" + 0.004*"article"') 
(1, '0.014*"edu" + 0.008*"university" + 0.008*"mail" + 0.007*"information" + 0.006*"list" + 0.005*"com" + 0.005*"address" + 0.005*"research"') 
(2, '0.014*"writes" + 0.012*"article" + 0.010*"like" + 0.007*"car" + 0.007*"edu" + 0.007*"people" + 0.006*"think" + 0.006*"know"') 
(3, '0.007*"health" + 0.006*"edu" + 0.005*"writes" + 0.005*"insurance" + 0.005*"private" + 0.004*"pictures" + 0.004*"canada" + 0.003*"care"')

The size of the vector is fixed no matter how long (or how short) the source sentence is. The intermediate vector is a huge bottleneck.  (from Hagiwara, 2021, Ch 8)

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
tokenizer_output = tokenizer.tokenize("This is an example of the bert tokenizer")
print(tokenizer_output)

output:
# ['this', 'is', 'an', 'example', 'of', 'the', 'bert', 'token', '##izer']

pip install transformers torch

from transformers import pipeline

def sentiment_analysis(text):
# Load the sentiment analysis pipeline with the BERT model
classifier = pipeline("sentiment-analysis", model="nlptown/bert-base-multilingual-uncased-sentiment")

# Perform sentiment analysis
result = classifier(text)

return result

# Example usage:
financial_news = "The stock market experienced a significant downturn today amid concerns over inflation."
result = sentiment_analysis(financial_news)
print("Sentiment Analysis Result:", result)



output:
Sentiment Analysis Result: [{'label': '2 stars', 'score': 0.47418859601020813}]

https://research.google/blog/open-sourcing-bert-state-of-the-art-pre-training-for-natural-language-processing/

https://huggingface.co/docs/transformers/en/model_doc/bert


https://www.youtube.com/watch?v=t45S_MwAcOw


https://www.geeksforgeeks.org/explanation-of-bert-model-nlp/

https://www.geeksforgeeks.org/explanation-of-bert-model-nlp/


from transformers import BertTokenizer

# Load pre-trained BERT tokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

# Input text
text = 'ChatGPT is a language model developed by OpenAI, based on the GPT (Generative Pre-trained Transformer) architecture. '

# Tokenize and encode the text
encoding = tokenizer.encode(text)

# Print the token IDs
print("Token IDs:", encoding)

# Convert token IDs back to tokens
tokens = tokenizer.convert_ids_to_tokens(encoding)

# Print the corresponding tokens
print("Tokens:", tokens)

https://medium.com/prosus-ai-tech-blog/finbert-financial-sentiment-analysis-with-bert-b277a3607101

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3910214

pip install transformers pandas

from transformers import pipeline

# Initialize the sentiment analysis model
model_name = "yiyanghkust/finbert-tone"
finbert = pipeline("sentiment-analysis", model=model_name)

# Function to analyze sentiment
def analyze_sentiment(text):
    results = finbert(text)
    return results

# Example usage:
financial_news = "The company's revenue has exceeded expectations, but concerns about regulatory challenges remain."
sentiment_result = analyze_sentiment(financial_news)

# Print the results
print("Sentiment Analysis Result:", sentiment_result)


output:
Sentiment Analysis Result: [{'label': 'Positive', 'score': 0.9999998807907104}]

Repurposing and Fine-Tuning

https://www.turing.com/resources/finetuning-large-language-models

Papers

https://www.sciencedirect.com/science/article/pii/S266734522300024X

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4402499

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4391863

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4316650

https://mgmt.wharton.upenn.edu/profile/emollick/

https://arxiv.org/abs/2305.18339

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4429658


https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4322651

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4660148

https://arxiv.org/pdf/2403.14380

https://bfi.uchicago.edu/wp-content/uploads/2023/04/BFI_WP_2023-59.pdf

https://rpc.cfainstitute.org/-/media/documents/article/industry-research/unstructured-data-and-ai.pdf


https://rpc.cfainstitute.org/-/media/documents/article/rf-brief/ai-and-big-data-in-investments.pdf

BloombergGPT

BloombergGPT is a specialized version of a large language model tailored specifically for financial data and applications. This model leverages Bloomberg’s vast repository of proprietary financial data, including financial reports, market analyses, news articles, and other relevant financial documentation. Here are some key points about BloombergGPT:

1. Training and Data: BloombergGPT is trained on a mix of 363 billion tokens from Bloomberg’s proprietary datasets and 345 billion tokens from general-purpose datasets. This extensive training helps the model to develop a deep understanding of financial contexts and terminology.
2. Purpose and Use: The primary goal of BloombergGPT is to provide finance-specific insights and analytics, making it an invaluable tool for financial analysts, investors, and other market participants. It’s designed to integrate and analyze complex financial data sets, extracting meaningful insights that can drive investment decisions.
3. Customization and Performance: Unlike generic language models that are trained on broad data from various domains, BloombergGPT’s training on specialized financial data allows it to perform exceptionally well in tasks that require a deep understanding of the financial sector. This customization makes it particularly useful for analyzing financial texts, generating reports, and even predicting market trends based on historical data.
4. Technological Advancement: By using a language model specifically tuned for financial applications, Bloomberg leverages its industry expertise and unique data assets to create a tool that can significantly enhance productivity and decision-making in finance.
5. Integration: BloombergGPT is not just a standalone tool but is designed to be integrated into Bloomberg’s existing suite of financial analysis tools and platforms. This integration ensures that users can seamlessly utilize the model within their regular workflows.

In summary, BloombergGPT represents a significant advancement in applying AI and language models within the financial industry, providing tailored solutions that enhance understanding and decision-making from complex datasets.