Comparing Text using Word Vectors

 

• Word vectors try to represent meaning of words
• Words which appear in similar context have similar vectors

This will require training word vectors based on a huge corpus of data.

We can use pretrained word vectors using the glove algorithm, Spacy library in python makes it easy to work with.

word vectors tpically have a lenth of few hundreds elements dimensions for word vectors

first pass the string to the nlp object

Using token we iterate over and try to print first few dimensions of the vector

actual values dont matter; what matters is how similar they are to other words

Similarity

• Direction of vectors matters
• “Distance” between words = angle between the vectors
• Cosine similarity
  • 1: If vectors point in the same direction
  • 0: If they are perpendicular
  • -1: If they point in opposite directions

In [5]: doc.similarity(nlp(“dog”))

Out[5]: 0.80168555173294953

https://nlp.stanford.edu/projects/glove/

https://spacy.io/usage/vectors-similarity

https://multithreaded.stitchfix.com/blog/2015/03/11/word-is-worth-a-thousand-vectors/

https://towardsdatascience.com/word-vectors-for-non-nlp-data-and-research-people-8d689c692353

 

https://nlpforhackers.io/word-embeddings/

https://nlpforhackers.io/word-embeddings/

https://www.shanelynn.ie/word-embeddings-in-python-with-spacy-and-gensim/

https://nlpforhackers.io/complete-guide-to-spacy/