Introduction

Word meaning is embedded in context. This simple assumption is rooted in contemporay language philosophy as well as cognitive linguistics and recently led to very successful models of distributed word representations, namely in the form of word embeddings trained by deep forward neural networks.

Word embeddings are trained on large unannotated corpora. Each update is conditioned on its immediate context, most often the history of preceding words within a limited window. The core idea is that a sequence of words contains enough information to predict the next word, or in other words, that the meaning of a word can be defined by the words that typically precede (or surround it) in a corpus.

Despite its success, there are several conceptual shortcomings to this approach that suggest poor performance in affected linguistic tasks:

• Limited/ uncertain window size: The context is usually confined to five to ten words. The optimal window size is difficult to determine.
• Inflexibility w.r.t. context dependencies: Fixed time windows do not capture dynamic long-term dependencies.
• Unscalability w.r.t. larger contexts: Depending on the architecture, the training complexity grows by a magnitude larger than the window size.
• Insensitivity w.r.t. context distances: All words within the window are given equal importance for the meaning of the current word.

Humans exploit much longer contexts when assessing the meaning of a word [1]. This insight gave rise to recurrent neural networks for language modelling that are able to dynamically keep track of the context history [2]. However, implementation issues, such as the vanishing/ exploding gradient problem, pose very tight restrictions to the length of the history [3].

Optimization techniques, such as Hessian Free Optimization [4], are not suitable for language modelling tasks due to much higher computational costs [5].

Another solution to this problem is to compress the history to an abstract representation that is fed into the current time step via a separate context layer [6]. This in turn blurs the impact of specific words from the history which is fatal to the goal of pinpointing long-term dependencies, especially across sentence borders.

Outline

We claim that common models of word embeddings are still relatively local in nature and do not make use of important information given in the discourse or even in the same sentence.

Long-term dependencies between words contain valuable information about the meaning of a word. They should prove useful not only to enrich the representation of a word, but also to model representations of context that can be used for several context-dependent NLP tasks, such as sense disambiguation.

To address the shortcomings mentioned above, we propose the investigation of Long Short-Term Memory (LSTM) networks, first developed by Hochreiter and Schmidhuber [7]. LSTM networks make use of dedicated node types that learn to control the impact and memorization of specific information from the past. They have been applied successfully to a variety of NLP tasks, such as language modeling [5], handwriting recognition [8], named entity recognition [9] and speech recognition [10].

Our goal is to create task-independent distributed representations of contexts from running text (context embeddings, CE) and to enhance existing models of distributed word representations by long-term dependency information.

This idea is biologically motivated by conceptual short term memory which provides an understanding of how pieces of input information cycle inside the memory system to imbue the interpretation of future input [11].

Related work

LSTM for language modeling

Sundermeyer & Schlüter & Ney [5] train an LSTM Neural Network for language modeling. It predicts the probability distribution for the next word given the history via a softmax output layer. The training criterion is the cross entropy error (maximum likelihood).

Note that the projection layer is not mentioned as representation layer and is not explicitly kept after the training as usable word representations.

The projection layer does not seem to be a distributed representation drawn from a lookup-table [???]. The activations in the projection layer are simply computed by the identity function on the weighted linear combination of the current one-hot input vector. The weight matrix between the input and the projection layer is tied for all history words.

Multitask learning

Collobert & Weston [12] train one deep network on several tasks at once: language modelling, POS-tagging, chunking, named entity recognition and semantic role labeling. The idea is that knowledge learned in one tasks helps in learning another task.

All tasks except the language model is trained on labeled text.

The first hidden layer consists of lookup tables (one for each task) and learns relevant features for each word. Features are represented as vectors of different sizes and fixed dictionaries. They deliver the supervised part.

Their concenation defines the word's position in the feature space.

Some lookup tables are shared, others and subsequent layers are task specific.

Multiple word prototypes

Huang et al. [13] learn multiple embeddings per word to capture homonymy and polysemy. For each word, the syntactic representation (local context) and the semantic representation (global context) are kept separate.

The local context model is trained via a contrastive estimation network on an large unlabeled corpus (preserves order information). The global context model is a network that takes the local context as input, plus a vector that is computed as the weighted average of all word vectors in the document (bag of words). Both models compute a score that is summed to the final score of the total network.

[14]

Experimental setup

Definitions:

• WE: word embedding
• CE: context embedding

1. Unsupervised training of the LSTM network

Given:

• Unlabeled text, preferrably as meaningful chunks of standalone topics (e.g. Wikipedia)
• Pretrained word embeddings (e.g. Polyglot)

Procedure:

1. For each token t_i (i ≥ 0)

   1. Retrieve WE w_i for t_i from lookup table
   2. Input w_i
   3. Predict the next word via softmax on a probability distribution layer P_i(w∣h)
      • Alternatively: predict the embedding of the next word WE_i + 1 via log-bilinear
   4. Backpropagate the error (cross entropy)
   5. Update the network parameters (input weights of cells and gates, output weights)

LSTM networks for language modelling

It is possible to utilize the topic-chunks given by the training data. For example, the forget gate at the constant error carousel (CEC) can be reset to 1 ("do not forget") after each topic-chunk.

Important keywords in the discourse are expected to have a larger impact on the update. The network is expected to learn when a token is worth remembering, in the course of processing the discourse.

2. Retrieval of training set for WSD

Given:

• Sense-labeled text (e.g. MASC corpus)
• LSTM network trained to predict CEs (probability distributions) P(w∣h)

Procedure:

1. For each token t_i with label l_i (i ≥ 0)

   1. Retrieve WE w_i for t_i from lookup table
   2. Input w_i
   3. Output: CE_i
   4. Save tupel ⟨l_i, CE_i⟩

3. Supervised training of binary classifier

Given:

• Set of label-CE-tupels

Procedure:

1. Split the set of tupels into training set and test set
2. Train an SVM on the training set
3. Compute precision and recall on the test set

Future experiments

We will train and cluster generic context embeddings via k-means. Context embeddings of homonyms are expected to aggregate according to their respective meanings in the context meaning space. Each cluster provides a representative centroid.

Further application would include enhancing existing word embeddings by concatenation or convolution of both types of representation. Several tasks are possible for comparison.

Sketch of work plan

• Evaluate the applicability Graves' recurrent neural network library (http://sourceforge.net/projects/rnnl/) for our purposes.
• Pick and prepare a corpus, preferably one with short chunks of pragmatically coherent text (e.g. Wikipedia, BBC News).
• Sketch and implement an LSTM network to train the context embeddings.
• Visualize the context embeddings (t-SNE). Compare a set of related context embeddings (e.g. the context embeddings of the word "bark") to the respective word embedding.
• Evaluate the context embeddings on a real-life task (word sense disambiguation). Compare them to existing word embeddings as well as distributional state-of-the-art models.
• If successful: write a publication.

Parallel work: collect and outline related work and research around recurrent and LSTM networks.

Funding