https://arxiv.org/abs/1802.05365

Our representations differ from traditional word type embeddings in that each token is assigned a representation that is a function of the entire input sentence. We use vectors derived from a bidirectional LSTM that is trained with a coupled language model (LM) objective on a large text corpus. For this reason, we call them ELMo (Embeddings from Language Models) representations. Unlike previous approaches for learning contextualized word vectors (Peters et al., 2017; McCann et al., 2017), ELMo representations are deep, in the sense that they are a function of all of the internal layers of the biLM. More specifically, we learn a linear combination of the vectors stacked above each input word for each end task, which markedly improves performance over just using the top LSTM layer.

Combining the internal states in this manner allows for very rich word representations. Using intrinsic evaluations, we show that the higher-level LSTM states capture context-dependent aspects of word meaning (e.g., they can be used without modification to perform well on supervised word sense disambiguation tasks) while lowerlevel states model aspects of syntax (e.g., they can be used to do part-of-speech tagging). Simultaneously exposing all of these signals is highly beneficial, allowing the learned models select the types of semi-supervision that are most useful for each end task.

我们的表示不同于传统的单词类型嵌入，因为每个标记都被分配了一个表示，该表示是整个输入句子的函数。我们使用来自双向LSTM的向量，该LSTM是用耦合语言模型(LM)目标在大型文本语料库上训练的。出于这个原因，我们称它们为ELMo(来自语言模型的嵌入)表示。不同于以往的方法学习上下文化的单词向量(Peters et al.， 2017;McCann et al.， 2017)， ELMo表示是深度的，因为它们是biLM的所有内层的功能。更具体地说，我们学习了每个终端任务的每个输入字之上的向量的线性组合，这比仅使用顶层LSTM层显著提高了性能。

以这种方式组合内部状态可以实现非常丰富的单词表示。使用内在的评价,我们表明,高级LSTM州捕获词义的上下文相关的方面(例如,他们可以使用不需要修改监督词义消歧任务上的表现良好)虽然lowerlevel状态模型方面的语法(例如,他们可以用来做词性标注)。同时暴露所有这些信号是非常有益的，允许学习的模型选择对每个最终任务最有用的半监督类型。

Due to their ability to capture syntactic and semantic information of words from large scale unlabeled text, pretrained word vectors (Turian et al., 2010; Mikolov et al., 2013; Pennington et al., 2014) are a standard component of most state-ofthe-art NLP architectures, including for question answering (Liu et al., 2017), textual entailment (Chen et al., 2017) and semantic role labeling (He et al., 2017). However, these approaches for learning word vectors only allow a single contextindependent representation for each word.

Previously proposed methods overcome some of the shortcomings of traditional word vectors by either enriching them with subword information (e.g., Wieting et al., 2016; Bojanowski et al., 2017) or learning separate vectors for each word sense (e.g., Neelakantan et al., 2014). Our approach also benefits from subword units through the use of character convolutions, and we seamlessly incorporate multi-sense information into downstream tasks without explicitly training to predict predefined sense classes.

由于它们能够从大规模的无标签文本中捕捉单词的句法和语义信息，预训练的单词向量(Turian et al.， 2010;Mikolov等人，2013;Pennington等人，2014)是大多数最先进的NLP架构的标准组件，包括问题回答(Liu等人，2017)、文本蕴涵(Chen等人，2017)和语义角色标注(He等人，2017)。然而，这些学习单词向量的方法只允许每个单词有一个独立于上下文的表示。

之前提出的方法通过丰富子词信息来克服传统单词向量的一些缺点(如Wieting et al.， 2016;Bojanowski等人，2017年)或学习每个单词意义的单独向量(如Neelakantan等人，2014年)。我们的方法还受益于通过使用字符卷积的子词单元，我们可以无缝地将多感觉信息整合到下游任务中，而无需明确地训练来预测预定义的感觉类。

Previous work has also shown that different layers of deep biRNNs encode different types of information. For example, introducing multi-task syntactic supervision (e.g., part-of-speech tags) at the lower levels of a deep LSTM can improve overall performance of higher level tasks such as dependency parsing (Hashimoto et al., 2017) or CCG super tagging (Søgaard and Goldberg, 2016). In an RNN-based encoder-decoder machine translation system, Belinkov et al. (2017) showed that the representations learned at the first layer in a 2- layer LSTM encoder are better at predicting POS tags then second layer. Finally, the top layer of an LSTM for encoding word context (Melamud et al., 2016) has been shown to learn representations of word sense. We show that similar signals are also induced by the modified language model objective of our ELMo representations, and it can be very beneficial to learn models for downstream tasks that mix these different types of semi-supervision.

Dai and Le (2015) and Ramachandran et al. (2017) pretrain encoder-decoder pairs using language models and sequence autoencoders and then fine tune with task specific supervision. In contrast, after pretraining the biLM with unlabeled data, we fix the weights and add additional taskspecific model capacity, allowing us to leverage large, rich and universal biLM representations for cases where downstream training data size dictates a smaller supervised model.

先前的研究也表明，不同层次的深层birnn编码不同类型的信息。例如，在深度LSTM的低层引入多任务句法监督(如词性标签)可以提高更高级别任务的整体性能，如依赖句法分析(Hashimoto等，2017)或CCG超级标签(Søgaard和Goldberg, 2016)。在基于rnn的编码器-解码器机器翻译系统中，Belinkov等人(2017)表明，在2层LSTM编码器中，在第一层学习到的表示比第二层更能预测POS标签。最后，LSTM的顶层用于编码单词上下文(Melamud et al.， 2016)已经被证明可以学习单词意义的表示。我们发现，我们的ELMo表示的修改的语言模型目标也会诱导出类似的信号，这对于学习混合了这些不同类型的半监督的下游任务的模型非常有益。

Dai and Le(2015)和Ramachandran等人(2017)使用语言模型和序列自动编码器预训练编码器-解码器对，然后通过任务特定监督进行微调。相反，在使用未标记数据预训练biLM之后，我们确定了权重，并添加了额外的特定任务模型容量，允许我们利用大型、丰富和通用的biLM表示，在下游训练数据大小决定较小监督模型的情况下。

Given a sequence of N tokens, (t1, t2, ..., tN ), a forward language model computes the probability of the sequence by modeling the probability of token tk given the history (t1, ..., tk−1):

 Recent state-of-the-art neural language models (Jozefowicz et al. ´ , 2016; Melis et al., 2017; Merity et al., 2017) compute a context-independent token representation x LM k (via token embeddings or a CNN over characters) then pass it through L layers of forward LSTMs. At each position k, each LSTM layer outputs a context-dependent representation −→h LM k,j where j = 1, . . . , L. The top layer LSTM output, −→h LM k,L , is used to predict the next token tk+1 with a Softmax layer

​
给定N个令牌序列，(t1, t2，…)， tN)，正向语言模型通过建模给定历史(t1，…)的标记tk的概率来计算序列的概率。, tk−1):

最新的最先进的神经语言模型(Jozefowicz等人，2016;Melis等人，2017;Merity等人，2017)计算一个上下文无关的令牌表示x LM k(通过令牌嵌入或字符上的CNN)，然后通过L层转发lstm。在每个位置k，每个LSTM层输出一个上下文相关的表示-→h LM k,j，其中j = 1，…，L顶层LSTM输出，−→h LM k,L，用于预测下一个令牌tk+1，具有Softmax层

​

A backward LM is similar to a forward LM, except it runs over the sequence in reverse, predicting the previous token given the future context:

 It can be implemented in an analogous way to a forward LM, with each backward LSTM layer j in a L layer deep model producing representations ←− h LM k,j of tk given (tk+1, . . . , tN ). A biLM combines both a forward and backward LM. Our formulation jointly maximizes the log likelihood of the forward and backward directions:

 We tie the parameters for both the token representation (Θx) and Softmax layer (Θs) in the forward and backward direction while maintaining separate parameters for the LSTMs in each direction. Overall, this formulation is similar to the approach of Peters et al. (2017), with the exception that we share some weights between directions instead of using completely independent parameters. In the next section, we depart from previous work by introducing a new approach for learning word representations that are a linear combination of the biLM layers.

​
它可以以类似于正向LM的方式实现，在L层深层模型中，每个向后LSTM层j产生表示←−h LM k,j of tk given (tk+1，…tN)。biLM结合了向前LM和向后LM。我们的公式共同使正向和反向方向的对数似然最大化:

我们在正反两个方向上绑定令牌表示(Θx)和Softmax层(Θs)的参数，同时在每个方向上为lstm维护单独的参数。总的来说，这个公式类似于Peters等人(2017)的方法，除了我们在方向之间共享一些权重，而不是使用完全独立的参数。在下一节中，我们将从前面的工作中分离出来，介绍一种学习单词表示的新方法，它是biLM层的线性组合。

​

ELMo is a task specific combination of the intermediate layer representations in the biLM. For each token tk, a L-layer biLM computes a set of 2L + 1 representations

 For inclusion in a downstream model, ELMo collapses all layers in R into a single vector, ELMok = E(Rk; Θe). In the simplest case, ELMo just selects the top layer, E(Rk) = h LM k,L , as in TagLM (Peters et al., 2017) and CoVe (McCann et al., 2017). More generally, we compute a task specific weighting of all biLM layers:

 In (1), s task are softmax-normalized weights and the scalar parameter γ task allows the task model to scale the entire ELMo vector. γ is of practical importance to aid the optimization process (see supplemental material for details). Considering that the activations of each biLM layer have a different distribution, in some cases it also helped to apply layer normalization (Ba et al., 2016) to each biLM layer before weighting.

​
ELMo是biLM中特定于任务的中间层表示的组合。对于每个令牌tk, l层biLM计算一组2L + 1表示

为了包含在下游模型中，ELMo将R中的所有层合并为一个单个向量，ELMok = E(Rk;Θe)。在最简单的情况下，ELMo只选择顶层，E(Rk) = h LM k,L，如TagLM (Peters et al.， 2017)和CoVe (McCann et al.， 2017)。更一般地，我们计算所有biLM层的特定任务权重:

在(1)中，s任务是softmax归一化的权值，标量参数γ任务允许任务模型缩放整个ELMo向量。γ对于辅助优化过程具有实际重要性(详情见补充材料)。考虑到每个biLM层的活化分布不同，在某些情况下，在加权前对每个biLM层进行层归一化(Ba et al.， 2016)也有帮助。

​

Given a pre-trained biLM and a supervised architecture for a target NLP task, it is a simple process to use the biLM to improve the task model. We simply run the biLM and record all of the layer representations for each word. Then, we let the end task model learn a linear combination of these representations, as described below.

First consider the lowest layers of the supervised model without the biLM. Most supervised NLP models share a common architecture at the lowest layers, allowing us to add ELMo in a consistent, unified manner. Given a sequence of tokens (t1, . . . , tN ), it is standard to form a context-independent token representation xk for each token position using pre-trained word embeddings and optionally character-based representations. Then, the model forms a context-sensitive representation hk, typically using either bidirectional RNNs, CNNs, or feed forward networks.

The pre-trained biLMs in this paper are similar to the architectures in Jozefowicz et al. ´ (2016) and Kim et al. (2015), but modified to support joint training of both directions and add a residual connection between LSTM layers. We focus on large scale biLMs in this work, as Peters et al. (2017) highlighted the importance of using biLMs over forward-only LMs and large scale training.

To balance overall language model perplexity with model size and computational requirements for downstream tasks while maintaining a purely character-based input representation, we halved all embedding and hidden dimensions from the single best model CNN-BIG-LSTM in Jozefowicz et al. ´ (2016). The final model uses L = 2 biLSTM layers with 4096 units and 512 dimension projections and a residual connection from the first to second layer. The context insensitive type representation uses 2048 character n-gram convolutional filters followed by two highway layers (Srivastava et al., 2015) and a linear projection down to a 512 representation. As a result, the biLM provides three layers of representations for each input token, including those outside the training set due to the purely character input. In contrast, traditional word embedding methods only provide one layer of representation for tokens in a fixed vocabulary.

After training for 10 epochs on the 1B Word Benchmark (Chelba et al., 2014), the average forward and backward perplexities is 39.7, compared to 30.0 for the forward CNN-BIG-LSTM. Generally, we found the forward and backward perplexities to be approximately equal, with the backward value slightly lower.

Once pretrained, the biLM can compute representations for any task. In some cases, fine tuning the biLM on domain specific data leads to significant drops in perplexity and an increase in downstream task performance. This can be seen as a type of domain transfer for the biLM. As a result, in most cases we used a fine-tuned biLM in the downstream task. See supplemental material for details.

在1B Word Benchmark (Chelba et al.， 2014)上进行了10个epoch的训练后，平均向前和向后困惑为39.7，而向前CNN-BIG-LSTM为30.0。一般来说，我们发现正向和向后的困惑是近似相等的，向后的值略低。

一旦经过预处理，biLM可以计算任何任务的表示。在某些情况下，对特定领域数据的biLM进行微调可以显著降低复杂程度，并提高下游任务的性能。这可以看作是biLM的一种域转移。因此，在大多数情况下，我们在下游任务中使用了一个经过微调的biLM。详见补充资料。

Table 1 shows the performance of ELMo across a diverse set of six benchmark NLP tasks. In every task considered, simply adding ELMo establishes a new state-of-the-art result, with relative error reductions ranging from 6 - 20% over strong base models. This is a very general result across a diverse set model architectures and language understanding tasks. In the remainder of this section we provide high-level sketches of the individual task results; see the supplemental material for full experimental details.

Question answering The Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) contains 100K+ crowd sourced questionanswer pairs where the answer is a span in a given Wikipedia paragraph. Our baseline model (Clark and Gardner, 2017) is an improved version of the Bidirectional Attention Flow model in Seo et al. (BiDAF; 2017). It adds a self-attention layer after the bidirectional attention component, simplifies some of the pooling operations and substitutes the LSTMs for gated recurrent units (GRUs; Cho et al., 2014). After adding ELMo to the baseline model, test set F1 improved by 4.7% from 81.1% to 85.8%, a 24.9% relative error reduction over the baseline, and improving the overall single model state-of-the-art by 1.4%. A 11 member ensemble pushes F1 to 87.4, the overall state-of-the-art at time of submission to the leaderboard.2 The increase of 4.7% with ELMo is also significantly larger then the 1.8% improvement from adding CoVe to a baseline model (McCann et al., 2017).

表1显示了ELMo在6个不同的基准NLP任务集上的性能。在考虑的每一项任务中，只需添加ELMo，就可以建立一个新的最先进的结果，相对于强大的基础模型，误差降低幅度从6 - 20%不等。在不同的集合模型体系结构和语言理解任务中，这是一个非常普遍的结果。在本节的其余部分中，我们将提供个别任务结果的高级草图;完整的实验细节见补充材料。

斯坦福问答数据集(SQuAD) (Rajpurkar等人，2016年)包含100K+众包问题回答对，其中答案是给定维基百科段落中的一个跨度。我们的基线模型(Clark and Gardner, 2017)是Seo等人的双向注意流模型的改进版本(BiDAF;2017)。它在双向注意组件后增加了一个自注意层，简化了部分池化操作，并将lstm替换为门选循环单元(gru;赵等人，2014)。在基线模型中加入ELMo后，测试集F1从81.1%提高到85.8%，提高了4.7%，相对于基线降低了24.9%的相对误差，并将单个模型的整体水平提高了1.4%。11名成员将F1推向87.4，在提交到排行榜时的整体最先进的与将CoVe添加到基线模型中1.8%的改善相比，ELMo增加4.7%的改善也显著增加(McCann et al.， 2017)。

 Table 1: Test set comparison of ELMo enhanced neural models with state-of-the-art single model baselines across six benchmark NLP tasks. The performance metric varies across tasks – accuracy for SNLI and SST-5; F1 for SQuAD, SRL and NER; average F1 for Coref. Due to the small test sizes for NER and SST-5, we report the mean and standard deviation across five runs with different random seeds. The “increase” column lists both the absolute and relative improvements over our baseline.

​

表1:ELMo增强神经模型与最先进的单模型基线在六个基准NLP任务上的测试集比较。性能指标因任务而异——SNLI和SST-5的准确性;F1代表SQuAD, SRL和NER;Coref的平均F1。由于NER和SST-5的测试规模较小，我们报告了五次不同随机种子运行的平均值和标准偏差。“增加”列列出了相对于基线的绝对改进和相对改进。

​

Textual entailment Textual entailment is the task of determining whether a “hypothesis” is true, given a “premise”. The Stanford Natural Language Inference (SNLI) corpus (Bowman et al., 2015) provides approximately 550K hypothesis/premise pairs. Our baseline, the ESIM sequence model from Chen et al. (2017), uses a biLSTM to encode the premise and hypothesis, followed by a matrix attention layer, a local inference layer, another biLSTM inference composition layer, and finally a pooling operation before the output layer. Overall, adding ELMo to the ESIM model improves accuracy by an average of 0.7% across five random seeds. A five member ensemble pushes the overall accuracy to 89.3%, exceeding the previous ensemble best of 88.9% (Gong et al., 2018).

Semantic role labeling A semantic role labeling (SRL) system models the predicate-argument structure of a sentence, and is often described as answering “Who did what to whom”. He et al. (2017) modeled SRL as a BIO tagging problem and used an 8-layer deep biLSTM with forward and backward directions interleaved, following Zhou and Xu (2015). As shown in Table 1, when adding ELMo to a re-implementation of He et al. (2017) the single model test set F1 jumped 3.2% from 81.4% to 84.6% – a new state-of-the-art on the OntoNotes benchmark (Pradhan et al., 2013), even improving over the previous best ensemble result by 1.2%.

Coreference resolution Coreference resolution is the task of clustering mentions in text that refer to the same underlying real world entities. Our baseline model is the end-to-end span-based neural model of Lee et al. (2017). It uses a biLSTM and attention mechanism to first compute span representations and then applies a softmax mention ranking model to find coreference chains. In our experiments with the OntoNotes coreference annotations from the CoNLL 2012 shared task (Pradhan et al., 2012), adding ELMo improved the average F1 by 3.2% from 67.2 to 70.4, establishing a new state of the art, again improving over the previous best ensemble result by 1.6% F1.

文本蕴涵是在给定一个“前提”的情况下，确定一个“假设”是否为真。斯坦福自然语言推理(SNLI)语料库(Bowman et al.， 2015)提供了大约550K的假设/前提对。我们的基线是Chen et al.(2017)的ESIM序列模型，使用biLSTM编码前提和假设，然后是矩阵注意层、局部推理层、biLSTM推理合成层，最后在输出层之前进行池化操作。总的来说，将ELMo添加到ESIM模型中，在5个随机种子中平均提高了0.7%的精度。五成员集合将整体准确率推高到89.3%，超过了之前的最佳集合88.9% (Gong et al.， 2018)。

语义角色标注(Semantic role labeling, SRL)系统对句子的谓语-argument结构进行建模，通常描述为回答“谁对谁做了什么”。继Zhou和Xu(2015)之后，He等人(2017)将SRL建模为一个BIO标记问题，并使用了一个前后方向交错的8层深层biLSTM。如表1所示,当添加艾尔摩的重新实现他et al .(2017)单一模型测试集F1上涨3.2%从81.4%降至84.6%——一个新的先进的OntoNotes基准(Pradhan et al ., 2013),甚至过去最好的整体结果提高1.2%。

共引用解析共引用解析是将文本中提到的同一潜在现实世界实体进行聚类的任务。我们的基线模型是Lee等人(2017)的端到端跨神经模型。该方法利用biLSTM和注意机制，先计算跨度表示，然后应用softmax提及排序模型求解共参考链。在我们的实验中，使用CoNLL 2012共享任务的OntoNotes共参考评注(Pradhan等人，2012)，添加ELMo将平均F1从67.2提高到70.4，提高了3.2%，建立了一个新的技术状态，再次比之前的最佳集合结果提高了1.6% F1。

Named entity extraction The CoNLL 2003 NER task (Sang and Meulder, 2003) consists of newswire from the Reuters RCV1 corpus tagged with four different entity types (PER, LOC, ORG, MISC). Following recent state-of-the-art systems (Lample et al., 2016; Peters et al., 2017), the baseline model uses pre-trained word embeddings, a character-based CNN representation, two biLSTM layers and a conditional random field (CRF) loss (Lafferty et al., 2001), similar to Collobert et al. (2011). As shown in Table 1, our ELMo enhanced biLSTM-CRF achieves 92.22% F1 averaged over five runs. The key difference between our system and the previous state of the art from Peters et al. (2017) is that we allowed the task model to learn a weighted average of all biLM layers, whereas Peters et al. (2017) only use the top biLM layer. As shown in Sec. 5.1, using all layers instead of just the last layer improves performance across multiple tasks.

Sentiment analysis The fine-grained sentiment classification task in the Stanford Sentiment Treebank (SST-5; Socher et al., 2013) involves selecting one of five labels (from very negative to very positive) to describe a sentence from a movie review. The sentences contain diverse linguistic phenomena such as idioms and complex syntac-Sentiment analysis Ttic constructions such as negations that are difficult for models to learn. Our baseline model is the biattentive classification network (BCN) from McCann et al. (2017), which also held the prior state-of-the-art result when augmented with CoVe embeddings. Replacing CoVe with ELMo in the BCN model results in a 1.0% absolute accuracy improvement over the state of the art.

CoNLL 2003 NER任务(Sang and Meulder, 2003)由来自路透社RCV1语料库的新闻通讯社(newswire)组成，标记有四种不同的实体类型(PER, LOC, ORG, MISC)。根据最近最先进的系统(Lample等，2016;Peters等人，2017)，基线模型使用预训练的词嵌入、基于字符的CNN表示、两个biLSTM层和一个条件随机场(CRF)损失(Lafferty等人，2001)，类似于Collobert等人(2011)。如表1所示，我们的ELMo增强biLSTM-CRF在5次运行中平均F1达到92.22%。我们的系统与Peters等人(2017)之前的技术状态之间的关键区别是，我们允许任务模型学习所有biLM层的加权平均，而Peters等人(2017)只使用顶层的biLM层。如第5.1节所示，使用所有层而不是仅使用最后一层可以提高跨多个任务的性能。

斯坦福情感树库(SST-5)的细粒度情感分类任务;Socher et al.， 2013)涉及从五种标签中选择一种(从非常消极到非常积极)来描述电影评论中的一句话。这些句子中包含了各种各样的语言现象，如成语，复杂的句法-情感分析，否定等语法结构是模型难以学习的。我们的基线模型是McCann等人(2017)提出的双注意分类网络(BCN)，当使用CoVe嵌入进行增强时，BCN也保持了先前的最先进的结果。在BCN模型中，用ELMo替换CoVe，与现有技术相比，绝对精度提高了1.0%。

 Table 3: Development set performance for SQuAD, SNLI and SRL when including ELMo at different locations in the supervised model.

Table 5: All-words fine grained WSD F1. For CoVe and the biLM, we report scores for both the first and second layer biLSTMs.

Table 6: Test set POS tagging accuracies for PTB. For CoVe and the biLM, we report scores for both the first and second layer biLSTMs.

​

表5:全字细粒度WSD F1。对于CoVe和biLM，我们报告第一层和第二层biLSTMs的分数。

表6:PTB测试集POS标记精度。对于CoVe和biLM，我们报告第一层和第二层biLSTMs的分数。

​

These observations can be quantified using an intrinsic evaluation of the contextual representations similar to Belinkov et al. (2017). To isolate the information encoded by the biLM, the representations are used to directly make predictions for a fine grained word sense disambiguation (WSD) task and a POS tagging task. Using this approach, it is also possible to compare to CoVe, and across each of the individual layers.

Word sense disambiguation Given a sentence, we can use the biLM representations to predict the sense of a target word using a simple 1- nearest neighbor approach, similar to Melamud et al. (2016). To do so, we first use the biLM to compute representations for all words in SemCor 3.0, our training corpus (Miller et al., 1994), and then take the average representation for each sense. At test time, we again use the biLM to compute representations for a given target word and take the nearest neighbor sense from the training set, falling back to the first sense from WordNet for lemmas not observed during training.

这些观察可以使用与Belinkov等人(2017)相似的上下文表征的内在评估来量化。为了隔离biLM编码的信息，使用这些表示直接对细粒度词义消除歧义(WSD)任务和POS标记任务进行预测。使用这种方法，还可以与CoVe进行比较，并跨每个单独的层。

对于一个句子，我们可以使用biLM表示来预测目标词的意义，使用简单的1-最近邻方法，类似于Melamud et al.(2016)。为此，我们首先使用biLM计算SemCor 3.0中所有单词的表示，我们的训练语料库(Miller et al.， 1994)，然后取每个意义的平均表示。在测试时，我们再次使用biLM来计算给定目标单词的表示，并从训练集中获取最近的邻居意义，对于在训练中未观察到的词理，返回到WordNet中的第一个意义。

Table 5 compares WSD results using the evaluation framework from Raganato et al. (2017b) across the same suite of four test sets in Raganato et al. (2017a). Overall, the biLM top layer representations have F1 of 69.0 and are better at WSD then the first layer. This is competitive with a state-of-the-art WSD-specific supervised model using hand crafted features (Iacobacci et al., 2016) and a task specific biLSTM that is also trained with auxiliary coarse-grained semantic labels and POS tags (Raganato et al., 2017a). The CoVe biLSTM layers follow a similar pattern to those from the biLM (higher overall performance at the second layer compared to the first); however, our biLM outperforms the CoVe biLSTM, which trails the WordNet first sense baseline.

POS tagging To examine whether the biLM captures basic syntax, we used the context representations as input to a linear classifier that predicts POS tags with the Wall Street Journal portion of the Penn Treebank (PTB) (Marcus et al., 1993). As the linear classifier adds only a small amount of model capacity, this is direct test of the biLM’s representations. Similar to WSD, the biLM representations are competitive with carefully tuned, task specific biLSTMs (Ling et al., 2015; Ma and Hovy, 2016). However, unlike WSD, accuracies using the first biLM layer are higher than the top layer, consistent with results from deep biLSTMs in multi-task training (Søgaard and Goldberg, 2016; Hashimoto et al., 2017) and MT (Belinkov et al., 2017). CoVe POS tagging accuracies follow the same pattern as those from the biLM, and just like for WSD, the biLM achieves higher accuracies than the CoVe encoder.

表5比较了Raganato等人(2017b)的评估框架在Raganato等人(2017a)的四个测试集相同套件中的WSD结果。总体而言，biLM顶层表示的F1为69.0,WSD优于第一层。这可以与使用手工制作功能的最先进的特定于wsd的监督模型(Iacobacci等人，2016年)和也使用辅助粗粒度语义标签和POS标签进行训练的特定于任务的biLSTM (Raganato等人，2017a)相媲美。CoVe biLSTM层遵循与biLM层相似的模式(与第一层相比，第二层的总体性能更高);然而，我们的biLM优于CoVe biLSTM，后者落后于WordNet的第一感觉基线。

为了检查biLM是否捕获基本语法，我们使用上下文表示作为输入到线性分类器，该分类器预测POS标签与宾夕法尼亚树库(PTB)的华尔街日报部分(Marcus et al.， 1993)。由于线性分类器只增加了少量的模型容量，这是对biLM表示的直接测试。与WSD类似，biLM表示与经过仔细调整的、特定于任务的biLSTMs具有竞争性(Ling et al.， 2015;Ma和Hovy, 2016)。然而，与WSD不同的是，使用第一层biLM的准确性高于顶层，这与在多任务训练中使用深层biLSTMs的结果一致(Søgaard和Goldberg, 2016;Hashimoto等人，2017)和MT (Belinkov等人，2017)。CoVe POS标记的准确性遵循与biLM相同的模式，并且就像对于WSD一样，biLM实现了比CoVe编码器更高的准确性。