Recent progress within protein informatics has led to the development of pre-trained protein representations, derived from protein language models such as ESM-1b (Rives et al., 2021) , which have been used to perform state-of-the-art predictive tasks. Such protein language models require vast amounts of training data and so far have tended to use all protein sequences and therefore be general protein representations (Alley et al., 2019, Elnaggar et al., 2021, Rives et al., 2021) . With the creation of the Observed Antibody Space (OAS) database (Kovaltsuk et al., 2018) and subsequent update (Olsen et al., 2021) , enough curated antibody sequences are now available to train a language model specifically for antibodies. An antibody specific model that learnt the semantics of their sequences would allow for more precise predictions of antibody properties and new use cases.

Over the last decade, billions of antibodies have been sequenced (Chaudhary and Wesemann, 2018) . However, in some cases the sequenced antibodies are missing residues due either to sequencing errors, such as ambiguous bases (Huse et al., 2007) , or the limitations of the sequencing techniques used (Kim and Park, 2019) . We find that in OAS, ∼ 80% of the sequences are missing more than one residue at the N-terminal and ∼ 43% the first 15 positions, and ∼ 1% contain at least one ambiguous residue. The ability to accurately restore these missing residues would increase data availability and be of benefit to antibody drug discovery. Currently, sequence imputation can only be done by identifying the correct ImMunoGeneTics (IMGT) germlines from the IMGT/GENE-DB (Giudicelli et al., 2005) and using the germline sequence to add the missing residues. This approach requires correctly determining the allele of the sequence, a process that can be time consuming and/or produce ambiguous results.

Here, we present AbLang, an antibody specific language model trained on either the heavy or light chain antibody sequences from OAS. While AbLang can be used to create representations for residue or sequence specicfi predictions and residue engineering, in this paper we focus on showing how AbLang can be used to accurately restore missing residues in antibody sequences, more accurately than using IMGT germlines or a general protein model like ESM-1b. 2

Two AbLang models were trained, one for heavy and one for light chains. Each AbLang model consists of two parts: AbRep, which creates representations from antibody sequences, and AbHead, which uses the representations to predict the likelihood of each amino acid at each position (Figure 1).

AbLang was implemented using PyTorch 1.8.1 and was inspired by HuggingFace’s (Wolf et al., 2020) Transformer 3.0.2 library. AbRep follows the architecture of RoBERTa (Liu et al., 2019) , except it uses a learned positional embedding layer with a max length of 160. Each of its 12 transformer blocks has 12 attenuated heads, an inner hidden size of 3072 and a hidden size of 768. From AbRep, the res-codings (768 values for each residue) are obtained. AbHead follows the design of RoBERTa’s head model, with a hidden size of 768.

During training, between 1-25% of residues from each sequence were selected, and of these, 80% were masked, 10% randomly changed to another residue and 10% left unchanged. One AbLang model was trained on heavy chain sequences for 20 epochs with a batch size of 8,192, and another on light chain sequences for 40 epochs with a batch size of 4,096. Both models were optimized using an Adam optimizer with a linear warm-up period for 5% of the steps, a peak learning rate of 0.0002, a learning rate decrease following a cosine function, and a weight decay of 0.01. For every dropout and layer normalization, a 0.1 rate and 1e− 12 epsilon was used. The hyperparameters were selected to be similar to those used in the RoBERTa paper (Liu et al., 2019) .

A language model specific for antibodies, should learn a deeper understanding of the semantics of antibodies than current general protein language models. In this work we present AbLang, a language model trained on a large dataset of antibody sequences from OAS. To demonstrate that AbLang has learnt a useful representation of antibodies, we show how AbLang’s sequence representations contain knowledge of the germlines, originating cell type and number of mutations.

We also demonstrate the use of AbLang to restore missing residues in antibody sequences, and show how AbLang performs on par or better than using IMGT germlines, but without the need to have knowledge of the germline. Further, we show how AbLang restores residues more accurately and faster than a current state-of-the-art protein language model ESM-1b, emphasising the benefits and potential of an antibody specific language model.