Let SN = fwigiN=1 be a sequence of N input tokens with wi being the ith element. The corresponding word embedding of SN can be denoted as EN = fxigiN=1, where xi 2 Rd is the d-dimensional word embedding vector of token wi without position information. The self-attention first incorporates position information to the word embeddings and transforms them into queries, keys, and value representations. ( 1 ) ( 2 ) ( 3 ) ( 4 ) qm = fq(xm; m) kn = fk(xn; n) vn = fv(xn; n) am;n = om = exp( q|mpkn )

d j=1 exp( qp|mdkj ) PN N n=1

X am;nvn Where qm; kn; vn incorporates the mth and nth position by functions fq; fk and fv respectively. The attention weights are then calculated using query and key representations, and the output can be computed as the weighted sum of value representations.

The research on position encoding of transformer mainly focuses on choosing suitable function forms of eq. ( 1 ). 2.2

ft:t2fq;k;vg(xi; i) := W t:t2fq;k;vg(xi + pi) Where pi 2 Rd is a d-dimensional vector depending of the position of token xi. [4, 12, 2, 16, 15] used a set of trainable vectors pi 2 fptgtL=1, where L is the maximum sequence length. On the other hand, [21] has proposed to generate pi using the sinusoidal function.

pi;2t pi;2t+1 = sin(k=100002t=d) = cos(k=100002t=d) Where pi;2t is the 2tth element of the d-dimensional vector pi. In Section x, we will show that our proposed RoPE is related to this approach from the perspective of using the sinusoidal function, but ours incorporates relative position information by multiplying sinusoidal function to the context representation instead of adding.

( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) [18] used a different setting of eq. ( 1 ) as following:

fq(xm) := W qxm fk(xn; n) := W k(xn + p~rk) fv(xn; n) := W v(xn + p~rv) Where p~rk; p~rv 2 Rd are trainable relative position embeddings. Note that r = clip(m n; rmin; rmax) represents the relative distance between position m and n. They clipped the relative distance with the hypothesis that precise relative position information is not useful beyond a certain distance.

Keeping the form of eq. ( 3 ), [3] has proposed to decompose the q|mkn term in eq. ( 2 ) as

q|mkn = x|mW q|W kxn + x|mW q|W kpn + p|mW q|W kxn + p|mW q|W kpn They replaced absolute position embedding pn with its sinusoid-encoded relative counterpart p~m n and replaced absolute position pm in the third and fourth term with two trainable vectors u, v independent of the query positions. Further, W k is distinguished for the content-based and location-based key vectors xn and pn, denoted as W k and Wf k, resulting in:

q|mkn = x|mW q|W kxn + x|mW q|Wf kp~m n + u|W q|W kxn + v|W q|Wf kp~m n It is worth mentioning that they remove the position information in the value term by setting fv(xj ) := W vxj . Later works [17, 6, 11, 8] followed this step by only considering inject relative position information into the attention weights. [17] revised eq. ( 6 ) as

q|mkn = x|mW q|W kxn + bi;j Where bi;j is a trainable bias. [11] investigated the middle two terms of ?? and found little correlations between absolute positions and words. Follow [17], they have proposed to model a pair of words or positions by using different projection matrices.

q|mkn = x|mW q|W kxn + p|mUq|Ukpn + bi;j [6] argued that a relative positions of word pair can only be fully modeled by using both the middle two terms of ??, so they have proposed to replace the absolute position embeddings pm and pn in these two terms with relative position embeddings p~m n.

| q|mkn = x|mW q|W kxn + x|mW q|W kp~m n + p~m nW q|W kxn ( 10 ) [15] has compared four variants of relative position embeddings and shown that the variant similar to eq. ( 10 ) is the most efficient among the other three.

All these works modified eq. ( 6 ) based on the decomposition of eq. ( 3 ) under the self-attention setting in eq. ( 2 ), which is originally from [21]. They share the same nature that the position information is injected by deliberately adding to the context representations. Different from these work, our approach aims to derive the relative position encoding directly from eq. ( 1 ) under some constraints. In Section xxx, we show that the derived approach is more interpretable by incorporating relative position information with the rotation of context representations. 3

In this section, we discuss the proposed rotary position embedding (RoPE). We first formulate the relative position encoding problem in section (3.1), we then derive the RoPE in section (3.2) and investigate its properties in section (3.3). Language modeling in Transformer integrates position information of individual tokens through self-attention. We start from eq. ( 1 ) and notice that the q|mkn term in eq. ( 2 ) actually facilitates information exchange between tokens at different positions. In order to incorporate relative position information, we require the inner product of query qm and key kn be formulated by a function g, which takes only the word embeddings xm, xn, and their relative position m n as input variables. In other words, we hope the inner product encodes position information only in the relative form: 3.2.2

General form where g(xm; xn; m

n) = Re[(W qxm)(W kxn) ei(m n) ] where Re[ ] is the real part of a complex number and (W kxn) represents the conjugate complex number of (W kxn). 2 R is a preset non-zero constant. We can further write ffq;kg in matrix multiplication: ffq;kg(xm; m) = cos m sin m

sin m cos m

Wf(q1;1k)g Wf(q2;1k)g

Wf(q1;2k)g ! Wf(q2;2k)g x(m1) ! x(m2) where (x(m1); x(m2)) is xm expressed in the 2D coordinates. Similarly, function g can be turned into matrix form. Thus, we come up with a solution to formulation in section 3.1 under the 2D case. Specifically, incorporating relative position embedding is straightforward: simply rotate the affine-transformed word embedding vector by amount of angle in multiples of its position index. Due to this characteristic, we name it Rotary Position Embedding. In order to generalize our result in 2D to any xi 2 Rd where d is even, we divide the d-dimension space in to d=2 sub-spaces and combine them in merit of the linearity of inner product, turning ffq;kg into: hfq(xm; m); fk(xn; n)i = g(xm; xn; m n) Next, finding such a encoding mechanism is equivalent to solve the function fq(xm; m) and fk(xn; n) that conforms above relation. 3.2 3.2.1

We start from simple case with dimension d = 2. Under this setting, we make use of the geometric property of vectors on 2D plane and its complex form to prove (refer to Appendix A for more details) that a solution to our formulation eq. ( 11 ) is: ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) fk(xn; n) = (W kxn)ein 0 0 sin m 2 cos m 2 . . . 0 0 . . .

q|mkn = (Rd ;mW qxm)|(Rd ;nW kxn) = x|W qRd ;n mW kxn where Rd ;n m = (Rd ;m)|Rd ;n. Notice that Rd is an orthogonal matrix, which ensures the stability during the process of encoding position information. In addition, due to the sparsity of Rd , applying matrix multiplication directly as in eq. ( 16 ) is not computational efficient, we provide another realization in Appendix B. In contrast to the additive nature of position embedding method use by other works, i.e. eqs. ( 3 ) to ( 10 ), our approach is multiplicative. Moreover, our RoPE naturally incorporates relative position through rotation matrix product instead of altering terms of additive position encoding in self-attention.

1 2 3 4 5 6

Long-term decay: Following [21], we choose i = 10000 2i=d. One can prove that this setting provides a long-term decay property (refer to Appendix C for more details), which means the inner-product will decay when the relative position increase. This property coincides with the intuition that a pair of tokens with long relative distance should have less connection.

In this section, we show some preliminary experiment results of RoFormer in Chinese language modeling, complete benchmark on English tasks is in progress and will be released once done. We discuss the implementation of RoFormer in section (4.1) and show the pre-training results with Chinese language in section (4.2). To illustrate the performance of RoFormer on long texts, we choose a task with most documents exceeding 512 characters (section (4.3)) for downstream evaluation and discuss the results in section (4.4).

Our RoFormer model is based on WoBERT[20], from which we replace the absolute position embedding with our proposed RoPE. For cross-comparison with other pre-trained Transformer-based models in Chinese, i.e. BERT[4], WoBERT[20], and NEZHA[23], we tabulate their tokenization level and position embedding in table ( 1 ). We pre-train RoFormer on approximately 34GB data which consists of contents from Chinese Wikipedia, news, forums, etc. The pre-training was done in multiple stages with changing batch size and maximum input sequence length in order to adapt the model with various scenarios, as shown in table ( 2 ). According to table ( 2 ), the accuracy of RoFormer elevates with increasing upper bound of sequence length, which demonstrates the ability of RoFormer in dealing with long texts. We claim that this is attribute to the excellent generalizability of the proposed RoPE. 4.3

We choose Chinese AI and Law 2019 Similar Case Matching dataset(CAIL2019-SCM)[24] to illustrate the ability of

which denotes the vectors with empty position information encoded. With above settings, we manage to find a solution of fq, fk.

First, we take advantage of the geometric meaning of vector in 2D and its complex counter part, decompose functions in eqs. ( 20 ) and ( 21 ) into: fq(xq; m) = Rq(xq; m)ei q(xq;m) fk(xk; n) = Rk(xk; n)ei k(xk;n) g(xq; xk; n m) = Rg(xq; xk; n

m)ei g(xq;xk;n m) Rq(xq; m)Rk(xk; n) = Rg(xq; xk; n k(xk; n) q(xq; m) = g(xq; xk; n m) m) q = kqkei q = Rq(xq; 0)ei q(xq;0) k = kkkei k = Rk(xk; 0)ei k(xk;0) Where Rf , Rg and f , g are the radical and angular components for ffq;kg and g, respectively. Plug them into eq. ( 21 ), we get relation: with the corresponding initial condition as:

On the other hand, according to eq. (26b), notice q(xq; m) q = k(xk; m) does not dependent on query and key, we set them to f := q = k and term f (xfq;kg; m) of position m and is independent of word embedding xfq;kg, we denote it as (m), yielding: k indicates that the angular functions fq;kg is a function Further, by plugging in n = m + 1 in eq. ( 24 ) and consider above equation, we have:

Rq(xq; m) = Rq(xq; 0) = kqk

Rk(xk; n) = Rk(xk; 0) = kkk Rg(xq; xk; n

m) = Rg(xq; xk; 0) = kqkkkk f (xfq;kg; m) = (m) + fq;kg (m + 1) (m) = g(xq; xk; 1) + q

k Since the RHS of the equation is a constant irrelevant to m, function (m) with continuous integer inputs produce an arithmetic progression. Thus, it is straightforward to get:

(m) = m +

2 R are constants and is non-zero.

To summarize our solutions from Equations (27) to (30): fq(xq; m) = kqkei q+m + fk(xk; n) = kkkei k+n + = qei(m + ) = kei(n + ) q = fq(xm; 0) = W qxn k = fk(xn; 0) = W kxn Finally, notice that we haven’t set any constrains to functions in eq. ( 22 ), thus fq(xm; 0) and fk(xn; 0) are left to choose freely. To make our result be comparable to 3, here we simply set:

= 0 in eq. (31), the ultimate solution is: B

Computational efficient realization of rotary matrix multiplication Taking the advantage of the sparsity of Rd ;m in eq. ( 15 ), a more computational efficient realization of multiplication operation between matrix Rd and vector x 2 Rd is: We can group entries of vectors q = W qxm and k = W kxn in pairs, and the inner product of RoPE in 16 can be written as complex number multiplication.

(Rd ;mW qxm)|(Rd ;nW kxn) = Re d=2 1 X q[2i:2i+1]k[2i:2i+1]ei(m n) i i=0 where q[2i:2i+1] represents the 2ith to (2i + 1)th entries of q. Denote hi = q[2i:2i+1]k[2i:2i+1] and Sj = Pij=01 ei(m n) i , and let hd=2 = 0 and S0 = 0, we can rewrite the summation using Abel transformation d=2 1 X i=0 i=0 d=2 1 X i=0 hi(Si+1

Si) =

Si+1(hi+1 hi) (36) (30) (31) (32) (33) (34) (35) 50 relative distance d=2 1 X Si+1(hi+1 i=0 d=2 1 X jSi+1jj(hi+1 i=0

(37)

d=2 1 hij Xi=0 jSi+1j n increases by setting i = 10000 2i=d, as shown in So we have the value of d1=2 Pid==21 jSij decay with the relative distance m fig. ( 2 ).