The work of this paper is in the recall stage to satisfy the users' effective retrieval of interested items. It is very important to establish a user interest model and find the expression of user interest, but it is not easy because of the diversity of user interests.
Some existing user interest expression methods:
1. The method based on collaborative filtering expresses users' interests through historical interactive items or hidden factors: it will encounter sparse and computational problems.
2. The method based on deep learning uses low-dimensional embedding vectors to express users' interests;
The author thinks that this is a bottleneck of multi-interest expression, because all the information related to the user's multi-interest must be compressed into a representation vector, so all the information about the user's multi-interest is mixed together, which leads to inaccurate item detection in the recall stage.
3.DIN adds attention to wit on the basis of embedding: However, the attention mechanism needs to recalculate the user representation for each target item, so it cannot be used in the recall stage.
About the capsule network:
Dynamic routing algorithm between capsules, the core of dynamic routing capsule algorithm lies in the updating method of parameter B here: when updating parameters, both low-level features and output capsule features are considered, because they are vectors, when they are in the same direction, that is, their similarity is high, the current low-level features can better reflect the image features, the product is positive, and the weight of B is increased, indicating that the current low-level capsules are more "accepted" by high-level capsules; On the contrary, when the two are reversed, the matching degree between the current low-level feature and the output capsule is not high, the product is negative, and the B weight is reduced, indicating that the current low-level capsule is "rejected" by the high-level capsule. In this way, the relationship between low-level features and high-level features is established, and the model can better "understand" the image.
A "capsule" is a group of small neurons that gather to output the whole vector. Dynamic routing is used to learn the connection weight between capsules, and expectation maximization algorithm is used to improve it, which overcomes some shortcomings and obtains better accuracy.
Main contributions:
Aiming at the multi-interest problem of users in the recall stage, a multi-interest network using dynamic routing to learn user representation is proposed.
The main "innovation" is to use the dynamic routing algorithm of capsule network to obtain the multi-interest representation of users, aggregate the historical behaviors of users into multiple groups of content, and further infer the user representation vector corresponding to specific interests by using each group of historical behaviors. In this way, for a specific user, MIND outputs multiple representation vectors, which respectively represent different interests of the user. The user representation vector is only calculated once, and can be used to retrieve related items from a billion-scale project in the matching stage.
Mission objective
The goal of the recall task is to retrieve thousands of item sets related to users' interests for each user from a billion-level item library.
Model input
For the model, the input of each sample can be represented as a triple, which represents the set of items that have interacted with the user, that is, the user's historical behavior; Represents the user's attributes, such as gender, age, etc. Represented as some characteristics of the target project, such as project id and category id.
Core task
The learning function can map user project instances (original features) to users' interests. The embedded expression set is the vector representation of users, and the embedding dimension represents the number of vectors, that is, the number of interests.
If = 1, that is, the embedding expression of other models (such as Youtube DNN), the embedding function of the item is: which represents an embedding &; Common floor
Final results
Search the top n candidates according to the scoring function:
Search according to the scoring function: that is, according to the maximum value of the inner product of the target item and the user-represented vector as the similarity basis, DIN's attention part also measures the similarity between the two in this way.
The input of the embedding layer consists of three parts: user attributes, user behaviors and target item tags. Each part is composed of multiple id features, which belong to high-dimensional sparse data, and needs embedding technology to map it into low-dimensional dense vectors.
Compared with a single vector to express users' interests, the author uses multiple expression vectors to express users' different interests respectively. In this way, in the recall stage, users' multiple interests can be considered separately, and each interest can be searched more accurately.
In order to learn multi-interest representation, the author uses dynamic routing of capsule network representation learning to group users' historical behaviors into multiple clusters. Projects from a cluster should be closely related and represent a specific aspect of users' interests.
Dynamic routing
Capsule is a new neuron, which is represented by vector instead of scalar used in ordinary neural networks. Vector-based capsules are expected to represent different attributes of an entity, where the direction of the capsule represents the attribute and the length of the capsule represents the probability of the existence of the attribute.
Dynamic routing is an iterative learning algorithm in capsule network, which is used to learn the routing logarithm between low-level capsules and high-level capsules and get the representation of high-level capsules.
We assume that the capsule network has two layers, namely, low-level capsules and high-level capsules, and the number of capsules represents the number of neurons (vector length) in each capsule. The routing logarithm is obtained and updated by the following calculation: where it represents the bilinear mapping matrix to be learned (capsule network is originally called transformation matrix).
By calculating the routing logarithm, the candidate vector of high-order capsules is calculated as the weighted sum of all low-order capsules:
The process of using multiple vectors to express users' different interests and grouping users' historical behaviors into multiple interest capsules. The implementation logic is as follows:
Input:
Output:
Definition:
(1) dynamic interest number
(2) Low-order behavior vector embedding expression: representing the user's behavior vector (same as above).
(3) Embedding high-order interest vector expression: representing the user's interest vector (same as above).
(4) Path logic between behavior vector and interest vector:
(5) Bilinear mapping matrix:
Steps:
(1) Calculate the embedding number of interest.
(2) Initialization (initialization using normal distribution)
(3) the number of traversal iterations
(3. 1) For all behavioral routes, calculate
(3.2) For all interest paths, calculations and
(3.3) Iteratively update the * * * enjoyment matrix.
Through multiple interest extraction layer, multiple interest capsules are established from user behavior embedding. In the training, we designed a label perception attention layer: let the label (target) items choose the used interest capsules. Especially, for each label item, the similarity between the interest capsule and the embedding of the label item is calculated, and the weights of the interest capsule and the user representation vector as the target item are calculated, and the weight of the interest capsule is determined through the corresponding compatibility.
train
After the embedding of the user vector and the tag item is obtained, the interaction probability between the user and the tag item is calculated:
One of the characteristics of the interview questions in the civil service examination is its strong timeliness, and the questions will be closely c