Is reinforcement learning helpful? about kobe HOT 3 CLOSED

hello,koukou

from kobe.

Hi @jiqiujia ,

Thanks for your interest in this work!

I'm glad that someone finally noticed that part of code! Indeed we've tried to add RL objective and optimize with policy gradient. But unfortunately, they didn't work well enough, so we gave it up before the submission of the paper.

Specifically, we've tried two types of reward, they are:

BLEU reward

As many people did (e.g., https://arxiv.org/pdf/1705.04304), BLEU is straight-forward for combining RL and NLP. We expected an improved BLEU after adding this objective. However, it doesn't work and slows down the training. Also no significant improvement is shown in the generated descriptions, so we gave up BLEU reward.

Attractiveness reward

Our ultimate objective in this paper is to generate good product descriptions, which means the more users are interested and attracted, the better. However, the current objective is merely maximizing the likelihood of generated product descriptions in the dataset, which ignores the users' reaction (e.g., click through rate, page stay time).

Considering this issue, we designed an "attractiveness reward function", which is a trained classifier. This classifier is trained to predict user CTR (Click Through Rate) of a product description. With this reward, we expect the generated descriptions to be more attractive to users (in Chinese, more like 标题党). However, the model tends to overfit the reward function, (i.e., the reward function gives a very high reward but the generated description is not actually attractive judged by human).

To address this problem, we considered adversarial training of the seq2seq model and the classifier (reward giver) to be necessary. We didn't have enough time back then and left that part for future work.

If you are interested in this direction and want further discussions, contact me on my Messenger or add my WeChat: chenqibin99

Best,
Qibin Chen

from kobe.

I get it. Thank you for your reply.

from kobe.