목차

ABSTRACT

[LM with human preferences에 RL 적용]

• 본 논문은 인간 선호도에 맞추는 pre-trained large language models (LMs) 문제를 해결한다.
• 텍스트 생성을 순차적 decision-making problem로 본다면, 강화 학습(RL)은 적절한 conceptual 프레임워크가 될 수 있다.

[단점]

• 그러나 LM 기반 생성을 위해 RL을 사용하는 것은 combinatorial action space으로 인한 training 불안전성 존재
• LM alignment을 위해 맞춤화된 오픈 소스 라이브러리와 벤치마크의 부족을 포함한 경험적 challenges에 직면한다.
  ➡ 따라서, RL이 NLP를 위한 실용적인 패러다임인가? 라는 질문이 제기된다.

[단점 해결]
이에 대한 답변을 돕기 위해 아래 3가지 방법을 제시한다,

• RL4LMs를 소개
  • RL로 언어 생성기를 optimizing하기 위한 오픈 소스 모듈식 라이브러리
  • 라이브러리는 임의의 reward 함수를 사용하여 HuggingFace 라이브러리(Wolf et al., 2020)에서 encoder 또는 encoder-decoder LM을 훈련하는 데 사용할 수 있는 on-policy RL algorithms으로 구성된다.
• GRUE (General Reinforced-language Understanding Evaluation) benchmark 제시
  • target 문자열이 아니라 인간 선호도의 자동화된 측정을 capture하는 reward 기능에 의해 supervised되는 6개 언어 생성 작업 세트
  • GRUE는 NLP 작업에 대한 RL algorithms의 첫 번째 leaderboard-style evaluation이다.
• NLPO(Natural Language Policy Optimization) 소개
  • 언어 생성에서 combinatorial action space을 효과적으로 줄이는 방법을 배우는 사용하기 쉬운 성능의 RL algorithm

[결과]
1) RL 기술이 일반적으로 인간 선호도에 LM을 맞추는 데 supervised 방법보다 낫다는 것을 보여준다.
2) NLPO가 자동 및 인간 평가를 모두 기반으로 previous policy gradient methods (PPO)보다 더 큰 안정성과 성능을 나타낸다

1 INTRODUCTION

언어 기술의 궁극적인 목표는 인간과 상호작용하는 것이다.
[단점]

• 그러나 대부분의 언어 모델은 인간 선호도의 직접적인 신호 없이 훈련되며, supervised target 문자열은 (때로는 조잡한) proxy 역할을 한다.
• 사용자 피드백을 통합하는 한 가지 옵션은 human-in-the-loop를 통한 것이다.
• 즉, 모델이 훈련할 때 사용자는 각 샘플에 대한 피드백을 온라인으로 제공해야 하지만, 이 정도의 밀도 높은 supervision은 종종 금지되고 비효율적이다.

[해결: Automated metrics]

• 위의 문제에 대해 Automated metrics은 괜찮은 절충안을 제공한다.
  • pairwise learned preference models(Ouyang et al., 2022), BERTScore(Zhang et al., 2019), BLEURT(Sellam et al., 2020)는 이전 메트릭(BLEU, METERE et al.)에 비해 인간 판단과의 상관관계가 크게 개선 되었으며 평가 비용이 저렴하다.

[Automated metrics의 문제]

• 그러나 이러한 기능은 일반적으로 per-token별로 차별화 불과.
• 사람과 마찬가지로 메트릭은 전체 generations에 대한 품질 추정치만 제공 가능

[해결: RL]

• 강화 학습(RL)은 순차적 의사 결정 문제로 캐스팅될 때 LM 기반 생성에 대한 미분 불가능한 스칼라 목표를 최적화하기 위한 자연스러운 경로를 제공한다.
• 최근 연구는 preference-based rewards을 제한하여 fluency 개념을 통합함으로써 RL을 통해 LM을 인간 선호도에 맞추는 유망한 결과를 보여주었지만,
  ⚠️ 이러한 방식의 진전은 오픈 소스 벤치마크와 알고리즘 구현의 부족으로 인해 방해받고 있다.

[본 논문]
LM을 더 잘 정렬하기 위해 RL 알고리듬을 구축하는 연구를 용이하게 하기 위해 라이브러리, 벤치마크 및 알고리듬을 출시한다. 먼저 PPO/A2C 등과 같은 다양한 기존 RL 방법을 사용하여 생성적 HuggingFace 모델(예: GPT-2 또는 T5)을 훈련할 수 있는 RL4LMs 라이브러리를 릴리스한다. 다음으로, 우리는 RL4를 사용하여 훈련된 모델을 적용한다새로운 GRUE(일반 강화 언어 이해 평가) 벤치마크에 대한 LM: GRUE는 7개의 현대 NLP 작업의 모음이다(자세한 내용은 표 1 참조). 다른 벤치마크와 달리, 우리는 감독된 훈련 대신 각 작업을 보상 기능과 쌍으로 구성한다. GRUE는 유창한 언어 생성기를 유지하면서 이러한 보상 기능을 최적화하기 위해 모델에 도전한다. 우리는 보상을 최적화하기 위해 작업별 감독 사전 훈련 유무에 관계없이 RL을 통해 언어 모델을 훈련한다. 마지막으로, 기존 RL 방법을 넘어 토큰 수준에서 언어 분포에 대한 작업별 제약 조건을 동적으로 학습하는 NLPO(Natural Language Policy Optimization)라는 새로운 정책 RL 알고리듬을 소개한다. To facilitate research in building RL algorithms to better align LMs, we release a library, a benchmark, and an algorithm. First, we release the RL4LMs library, which enables generative HuggingFace models (e.g., GPT-2 or T5) to be trained using a variety of existing RL methods like PPO/A2C/etc. Next, we apply models trained using RL4LMs to the new GRUE (General Reinforced-language Understanding Evaluation) benchmark: GRUE is a collection of 7 contemporary NLP tasks (see Table1 for details); in contrast to other benchmarks, instead of supervised training, we pair each task with reward function(s). GRUE challenges models to optimize these reward functions while remaining fluent language generators. We train language models via RL—both with and without task specific supervised pre-training—to optimize rewards. Finally, beyond existing RL methods, we introduce a novel on-policy RL algorithm called NLPO (Natural Language Policy Optimization), that dynamically learns task-specific constraints over the distribution of language at a token level.

GRUE와 인간 평가에 대한 실험은 NLPO가 PPO를 포함한 대안에 비해 언어 유창성을 유지하면서 학습 선호 보상의 균형을 더 잘 유지한다는 것을 보여준다(그림 1). 우리는 스칼라 보상 피드백에서 학습하기 위해 RL을 사용하는 것이 더 나을 수 있음을 발견했다. (1) 지도 학습을 통한 추가 전문가 시연을 사용하는 것보다 데이터 효율성이 더 높을 수 있다. 학습된 보상 함수는 5배 더 많은 데이터로 훈련된 지도 방법보다 RL 방법에 대한 신호로 사용될 때 더 큰 성능을 가능하게 한다nd (2) 매개 변수 효율성—감독과 NLPO의 조합으로 훈련된 2억 2천만 개의 매개 변수 모델이 30억 개의 감독 모델을 능가할 수 있도록 한다. 우리는 우리가 공개한 벤치마크, 기준선 및 빌딩 블록이 LM을 인간 선호도에 맞추는 연구를 추진하는 역할을 하기를 바란다. Experiments on GRUE and human evaluations show that NLPO better balances learning preference rewards while maintaining language fluency compared to alternatives, including PPO (Figure 1). We find that using RL to learn from scalar reward feedback can be more: (1) data efficient than using additional expert demonstrations via supervised learning (though a combination of both is best)—a learned reward function enables greater performance when used as a signal for an RL method than a supervised method trained with 5 times more data, and (2) parameter efficient—enabling a 220 million parameter model trained with a combination of supervision and NLPO to outperform a 3 billion supervised model. We hope that the benchmarks, baselines, and building blocks we release serve to drive forward research in aligning LMs to human preferences.

3 RL4LMS: RL로 LMS를 교육하기 위한 라이브러리

[RL4LM]

• LM-based generation에서 RL algorithms을 fine-tuning하고 평가하기 위한 빌딩 블록이 있는 오픈 소스 라이브러리인
• 라이브러리는 HuggingFace와 baselines-3(Raffin et al., 2021)을 기반으로 구축되어 인터페이스의 중요 구성 요소를 결합한다.
• RL4LM은 stable-baselines-3의 on-policy RL algorithm으로 HuggingFace의 decoder 전용 또는 encoder-decoder transformer 모델을 훈련시키는 데 사용될 수 있다.
• 또한 PPO(Schulman et al., 2017), TRPO(Schulman et al., 2015a), A2C(Mnih et al., 2016) 및 자체 NLPO(§4)와 같은 LM 미세 조정에 맞게 조정된 인기 있는 on-polic RL algorithm의 신뢰할 수 있는 구현을 제공한다.
• 라이브러리는 모듈식으로 되어 있어 사용자가 맞춤형 환경을 플러그인하고, 기능, 메트릭 및 알고리즘을 보상할 수 있다.
• 초기 릴리스에서는 6개의 서로 다른 NLP 작업, 16개의 평가 메트릭 및 보상, 4개의 RL 알고리듬에 대한 지원을 제공한다.

3.1 ENVIRONMENTS: 토큰 레벨 MDP로서의 생성

각 ENVIRONMENTS은 NLP task임

• dataset: 우리는 N개의 예제 중 supervised datase \(D = {(x_i, y_i)}^N_{i=1}\)이 주어짐.
  • \(x ∈ X\): 언어 입력(language input)
  • \(y ∈ Y\): task 문자열입니다.
• Generation: Markov Decision Process (MDP)
  • Generation은 유한한 어휘 V를 사용하여 MDP로 볼 수 있다.
  • \(MDP: (S,A,P,r, 𝜌_0, 𝛾, T)\).
    • \(S\): state들의 유한한 set
    • \(A\): action들의 유한한 set
    • \(P\): \(S * A * S\) ➡ \(ℝ\)은 전통적인 확률
      어떤 state에서 어떤 act를 하면은 다음 state로 갈 확률이 얼만지
    • \(r\): \(S\) ➡ \(ℝ\), reward 함수
    • \(𝜌_0\): \(S\) ➡ \(ℝ\), 처음 state(\(s_0\))의 분포
      시작하는 state가 여러개가 있을 수 있는데 이 시작 위치의 분포
    • \(𝛾 ∈ (0,1)\): discount factor
      0~1사이의 값을 갖는 Discount Factor (조금 더 효율적인 path를 찾게 해줌)
      현재 얻는 보상이 미래에 얻는 보상보다 얼마나 더 중요한지 의미하는 값 (미래와 비교한 현재 보상의 가치)
    • \(T\): horizon
  • MDP의 각 에피소드는 dataset에서 datapoint(x, y)를 샘플링하는 것으로 시작하여 현재 time step t가 horizon T를 초과하거나 문장 종료(EOS) 토큰이 생성될 때 끝난다.

3.2 REWARD FUNCTIONS AND EVALUATION METRICS

[REWARD]

• RL4LM은 각각의 token별 또는 각각의 sequence generation rewards을 위한 일반적인 인터페이스를 제공한다.
• 구체적으로, ROUGE(Lin, 2004), BLEU(Papineni et al., 2002), SacreBLEU(Post, 2018), METERE(Banerjee & Lavie, 2005), BLEURT(Sellam et al., 2020)와 같은 n-gram 중첩 메트릭에 대한 인터페이스를 제공
  ➡ 유사하게 다양한 범위의 task-specific metrics에 광범위한 RL algorithms을 신속하게 적용할 수 있다.

[EVALUATION METRICS]
복잡성/유효성/자연성 지표(diversity/fluency/naturalness metrics) 존재
즉, 아래의 지표들 존재

• perplexity
• Mean Segmented Type Token Ratio (MSSTR)
• Shannon entropy over unigrams and bigrams, he ratio of distinct n-grams over the total number of n-grams (Distinct-1, Distinct-2)
• count of n-grams that appear only once in the entire generated text
• task-specific, model-based human preference metrics such as classifiers trained on human preference data collected in the methodology

3.3 ON-POLICY ACTOR-CRITIC ALGORITHMS

RL4LMs는 언어 환경에 대한 on-policy actor-critic algorithms을 통해 LM을 처음부터 fine-tuning하고 훈련할 수 있도록 지원한다.

공식적으로, 이 algorithms 클래스는 아래 궤적에 대해 long term discounted rewards을 maximize하기 위해 주어진 상태에서 action을 선택하려고 시도하는 함수인 \(π_θ:S → A\)로 정의된 parameterized control policy을 훈련할 수 있게 한다.

우리의 벤치마크 실험은 agent’s policy \(π_θ = π_0\)을 초기화하는 ”\(π_0\)“으로 표시된 pre-trained LM 을 fine-tuning하는 데 중점을 둔다.

마찬가지로, value function를 추정하는 데 사용되는 value network \(V_φ\) 도 \(π_0\)부터 초기화된다.
except) 단일 스칼라 값을 출력하기 위해 임의로 초기화되는 최종 계층은 초기화 제외

다른 deep RL actor-critic algorithms과 마찬가지로, 우리는 우리의 value(\(V^π_t\))와 Q-value(\(Q^π_t\)) 함수를 아래와 같이 정의할 수 있다.

위의 식은 아래의 advantage function을 유도할 수 있다.
(이게 이해가 안된다…? 그럼 강화학습 처음부터 다시 익히고 오자,,,)

➡ training stability를 높이기 위해선 Generalized Advantage Estimation를 이용하여 advantage을 근사화한다.

agent의 input-output pair(x, y)과 generation 예측을 고려할 때,
environment rewards은 sequence-level이고 희소하다.

그러므로, 모든 on-policy algorithms에 대해 token-level KL penalty를 사용해 reward 기능을 정규화하여,
모델이 초기화된 LM ≥ 0에서 너무 멀리 벗어나지 않도록 한다.

형식적으로 정규화된 reward 함수는 아래와 같다:

• \(R hat\): 정규화된 KL reward
• \(y\): gold-truth predictions
• \(KL(π_θ(a_t \|s_t) \| π_0(a_t \|s_t)) = (log π_0(a_t \|s_t) − log π_θ(a_t \|s_t))\)을 만족한다.
• KL coefficient β는 동적으로 적용(dynamically) \

4 NLPO: NATURAL LANGUAGE POLICY OPTIMIZATION

언어 생성 작업 공간은 대부분의 이산 작업 공간 RL 알고리듬이 설계된 것보다 훨씬 큰 규모이다(Ranzato et al., 2016; Ammanabrolu, 2021). 예를 들어 GPT-2/3과 T5의 어휘 크기는 각각 50K와 32K이다. 우리는 기존 RL 방법으로 LM을 훈련할 때 동작 공간의 크기가 불안정의 핵심 원인이라고 가정한다. 이 문제를 해결하기 위해 조치 제거/무효 조치 마스킹에 대한 작업에서 영감을 얻은 NLPO(자연어 정책 최적화)를 소개한다(Zahavy et al., 2018; Huang & Ontánón, 2020; Ammanabrolu & Hausknecht, 2020). PPO의 매개 변수화된 마스킹 확장인 NLPO는 훈련할 때 컨텍스트에서 덜 관련된 토큰을 마스킹하는 방법을 학습한다. NLPO는 top-p 샘플링을 통해 이를 달성하는데, 이는 토큰을 확률 매개 변수 p보다 누적 확률이 큰 가능한 가장 작은 세트로 제한한다(Holtzman et al., 2018). Language generation action spaces are orders of magnitude larger than what most discrete action space RL algorithms are designed for (Ranzato et al., 2016; Ammanabrolu, 2021), e.g., GPT-2/3 and T5 have a vocabulary size of 50K and 32K respectively. We hypothesize that the size of the action space is a core cause of instability when training LMs with existing RL methods. To address this issue, we introduce NLPO (Natural Language Policy Optimization), which is inspired by work on action elimination/invalid-action masking (Zahavy et al., 2018; Huang & Ontañón, 2020; Ammanabrolu & Hausknecht, 2020). NLPO, a parameterized-masked extension of PPO, learns to mask out less relevant tokens in-context as it trains. NLPO accomplishes this via top-p sampling, which restricts tokens to the smallest possible set whose cumulative probability is greater than the probability parameter p (Holtzman et al., 2018).

특히, NLPO는 마스킹 정책 »를 유지 관리합니다. 마스킹 정책은 현재 정책(»)의 복사본이지만 » 단계마다 업데이트됩니다. 매개 변수화된 유효하지 않은 마스크는 먼저 어휘 4에서 top-p 토큰을 선택한 다음 나머지 토큰에 유효하지 않은 마스크를 적용하여 생성된다. 즉, 훈련 중에 π에서 동작을 샘플링할 때 확률을 0으로 설정한다. 이 주기적 업데이트 정책 π는 정책 외 Q-러닝 알고리즘(안드리코비치)에서 영감을 받았다et al., 2017), §0에서 도출된 KL 페널티보다 더 많은 작업 관련 정보를 포함하는 것의 이점과 보상 해킹의 위험 사이의 균형을 맞추는 추가 제약 조건을 정책 »에 제공한다. 우리는 알고리즘 1에서 유사 코드를 제공한다(녹색 부분은 PPO와의 차이점을 강조한다).

Specifically, NLPO maintains a masking policy πψ: the masking policy is a copy of the current policy (πθ), but is updated only every µ steps. A parameterized-invalid-mask is created from πψ by first selecting the top-p tokens from the vocabulary, 4 and then applying an invalid-mask to the remaining tokens—i.e. setting their probabilities to zero when sampling actions from πθ during training; this periodic updating policy πψ is inspired by off-policy Q-learning algorithms (Andrychowicz et al., 2017), providing the policy πθ with an additional constraint that balances between the benefits of containing more task relevant information than the KL penalty derived from π0 and the risk of reward hacking. We provide pseudocode in Algorithm 1 (green portions highlight the differences with PPO).

GRUE (GENERAL REINFORCED-LANGUAGE UNDERSTANDING EVAL)

GRUE is a collection of 7 generative NLP tasks. To combat reward hacking for any single metric, each task is evaluated at test time according to a task-specific mix of metrics, detailed in Table 1. The metrics span two categories. Task preference metrics capture how well the models produce generations that satisfy the desiderata of the specific generation task, e.g., for Commongen, if the generations contain all the required words, or for IMDB, how positive the generated completions are. Naturalness metrics capture fluency, readability, etc. and provide perspective on factors beyond semantics. At training time, there are no special restrictions: models are free to use the supervised data, compute metrics on intermediate generations, etc. Train/val/test splits follow the original works. All results are averaged over multiple seeds, with exact counts being found in Appendix B.

Experimental Setup. We use RL4LMs to test a large range of algorithms on the GRUE benchmark. Specifically: We compare 3 algorithms for direct fine-tuning — Supervised, PPO,5 and NLPO. In addition, we consider a hybrid approach of supervised learning and our RL methods by applying PPO and NLPO on checkpoints that have been fine-tuned in a supervised fashion—we call these Supervised+PPO, Supervised+NLPO. As an additional baseline, we additionally run zero-shot evaluations where we design prompts which aim to elicit task-specific generations, but with no training data or parameter updates.

For each task, to isolate the effect of training method, we select a single pre-trained LM backbone. For IMDB text continuation we use GPT-2 (117m parameters), and for the rest of the tasks we use T5-base (220m parameters). For our RL models (PPO, NLPO, Supervised+PPO, Supervised+NLPO), for a thorough investigation of how reward-hacking might interplay with GRUE, we run a separate set of experiments optimizing multiple task rewards for each task independently, e.g., for Commongen which has 6 task rewards (CIDER, ROUGE-2, ROUGE-L, BLEU-3, BLEU-4, METEOR) we run 6 different experiments optimizing each metric independently and report all possible metrics seen in Table 1 regardless of which individual metric was being optimized for.

Human Participant Study. We gather human judgments for five of the tasks in GRUE. In doing so, our goals are 1) to validate that the automated metrics we selected for GRUE correlate with human judgments with respect to relative ranking between models; and 2) to provide additional empirical comparisons regarding NLPO vs. PPO, ablations to study the effects of the KL naturalness penalty, etc. We specifically consider IMDB, Commongen, ToTTo, DailyDialog, and CNN Daily Mail. For each individual sample in a task, we ask 3 unique human raters to provide Likert judgments of 1) quality, i.e., for the specific task, how correct/appropriate is the generation, given the context, and 2) fluency, i.e., how well-written is the generation. We used Amazon Mechanical Turk, and paid crowdworkers a minimum of $15/hr. More details, including qualification information, interface screenshots, instructions, etc. are given in the corresponding Appendicies.

5.1 RESULTS ON GRUE: WHICH ALGORITHM SHOULD BE USED TO LEARN PREFERENCES? Figures 2(a), 2(b) present the results on GRUE, split into task metrics and naturalness metrics, and Tables 2, 3 highlight key results via ablation studies. Full results are available in Appendix B. For text continuation and summarization, with non-trivial zero-shot performance, RL tends to perform better than supervised training, but for tasks like Commongen and ToTTo, which have very low zero-shot performance, supervised training performs best—with both approaches outperforming zero-shot. However, using RL+Supervised learning in conjunction works best; NLPO+supervised and PPO+supervised usually always outperforms NLPO/PPO (or supervised in isolation) across both task metrics and naturalness metrics. Supervised warm-starting is particularly effective for Commongen and ToTTo, which our results suggest are more prone to reward hacking. The one exception to this trend is DailyDialog where the RL models outperform warm-started Supervised+RL models likely

due to the low performance of the Supervised models. We note that Supervised+NLPO using a T5-base (220m parameter) LM currently outperforms all the models on the ToTTo leaderboard, many of which have ≥ 3b parameter supervised models—suggesting that RL is parameter efficient as well. In these cases, it is critical that the initial policy already contain (some) signal for the task due to it being used as a KL constraint and masking constraint in NLPO. If the mask contains no initial priors about task specific language, it will be eliminating the wrong actions—a better initial policy leads to better RL performance downstream.

Human agreement with automated metrics. As human judgments can be noisy, we run additional statistical analysis such as measuring inter-annotator agreement, via Krippendorf’s alpha score, and using a one-way ANOVA followed by a post-hoc Tukey HSD test to measure if differences in means of average scores between pairs of models are significant. We find that trends in our human evaluations generally match those seen in the automated metrics for both task and naturalness metrics (see Figures 2(c), 2(d) which summarize Appendix Tables 10,15,21,26, 35—Supervised+NLPO > Supervised ≥ Supervised+PPO > NLPO ≥ PPO > Zero-shot—with the exception of Supervised outperforming Supervised+PPO on 2 out of 5 tasks when automated metrics would indicate that Supervised+PPO outperforms Supervised on all of the tasks. We draw two conclusions from this: (1) if the generated text is above a certain threshold of naturalness, the automated metrics usually correlate with human judgements; (2) usually but not always as seen in the relative performance of Supervised and Supervised+PPO, potentially indicating reward hacking behaviors undetected by automated metrics but caught by human preference feedback.

5.2 PREFERENCE REWARD LEARNING, SELECTION, AND HACKING While the GRUE benchmark’s metric for each task is an average over several measures, the RL models we trained optimized only a single metric independently. Thus, we can empirically investigate which metric for which GRUE produces the best results. We observe that many possible single metric rewards provide task performance gains over supervised methods (results shown in Fig. 3(a), 2(c) are averaged across these reward functions) with the condition that the text is also coherent and natural.

Which constraints best prevent reward hacking? The reward function in Equation 1 balances a task-specific reward with a KL constraint — models are penalized from straying too far from a base LM in their pursuit of high reward (Table 3 and Appendix Table 5) clearly show that if KL constraints are removed entirely, models reward hack). But which model works best as a base regularizing LM? When the initial policy (i.e., the raw, pretrained model) has low performance on the task, the KL penalty pushes the policy towards nonsense, e.g. on Commongen and ToTTo the trained policy learns to simply repeat portions of the input (as seen in Tables B.4.5, B.6.4). This behavior is mitigated if the base regularizing LM is the supervised model—the reward encourages the policy to balance the task-specific reward and a more reasonable regularization term. Deriving KL penalties from warm-started initial policies is critical for performance on such tasks.

PPO vs. NLPO. Figure 2 shows that NLPO generally outperforms PPO and supervised, especially when applied after supervised training. We hypothesize that the primary reason for NLPO’s improved performance and stability is because the masking policy provides an additional constraint for the current policy. This constraint is not based on the initial untuned policy like the KL penalty but of the policy from µ iterations ago and likely contains more task-relevant information learned during RL training. Table 3 (and Appendix Table 8) shows how performance increases up to a point and then decreases as p in top-p sampling is increased for the masking policy, relaxing the constraint by eliminating less tokens at each step, implying that there is a balance to be found in how much the model should be constrained during RL training.

Human Preference Reward Learning. To this point, our experiments have largely focused on optimizing evaluation metrics that correlate with human judgments, e.g., METEOR. Here: we additionally test how well preferences can be learned from direct human feedback. For this, we focus on Commongen — a GRUE dataset well-suited for displaying differences due to human preferences. First, we randomly select prompts from the Commongen train dataset and sample a single completion from both the Supervised and Supervised+NLPO models. We then present the prompt and the two completion candidates to 3 unique crowdworkers and ask them to select which one they prefer with respect to commonsense/fluency for 417 unique pairs (Krippendorf α = .28). We use this data to train a reward model, T5-11B Raffel et al. (2020), on the balanced binary classification task of predicting which of the pair was preferred by a majority of 3 annotators, conditioned on the prompt and completion. The resulting model achieved 69.5 test ROC AUC suggesting it indeed captures average human preferences. Additional details on this process are found in Appendix B.4.4. We train Supervised+RL with a METEOR-only reward as a baseline, and compare it to a reward function that uses the fine-tuned T5-11B model. Finally, we rerun the same pairwise preference collection procedure—this time sampling from Commongen test—with human participants to compare the generations from a preference optimized RL policy to the previously best Supervised+NLPO policy. Comparing the METEOR-only to the preference model, the generations produced by the human feedback model are preferred in 682 cases, compared to the METEOR-only model which is preferred in 587 cases (p < 0.01 the models are equally preferred). This implies that this pipeline of collecting preferences, training a reward, and further tuning the policy improves alignment to human preferences.

5.3 DATA BUDGET: IMPROVE YOUR REWARD OR GATHER MORE DEMONSTRATION? Given a fixed data collection budget, is it more efficient to gather feedback to improve a learned reward function or to gather more expert demonstrations? We use the IMDB text continuation task as a case study. In the IMDB task, a model is given a partial movie review as a prompt, and is asked to continue it as positively as possible (even if the prompt was negative). The original dataset consists of movie reviews and sentiment labels of positive, negative, or neutral. A DistilBERT (Sanh et al., 2019) classifier is trained on these labels and used to provide sentiment scores on how positive a given piece of text is, which serves as the task reward. The trade-off is between gathering more: 1) sentiment labels (improving the reward); or 2) positive sentiment reviews (improving supervised training). We train a classifier on varying amounts of training data and evaluate on the held out test dataset— finding as expected that more training data improves test accuracy and so results in a higher quality reward. We then use each of these rewards of varying quality during RL training, and evaluate using the same metric as GRUE (i.e., a classifier trained with the entire training set). As seen in Table 3, we find that improving the reward quality improves LM performance as well. Further, we trained a supervised model with at least as many samples used to train each of these reward classifiers. We find that a learned reward function enables greater performance when used as a signal for an RL method than a supervised method trained with 5 times more data. This implies that improving reward models can be more data efficient than collection expert demonstrations for a task—and that’s not accounting for the fact that assigning sentiment labels is likely a simpler task than writing full demonstrations. Further details on this ablation are found in Appendix Table 7.

5.4 PRACTICAL CONSIDERATIONS: WHICH IMPLEMENTATION DETAILS MATTER MOST? Generation as a token-level MDP, not a bandit environment. Most recent works that tune LMs using RL do so by calculating a reward for all the tokens in the sentence (Wu et al., 2021; Ouyang et al., 2022; Lu et al., 2022). This setting is equivalent to a bandit feedback environment where the action space is the space of all possible generations for the task (Sutton & Barto, 2018). This type of environment can be simulated within our RL formulation by setting the discount factor γ = 1. Table 3 (and Appendix Table 6) shows that this causes instability in training with respect to naturalness in both PPO and NLPO for IMDB. Our standard setting is γ = 0.95 when calculating discounted rewards-to-go in the token-level MDP formulation, which reduces the magnitude of the reward that is applied to tokens selected at the beginning. The sentiment scores are approximately the same between both settings but the naturalness of language in the bandit setting is significantly less —indicating that discounting rewards with γ < 1 via a token-level MDP formulation is at least sometimes more effective for language generation.

Dropout and Sampling. We found two other implementation details to be critical for stability of RL training. The first is dropout, which in its standard form was found to cause instability in policy gradient methods in continuous control settings by Hausknecht & Wagener (2022). We find a similar effect when using dropout when RL training LMs as well, with training loss often diverging for dropout > 0 in training. The second important detail, particularly affecting the machine translation task, is sampling methods. We find that using the same sampling methods during exploration and inference is critical to translating training performance to test performance–else the model exhibits high train rewards but low test metrics.

6 CONCLUSIONS We’re hopeful that the GRUE benchmark and the RL4LMs library can push progress in aligning language models to human preferences via RL methods by providing the community with a standard means of comparing methods. Furthermore, we’re optimistic that, as the stability and consistency of training improves, our methods provide a path towards iterative improvement of language technologies, with deployment, user feedback collection, and re-optimization enabling better user experiences when interacting with generative models.