RE: LeoThread 2025-07-13 23:00

Fundamentally, reinforcement learning adjusts the likelihood of actions based on whether an outcome was favorable or not.

This approach offers significant benefits compared to explicit supervision, especially when leveraging verifier functions.

However, a potential limitation becomes evident for longer tasks that involve minutes or hours of interaction, where substantial effort is spent to derive a single scalar outcome to guide the gradient.

Moreover, this mechanism does not entirely mirror the human method of learning for most intelligence tasks, which typically involves a reflective review phase.

In such cases, the process includes evaluating what worked, what did not, and how to improve next time—generating explicit lessons akin to adding a new instruction to the system prompt, sometimes later integrated into the underlying model

capacities. For instance, the recent memory feature is an early approach within some models, though it primarily supports customization rather than robust problem-solving.

This reflective aspect is missing in domains like Atari reinforcement learning, which lack large language models and in-context learning.

An illustrative algorithm involves executing several rollouts for a given task, compiling them with their respective rewards, and using a meta-prompt to review the successes and failures.

This process produces a “lesson” that can be incorporated into a system prompt or broader lessons database.

One example of such a lesson addresses the difficulty language models have with tasks like counting letters due to tokenization issues.

An early fix involved explicitly instructing the model to list letters with commas and count them individually.

The challenge now lies in making these lessons emerge dynamically from the learning process rather than being manually engineered, and in distilling these lessons over time without unnecessarily bloating the context.

In summary, while reinforcement learning promises greater gains through a leveraged and economically effective approach, it does not seem to fully capture all aspects of learning—especially as tasks grow longer.

There may be additional improvement curves to explore that are specific to large language models, potentially opening up exciting new avenues beyond traditional game or robotics environments.