Hindsight Experience Replay Improves Reinforcement Learning for Control of a MIMO Musculoskeletal Model of the Human Arm

IEEE Trans Neural Syst Rehabil Eng, 2021 · DOI: 10.1109/TNSRE.2021.3081056 · Published: January 1, 2021

Simple Explanation

This paper explores the use of reinforcement learning to control a computer model of a human arm, specifically for individuals with spinal cord injuries. The study demonstrates that a technique called 'hindsight experience replay' can improve the performance of the control system while also decreasing the training time required. The controller is designed to move the arm to different target locations based on the desired final position, using information about the arm's movement, but without detailed knowledge of the internal state of the muscles.

Study Duration

Not specified

Participants

Not specified

Evidence Level

Not specified

Key Findings

1
Controllers trained using HER are more likely to learn to control the arm model within the specified training time.
2
They learn more quickly, and they produce better control than reinforcement learning algorithms that do not use HER.
3
Controllers trained using HER approach able-bodied performance for 7.5 cm targets in terms of the number of targets successfully acquired and the time required to acquire targets.

Research Summary

The paper introduces an FES controller that works across a wide repertoire of movements by using methods of training that are quick and automatic. The controller takes, as inputs, the goal state of the movement rather than the (possibly suboptimal) movement trajectory and only relies on easily-observable states, such as joint angle kinematics. The controller includes an improvement to the actor-critic reinforcement learning technique called hind-sight experience replay (HER) that was previously developed for controlling robots.

Practical Implications

Improved FES Controller Design

Hindsight experience replay (HER) can improve controller performance and decrease training time for FES systems.

Potential for Patient-Specific Customization

The pure reinforcement learning approach may generalize better when systems include more complex links between degrees of freedom and actuators.

Real-World Application Considerations

Before controller training, each electrode should be profiled to set safe thresholds for stimulation, as is common practice [2].

Study Limitations

1
A simplified model of the human arm was used, which had fewer mechanical degrees of freedom and fewer actuators than a human arm.
2
The model also did not include gravity, muscle fatigue, or time-varying dynamics such as those caused by muscle spasticity.
3
The hyperparameters that we chose were based on previous work, and then manually-tuned.

Your Feedback

Was this summary helpful?

Back to Neurology