Improving the Learning Rate, Accuracy, and Workspace of Reinforcement Learning Controllers for a Musculoskeletal Model of the Human Arm

IEEE Trans Neural Syst Rehabil Eng, 2022 · DOI: 10.1109/TNSRE.2021.3135471 · Published: February 15, 2022

Simple Explanation

This study explores ways to improve how quickly and accurately computer programs can learn to control a virtual model of a human arm using electrical stimulation. The goal is to restore movement in people with paralysis. The researchers used methods called transfer learning and curriculum learning to help the computer programs learn more effectively. These methods involve training the programs on simpler tasks first, or using knowledge gained from controlling other similar arm models. The results showed that these techniques can significantly improve the learning speed, accuracy, and range of motion achieved by the computer programs, bringing us closer to more effective FES controllers.

Study Duration

120 minutes of simulated time

Participants

Musculoskeletal model of the human arm

Evidence Level

Not specified

Key Findings

1
Curriculum learning improved the number of small targets acquired by as much as 50%.
2
Pretraining alone did not improve the learning rate or accuracy of trained controllers.
3
Transfer learning worked even when controllers were transferred between models with radically different parameters.

Research Summary

This study investigates methods to enhance reinforcement learning (RL) controllers for functional electrical stimulation (FES) of a human arm model, focusing on improving learning rates, accuracies, and workspaces. The researchers explored curriculum learning (gradually decreasing target size), pretraining (using existing data to initialize the controller), and transfer learning (using a controller trained on one arm model to control another). The results indicate that curriculum learning and transfer learning can significantly improve controller performance, with transfer learning showing promise for reducing the need for patient-specific training data.

Practical Implications

Improved FES Controller Training

The findings suggest more efficient methods for training FES controllers, potentially reducing the time and resources needed for patient-specific customization.

Enhanced Workspace and Accuracy

The techniques explored can lead to FES controllers with a greater range of motion and improved accuracy in reaching targets, increasing the potential for functional restoration.

Facilitated Transferability

Transfer learning enables the use of pre-trained controllers across individuals with varying anatomical parameters, reducing the need for extensive retraining for each patient.

Study Limitations

1
The simplified model of the human arm that we used did not consider gravity or time-varying dynamics (e.g. muscle spasticity).
2
Preliminary studies (Supplementary Figure 6) suggest that trained controllers are robust to high levels of fatigue that decrease muscle strengths by up to 90%.
3
Future studies should consider how fatigue may impact the training process since time-varying systems are difficult to learn to control.

Your Feedback

Was this summary helpful?

Back to Neurology