Abstract
Exoskeletons have enormous potential to improve human locomotive performance1,2,3. However, their development and broad dissemination are limited by the requirement for lengthy human tests and handcrafted control laws2. Here we show an experiment-free method to learn a versatile control policy in simulation. Our learning-in-simulation framework leverages dynamics-aware musculoskeletal and exoskeleton models and data-driven reinforcement learning to bridge the gap between simulation and reality without human experiments. The learned controller is deployed on a custom hip exoskeleton that automatically generates assistance across different activities with reduced metabolic rates by 24.3%, 13.1% and 15.4% for walking, running and stair climbing, respectively. Our framework may offer a generalizable and scalable strategy for the rapid development and widespread adoption of a variety of assistive robots for both able-bodied and mobility-impaired individuals.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 51 print issues and online access
$199.00 per year
only $3.90 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
All data supporting the findings of this study are available in the Article and its Supplementary Information. Source data are provided with this paper.
Code availability
Pseudocode for the learning-in-simulation algorithm and training process can be found in the GitHub repository https://github.com/IntelligentRobotLearning/pseudocode_learning_in_simulation.
References
Siviy, C. et al. Opportunities and challenges in the development of exoskeletons for locomotor assistance. Nat. Biomed. Eng. 7, 456–472 (2023).
Slade, P., Kochenderfer, M. J., Delp, S. L. & Collins, S. H. Personalizing exoskeleton assistance while walking in the real world. Nature 610, 277–282 (2022).
Awad, L. N. et al. A soft robotic exosuit improves walking in patients after stroke. Sci. Transl. Med. 9, eaai9084 (2017).
Geyer, H., Seyfarth, A. & Blickhan, R. Compliant leg behaviour explains basic dynamics of walking and running. Proc. R. Soc. B 273, 2861–2867 (2006).
Zhang, J. et al. Human-in-the-loop optimization of exoskeleton assistance during walking. Science 356, 1280–1284 (2017).
Ferris, D., Sawicki, G. & Domingo, A. Powered lower limb orthoses for gait rehabilitation. Top. Spinal Cord Inj. Rehabil. 11, 34–49 (2005).
Veneman, J. F. et al. Design and evaluation of the LOPES exoskeleton robot for interactive gait rehabilitation. IEEE Trans. Neural Syst. Rehabil. Eng. 15, 379–386 (2007).
Ding, Y., Kim, M., Kuindersma, S. & Walsh, C. J. Human-in-the-loop optimization of hip assistance with a soft exosuit during walking. Sci. Robot. 3, eaar5438 (2018).
Koller, J. R., Jacobs, D. A., Ferris, D. P. & Remy, C. D. Learning to walk with an adaptive gain proportional myoelectric controller for a robotic ankle exoskeleton. J. Neuroeng. Rehabil. 12, 97 (2015).
Zhang, Q. et al. Imposing healthy hip motion pattern and range by exoskeleton control for individualized assistance. IEEE Robot. Autom. Lett. 7, 11126–11133 (2022).
Dembia, C. L., Bianco, N. A., Falisse, A., Hicks, J. L. & Delp, S. L. Opensim moco: musculoskeletal optimal control. PLoS Comput. Biol. 16, e1008493 (2020).
Song, S. et al. Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation. J. Neuroeng. Rehabil. 18, 126 (2021).
Gordon, D. F., McGreavy, C., Christou, A. & Vijayakumar, S. Human-in-the-loop optimization of exoskeleton assistance via online simulation of metabolic cost. IEEE Trans. Robot. 38, 1410–1429 (2022).
Durandau, G., Rampeltshammer, W. F., van der Kooij, H. & Sartori, M. Neuromechanical model-based adaptive control of bilateral ankle exoskeletons: biological joint torque and electromyogram reduction across walking conditions. IEEE Trans. Robot. 38, 1380–1394 (2022).
Lim, B. et al. Delayed output feedback control for gait assistance with a robotic hip exoskeleton. IEEE Trans. Robot. 35, 1055–1062 (2019).
Meuleman, J., Van Asseldonk, E., Van Oort, G., Rietman, H. & Van Der Kooij, H. LOPES II—design and evaluation of an admittance controlled gait training robot with shadow-leg approach. IEEE Trans. Neural Syst. Rehabil. Eng. 24, 352–363 (2015).
Molinaro, D. D., Kang, I., Camargo, J., Gombolay, M. C. & Young, A. J. Subject-independent, biological hip moment estimation during multimodal overground ambulation using deep learning. IEEE Trans. Med. Robot. Bionics 4, 219–229 (2022).
Shepherd, M. K., Molinaro, D. D., Sawicki, G. S. & Young, A. J. Deep learning enables exoboot control to augment variable-speed walking. IEEE Robot. Autom. Lett. 7, 3571–3577 (2022).
Bellegarda, G. & Ijspeert, A. CPG-RL: learning central pattern generators for quadruped locomotion. IEEE Robot. Autom. Lett. 7, 12547–12554 (2022).
Li, Z. et al. Reinforcement learning for robust parameterized locomotion control of bipedal robots. In 2021 IEEE International Conference on Robotics and Automation (eds Meng, Q.-H. & Sun, Y.) 2811–2817 (IEEE, 2021).
Siekmann, J., Godse, Y., Fern, A. & Hurst, J. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In IEEE International Conference on Robotics and Automation (eds Meng, Q.-H. & Sun, Y.) 7309–7315 (IEEE, 2021).
Wen, Y., Si, J., Brandt, A., Gao, X. & Huang, H. H. Online reinforcement learning control for the personalization of a robotic knee prosthesis. IEEE Trans. Cybernet. 50, 2346–2356 (2019).
Luo, S. et al. Reinforcement learning and control of a lower extremity exoskeleton for squat assistance. Front. Robot. AI 8, 702845 (2021).
Luo, S. et al. Robust walking control of a lower limb rehabilitation exoskeleton coupled with a musculoskeletal model via deep reinforcement learning. J. Neuroeng. Rehabil. 20, 34 (2023).
CMU Graphics Lab Motion Capture Database. http://mocap.cs.cmu.edu/ (Carnegie Mellon University; accessed 10 November 2021).
Adadi, A. A survey on data‐efficient algorithms in big data era. J. Big Data 8, 24 (2021).
Salvato, E., Fenu, G., Medvet, E. & Pellegrino, F. A. Crossing the reality gap: a survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access 9, 153171–153187 (2021).
Kim, J. et al. Autonomous and portable soft exosuit for hip extension assistance with online walking and running detection algorithm. In 2018 IEEE International Conference on Robotics and Automation (eds Zelinksy, A. & Park, F.) 5473–5480 (IEEE, 2018).
Zhang, X. et al. Enhancing gait assistance control robustness of a hip exosuit by means of machine learning. IEEE Robot. Autom. Lett. 7, 7566–7573 (2022).
Cao, W., Chen, C., Hu, H., Fang, K. & Wu, X. Effect of hip assistance modes on metabolic cost of walking with a soft exoskeleton. IEEE Trans. Autom. Sci. Eng. 18, 426–436 (2020).
Kim, J. et al. Reducing the energy cost of walking with low assistance levels through optimized hip flexion assistance from a soft exosuit. Sci. Rep. 12, 11004 (2022).
Seo, K., Lee, J. & Park, Y. J. Autonomous hip exoskeleton saves metabolic cost of walking uphill. In International Conference on Rehabilitation Robotics (ed. Masia, L.) 246–251 (IEEE, 2017).
Nasiri, R., Ahmadi, A. & Ahmadabadi, M. N. Reducing the energy cost of human running using an unpowered exoskeleton. IEEE Trans. Neural Syst. Rehabil. Eng. 26, 2026–2032 (2018).
Ning, C., Li, Y., Feng, K., Gong, Z. & Zhang, T. Soochow Exo: a lightweight hip exoskeleton driven by series elastic actuator with active-type continuously variable transmission. In IEEE International Conference on Mechatronics and Automation (eds Hirata, H., Xu, W. & Guo, J.) 1432–1437 (IEEE, 2022).
Kim, D. S. et al. A wearable hip-assist robot reduces the cardiopulmonary metabolic energy expenditure during stair ascent in elderly adults: a pilot cross-sectional study. BMC Geriatr. 18, 230 (2018).
Kim, J. et al. Reducing the metabolic rate of walking and running with a versatile, portable exosuit. Science 365, 668–672 (2019).
Sawicki, G. S., Beck, O. N., Kang, I. & Young, A. J. The exoskeleton expansion: improving walking and running economy. J. Neuroeng. Rehabil. 17, 25 (2020).
Koos, S., Mouret, J.-B. & Doncieux, S. The transferability approach: crossing the reality gap in evolutionary robotics. IEEE Trans. Evol. Comput. 17, 122–145 (2012).
Halilaj, E. et al. Machine learning in human movement biomechanics: best practices, common pitfalls and new opportunities. J. Biomech. 81, 1–11 (2018).
Lee, G. et al. Reducing the metabolic cost of running with a tethered soft exosuit. Sci. Robot. 2, eaan6708 (2017).
Lee, D., Kwak, E. C., McLain, B. J., Kang, I. & Young, A. J. Effects of assistance during early stance phase using a robotic knee orthosis on energetics, muscle activity and joint mechanics during incline and decline walking. IEEE Trans. Neural Syst. Rehabil. Eng. 28, 914–923 (2020).
Franks, P. W. et al. Comparing optimized exoskeleton assistance of the hip, knee and ankle in single and multi-joint configurations. Wearable Technol. 2, e16 (2021).
Nuckols, R. W. et al. Individualization of exosuit assistance based on measured muscle dynamics during versatile walking. Sci. Robot. 6, eabj1362 (2021).
Shepertycky, M., Burton, S., Dickson, A., Liu, Y.-F. & Li, Q. Removing energy with an exoskeleton reduces the metabolic cost of walking. Science 372, 957–960 (2021).
Mooney, L. M. & Herr, H. M. Biomechanical walking mechanisms underlying the metabolic reduction caused by an autonomous exoskeleton. J. Neuroeng. Rehabil. 13, 4 (2016).
Seo, K., Lee, J., Lee, Y., Ha, T. & Shim, Y. Fully autonomous hip exoskeleton saves metabolic cost of walking. In 2016 IEEE International Conference on Robotics and Automation (eds Kragic, D., Bicchi, A. & De Luca, A.) 4628–4635 (IEEE, 2016).
Seth, A. et al. OpenSim: simulating musculoskeletal dynamics and neuromuscular control to study human and animal movement. PLoS Comput. Biol. 14, e1006223 (2018).
Blake, O. M. & Wakeling, J. M. Estimating changes in metabolic power from EMG. Springerplus 2, 229 (2013).
Bruns, R. E., Vos, P. & Wedge, R. D. Electromyography as a surrogate for estimating metabolic energy expenditure during locomotion. Med. Eng. Phys. 109, 103899 (2022).
Silder, A., Besier, T. & Delp, S. L. Predicting the metabolic cost of incline walking from muscle activity and walking mechanics. J. Biomech. 45, 1842–1849 (2012).
Jackson, R. W. & Collins, S. H. An experimental comparison of the relative benefits of work and torque assistance in ankle exoskeletons. J. Appl. Physiol. 119, 541–557 (2015).
Hortobágyi, T., Finch, A., Solnik, S., Rider, P. & DeVita, P. Association between muscle activation and metabolic cost of walking in young and old adults. J. Gerontol. A 66, 541–547 (2011).
Lee, S., Park, M., Lee, K. & Lee, J. Scalable muscle-actuated human simulation and control. ACM Trans. Graph. 38, 73 (2019).
Thelen, D. G. Adjustment of muscle mechanics model parameters to simulate dynamic contractions in older adults. J. Biomech. Eng. 125, 70–77 (2003).
Zhou, X. & Zheng, L. Model-based comparison of passive and active assistance designs in an occupational upper limb exoskeleton for overhead lifting. IISE Trans. Occup. Ergon. Hum. Factors 9, 167–185 (2021).
Hwangbo, J. et al. Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4, eaau5872 (2019).
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V. & Hutter, M. Learning quadrupedal locomotion over challenging terrain. Sci. Robot. 5, eabc5986 (2020).
Tan, J. et al. Sim-to-real: learning agile locomotion for quadruped robots. Preprint at https://doi.org/10.48550/arXiv.1804.10332 (2018).
Tobin, J. et al. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (eds Zhang, H. & Vaughan, R.) 23–30 (IEEE, 2017).
Yu, S. et al. Quasi-direct drive actuation for a lightweight hip exoskeleton with high backdrivability and high bandwidth. IEEE ASME Trans. Mechatron. 25, 1794–1802 (2020).
Huang, T.-H. et al. Modeling and stiffness-based continuous torque control of lightweight quasi-direct-drive knee exoskeletons for versatile walking assistance. IEEE Trans. Robot. 38, 1442–1459 (2022).
Acknowledgements
We thank Y. K. Chen at the Massachusetts Institute of Technology for constructive feedback and discussion of this work. This work was supported in part by the National Science Foundation (NSF) CAREER award CMMI 1944655, National Institute on Disability, Independent Living and Rehabilitation Research (NIDILRR) DRRP 90DPGE0019, Switzer Research Distinguished Fellow (SFGE22000372), NSF Future of Work 2231419 and National Institute of Health (NIH) 1R01EB035404.
Author information
Authors and Affiliations
Contributions
H.S. first proposed the research idea and approach of the paper and provided the project guidance with input from E.J.R., B.Z. and X.Z. S.L., M.J. and H.S. were responsible for the design of this experiment. The methodology, code implementation and development were conducted by S.L. on the basis of a framework created by S.L. and X.Z. The mechatronics design and fabrication of the exoskeleton were done by I.D.S., M.J., S.Z. and J.Z. The experiments and data analysis were conducted by S.L., M.J., S.Z. and J.Z. The first draft of the manuscript was prepared by S.L., M.J., J.Z., S.Y., I.D.S., T.W. and H.S. Important revisions were made by S.L., J.Z., T.W., E.J.R., B.Z., H.Y., X.Z. and H.S. to the final paper. All authors contributed to the article and approved the submitted version.
Corresponding author
Ethics declarations
Competing interests
S.L. and H.S. are co-inventors on intellectual property related to the controller discussed in this work. H.S. is a co-founder of and has a financial interest in Picasso Intelligence, LLC. The terms of this arrangement have been reviewed and approved by NC State University in accordance with its policy on objectivity in research. The remaining authors declare no competing interests.
Peer review
Peer review information
Nature thanks Joonho Lee and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data figures and tables
Extended Data Fig. 1 Overview of the learning-in-simulation control framework.
a,b, Schematic illustrations for the learning-in-simulation architecture (a) and the control structure for online deployment (b).
Extended Data Fig. 2 Motion imitation neural network for versatile activities.
Schematic illustrations for the autonomous learning framework of the reference motions (walking, running, stair climbing) from datasets based on human kinematics input and joint torque command output.
Extended Data Fig. 3 Muscle coordination neural network.
Schematic illustrations for the muscle coordination neural network based on human joint torque input and human muscle actuation output.
Extended Data Fig. 4 Exoskeleton control neural network.
Schematic illustrations for the exoskeleton control neural network based on exoskeleton state history input and joint torque command output.
Extended Data Fig. 5 Exoskeleton design.
a–c, Overall view of the whole system (a), actuator (b) and electronics (c).
Extended Data Fig. 6 Experiment protocol for metabolic rate and kinematic data collection during walking, running and stair climbing (also available on Protocol Exchange at: https://doi.org/10.21203/rs.3.pex-2632/v1)
.
Supplementary information
Supplementary Information
Supplementary Text 1–7, Methods, Figs. 1–7, Tables 1–8 and references.
Supplementary Video 1
We present a control approach that learns assistive control strategies in simulation and can be transferred to a physical wearable robot to generate continuous assistance for several locomotion activities, including walking, running and stair climbing.
Supplementary Video 2
Human response to robots is slow and controller development typically requires 24–60 min human testing and is limited to single activity (mostly walking) control only. Our method leverages dynamics-aware and data-driven simulation. It requires no human testing and can be immediately deployed to a physical exoskeleton for several activities. We only need simulation once for 8 h to learn the assistive control policy that is transferred to microcontrollers for real-time control of physical exoskeletons.
Supplementary Video 3
The learned controller in simulation is transferred to a physical exoskeleton for real-time control that immediately improves mobility. Using control policies trained in simulation, the controller is versatile to assist several locomotion modes and leads to significant metabolic expenditure savings by 24.3%, 13.1% and 15.4% during walking, running and stair climbing, respectively, compared with no exoskeleton conditions.
Supplementary Video 4
The robot learns control strategies by simultaneous training of muscle-coordination neural network and robot controller neural network. It also learns multilocomotion control by an activity imitation neural network. We bridge the sim-to-real gap by domain randomization of muscle models and robot parameters and the control policy only requires one wearable sensor per leg to control the exoskeleton.
Supplementary Video 5
The robot donning takes about 2 min and doffing takes less than 1 min. The 3.2 kg low-profile hip exoskeleton does not affect the natural range of motion and thus can assist various movements for heterogeneous able-bodied individuals.
Source data
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Luo, S., Jiang, M., Zhang, S. et al. Experiment-free exoskeleton assistance via learning in simulation. Nature 630, 353–359 (2024). https://doi.org/10.1038/s41586-024-07382-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41586-024-07382-4
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.
