![\"\"](\"http:\/\/haleyai.com\/wordpress\/wp-content\/uploads\/StickFigureDiagram.png\" "\\"StickFigureDiagram\\"")
<\/a><\/p>\nTherapists could also enter left-to-right balance and front-to-back weight bearing for each foot.\u00a0 In the anterior view you can see that pronation or supination could also be input.\u00a0 In the superior view, you can see that inverted or everted feet and knees were captured.\u00a0\u00a0 In the lateral views you can also input shoulder protraction.\u00a0 We\u2019ll skip over how spinal curvatures was considered here.<\/p>\n
Today we use vision and a balance board and\/or a foot pressure mat.\u00a0 First let\u2019s go over the basics and then dive into the vision\u2026<\/p>\n
The following shows the estimated locations of a person\u2019s joints from three of the four perspectives diagrammed above.\u00a0 Practically speaking, this is the same information rendered a different way, but it\u2019s a lot easier to capture and the subjectivity is gone.\u00a0 Polaroids are long gone, but the vision system also eliminates loading of files from digital cameras.<\/p>\n
Getting the skeleton to be accurate enough for clinical purposes was not trivial.\u00a0 The camera has to be carefully set up on a tripod with the right pitch and no roll, for example.\u00a0 Easy said than done across dozens of clinics!\u00a0 The software helps, though.<\/p>\n The Microsoft software estimates joint locations by matching a random forest of decision trees against the depth image it perceives.\u00a0 Rather than going into the details, let me just say that the Kinect was effectively trained to locate joints better in two dimensions than three.\u00a0 And the approach taken will never be accurate with baggy clothing.<\/p>\n The Kinect is a reliable device, but it is also cumbersome and power hungry.\u00a0 More importantly, from the standpoint of perception, it does not have the resolution or accuracy of devices such as Intel\u2019s RealSense camera which is cheaper and doesn\u2019t require a power adapter.\u00a0 Also, Microsoft is no longer manufacturing the Kinect.<\/p>\n To use the RealSense we had to find or write our own skeleton tracking \/ joint position estimation software.\u00a0 After an extensive review of the literature and evaluation of many pieces of software, I decided to write it as described further below.\u00a0 First, a few comments.\u00a0 Convolutional neural nets (CNN) do a marvelous job in 2 dimensions using only image data (no depth).\u00a0 We will integrate their estimates with the approach and work on 3-dimensional versions that incorporate depth information, whether from the RealSense or stereo cameras without depth (the RealSense is a stereo camera that uses an infrared laser to \u201chelp\u201d).<\/p>\n Skipping over a lot of research, we decided to focus on two things:\u00a0 a simple geometric model of an articulating body and a much more detailed polygonal mesh covering a model of the body.\u00a0 The geometric model consists of spheres and sections of cones or cylinders that articulate at joints.\u00a0 The mesh model has thousands of polygons each with a surface normal that is oriented according to a linear model.\u00a0 The linear equation is based on principal components of how the mesh best fits a number of different scans of people in various but mostly standing poses.\u00a0 The geometric model is astonishingly fast and useful for getting pose information and joint positions.\u00a0 The latter model gives more accurate joint positions (not to mention body parameters).<\/p>\n The following picture may give you an idea of how such models \u201cfits\u201d various body types:<\/p>\n Each of these articulating approaches performs more robustly than the Microsoft approach for loosely clothed bodies.<\/p>\n The articulating geometric model is too easy to go into, so let\u2019s talk a little more about the body mesh approach.<\/p>\n There are a few ways to go about this.\u00a0 You can obtain high resolution scans of people from various places or you can obtain your own using Kinect Fusion.\u00a0 Next, you need a body mesh with appropriate details where you care to have them.\u00a0 You can take your point cloud data and get a variety of such meshes or you can use a mesh such as you might find in animation software.\u00a0 Then you fit the mesh to each of your scans.\u00a0 Then you compute at least a dozen to perhaps a hundred principal components.<\/p>\n Once you have the principal components you have an optimization problem.\u00a0 The basic idea is to find the body model parameters that best fit the depth map you obtain from the Kinect or RealSense or other stereoscopic camera setup.<\/p>\n There are many details to be considered, but the first is pose.\u00a0 For example, if you can assume the person is standing facing the camera, it\u2019s a lot easier to adjust the parameters to the data.\u00a0 Pose includes joint angles and bone lengths, which are most of the parameters in an articulating geometric model.\u00a0 So, the first thing you can do is use the simple model to estimate those parameters or fit the body mesh while only optimizing on the pose parameters (holding the body type parameters constant).<\/p>\n Once you have the articulating geometry fit, you can continue to estimate the body model and, if you want the best results, repeat until changes to the parameters quiesce (among other cutoff criteria).\u00a0 The estimation itself seeks to minimize error.\u00a0 One way of characterizing the error minimization problem is least mean squared error where the error is the difference between the depth map perceived and the surface of the body model represented by the mesh.\u00a0 Each iteration seeks to reduce the error by adjusting the parameters.\u00a0 This is typically done using some form of gradient descent.\u00a0 We use a popular, open-source machine learning package to do this job.<\/p>\n Here’s an example of how the approach can perform, although we do not take exactly the same approach:<\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n