Teaching a Humanoid to Walk

We wanted to try two things. First, train a full-size humanoid to walk from scratch with reinforcement learning using a reward function and a lot of simulated practice. Second, once the walking policy existed, put a basic navigation stack on top of it and see what it takes to integrate the two. This post covers both.

The embodiment is a Unitree H1^[12], a full-size bipedal humanoid. Instead of hand-coding footstep timings and joint trajectories, we trained it with reinforcement learning inside a predefined Isaac Lab environment (the stock Isaac-Velocity-Flat-H1-v0 task^[15]), where 1,024 simulated robots practice velocity tracking in parallel and PPO folds their pooled experience into one shared policy. In under two hours the policy goes from random flailing to a stable gait that follows a commanded forward speed and turn rate across flat ground.

A walker that only follows velocity commands can be told how fast to move but not where to go. So I dropped the trained policy into a separate flat-floor house and put a simple navigation stack on top. A global A* planner routes to a goal room, and a local pure-pursuit controller turns that route into the velocity commands the policy already consumes. The walking policy itself never changes. The navigation stack just feeds it velocity commands, the same input it was trained on.

The pipeline

Three layers do the work. Isaac Sim^[7] runs the GPU physics and rendering for all 1,024 robots. Isaac Lab^[8][9] defines what the robot senses, does, and is rewarded for. RSL-RL^[10] is the PPO learner whose runner actually executes the training loop, turning experience into a better policy.

The robot and the task

The H1 is a full-size bipedal humanoid with 19 actuated joints. The environment is stock, Isaac Lab’s predefined Isaac-Velocity-Flat-H1-v0 task, used as-is, with its observations, reward, and command ranges all shipped by the framework rather than written by me (the later fine-tunes are the only place I change any of it). The job is velocity tracking. Over each 20-second episode the robot is handed random commands, a forward speed and a turning rate, resampled every 10 seconds so a typical episode chases about two of them, and must move that way while staying upright. It senses its own joint angles and velocities, its base motion and the direction of gravity, its own previous action, plus the current command. It outputs target joint positions ~50 times a second (driven by PD controllers).

From joint targets to torques: the PD controller

The policy never commands motor torques directly. It outputs target joint angles, and a classic PD (proportional-derivative) controller on each joint turns those targets into the torque the motor actually applies. The network’s output is a joint position target, not a velocity. The word velocity in the task name refers to the command the robot must follow (a base linear and angular velocity), not to what the policy emits, so a position PD loop is exactly the right thing to sit underneath it. This split, a slow neural policy riding on top of a fast, dumb, stable low-level loop, is standard in learned locomotion and is a big part of why training is stable at all.

The loop runs at two rates. Physics steps at 200 Hz (a 5 ms step). The policy acts only every 4th physics step (decimation = 4), issuing a fresh set of targets at 50 Hz. Between policy updates the target is held and the PD controller keeps recomputing torque from it. For each joint:

$$\tau = K_p\cdot(q^{\text{target}} - q)\; -\; K_d\cdot\dot q$$

The proportional term $K_p\cdot(q^{\text{target}}-q)$ pulls the joint toward the commanded angle - a virtual spring - while the derivative term $-K_d\cdot\dot q$ acts like a damper that resists fast motion and kills oscillation (here $\dot q$ is the measured joint velocity. A position action implies a target joint velocity of zero, so there is no $\dot q^{\text{target}}$ term). The per-joint stiffness $K_p$ and damping $K_d$ come from the H1 actuator model (stiff, high-$K_p$ legs that must carry the body, softer arms), and the resulting torque is clipped to each motor’s effort limit before it reaches the simulator.

The reinforcement-learning loop, end to end

So far we’ve met the robot, its velocity-tracking task, and the PD loop that turns joint targets into torque. The other pieces, the thousand parallel robots, the actor-critic network, the reward, we’ve only name-dropped so far, and each gets its own section below. But how do these pieces actually turn experience into a better walker? The answer is a loop that repeats 1,000 times, and every turn of that loop looks exactly the same. Let’s walk through a single iteration as a story.

Step 1: The rollout, act, and write down what happened

An iteration begins with our current policy taking control of all 1,024 environments at once. For 24 consecutive control steps (num_steps_per_env = 24), every robot:

1. Reads its observation $s_t$, joint angles and velocities, base motion, the projected gravity vector, its own previous action, and the current velocity command.
2. Feeds $s_t$ through the actor network, which outputs the mean of a Gaussian over the 19 joint targets. Because we’re training, we sample from that Gaussian rather than taking the mean, that randomness is our exploration, the little nudges that let the robot stumble into better gaits.
3. Sends those joint targets to the PD controllers, the physics steps forward, and the environment hands back a reward $r_t$ (the big weighted sum of tracking bonuses and penalties) and the next observation.

At each step we don’t just keep the reward. We record a full tuple into a shared rollout buffer:

Notice the critic is working alongside the actor the whole time. For every observation it emits a scalar $V(s_t)$, “from this state, under the current policy, how much total reward do I expect to collect?” We’ll lean on that guess in a moment.

After 24 steps across 1,024 robots, the buffer holds $1{,}024 \times 24 = 24{,}576$ transitions. That’s our entire dataset for this iteration.

Step 2: Score the actions, returns and advantages

Now the crucial question. For each action we took, was it a good idea? Raw reward alone can’t answer that, a step might pay off handsomely three steps later, or an action might look great simply because the robot was already in an easy state. We need two quantities.

First, the discounted return, the total future reward from step $t$ onward, with later rewards worth a little less:

$$G_t = \sum_{k=0}^{\infty} \gamma^k\, r_{t+k}$$

The discount factor $\gamma = 0.99$ means a reward 100 steps away counts for roughly a third of an immediate one. It keeps the sum finite and tells the agent to prefer payoffs it can reach soon.

Second, and more important, the advantage, how much better or worse an action turned out than the critic expected. The building block is the one-step TD (temporal-difference) residual:

$$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$

Read it as, “the reward I just got, plus what I now think the future is worth, minus what I previously predicted.” If $\delta_t > 0$, things went better than the critic guessed. If negative, worse. To smooth these one-step surprises into a stable signal, PPO uses Generalized Advantage Estimation (GAE)^[2], an exponentially weighted sum of future TD residuals:

$$\hat A_t = \sum_{l=0}^{\infty} (\gamma\lambda)^l\, \delta_{t+l}$$

The extra knob $\lambda = 0.95$ (lam) trades off bias against variance. Near 0 it trusts the critic and looks only one step ahead (low variance, more bias). Near 1 it sums many real rewards (low bias, more variance). At 0.95 we’re deliberately close to the “trust the real rewards” end while still borrowing the critic’s stabilizing guess.

Two practical wrinkles hide behind those infinite sums. First, they are not actually infinite. Each rollout is only 24 steps, so GAE is computed over that finite window and bootstrapped at the end with the critic’s own value estimate, $V(s_{24})$ stands in for “everything after the window.” Second, there are two ways an episode can stop, and they are treated differently. If the robot falls, that is a true terminal state and there is no future to bootstrap. But if the episode just hits its time limit (here a fixed 20 seconds, i.e. 1,000 control steps at 50 Hz), we bootstrap with $V(s')$ anyway, because the robot did not fail, we simply stopped watching. Getting this distinction right matters. Without it, every time-out would look like a catastrophe to the learner and it would become bizarrely afraid of the clock. (This truncation-versus-termination bootstrapping correction is due to Pardo et al., 2018.^[4])

The intuition is simple. Advantage is a per-action grade. Positive means “do more of this,” negative means “do less.” That grade is exactly what the policy update needs. The same pass also hands us the critic’s training target. The empirical return we regress $V(s_t)$ toward is just $\hat A_t + V(s_t)$, the advantage plus the value we already predicted, so the critic and the actor are fed from one shared computation. And before the update those advantages are standardized to zero mean and unit variance across the batch, which keeps the gradient step well-scaled no matter the absolute size of the rewards.

Step 3: The PPO update, improve, but don’t lurch

Now we finally change the network weights. We want to make high-advantage actions more likely and low-advantage actions less likely. The naive way, crank up the probability of every good action as hard as gradient descent allows, is dangerous. One overenthusiastic step can shove the policy somewhere terrible, and since the policy generates its own data, a bad policy produces bad data and the whole thing can collapse.

PPO’s^[1] fix is to measure how much the policy has moved on each action via the probability ratio

$$\rho_t = \frac{\pi_{\theta}(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)},$$

the new policy’s probability of the action divided by the old one’s (a ratio of 1 means “no change”). Then it maximizes the clipped objective:

$$L = \mathbb{E}\Big[\min\big(\rho_t\, \hat A_t,\; \mathrm{clip}(\rho_t,\, 1-\epsilon,\, 1+\epsilon)\, \hat A_t\big)\Big]$$

with clip range $\epsilon = 0.20$ (clip_param). This $L$ is written as something to maximize. Here’s what the clip does in plain terms. For a good action ($\hat A_t > 0$), we’re happy to raise its probability, but only up to $1+\epsilon = 1.2\times$. Push past that and the objective flattens out, so there’s no gradient reward for moving further. For a bad action, symmetrically, we lower its probability but stop caring once we’ve cut it to $1-\epsilon = 0.8\times$. The min always takes the less optimistic of the two terms. So the gradient survives only where it points back toward a ratio of 1. This is not a hard trust region. Clipping flattens the objective past the bound, but nothing stops the ratio going there. Over several reuse epochs the policy can still drift. That is why the KL-adaptive learning rate below is layered on top. The result is many small, safe steps instead of one reckless jump.

Each panel is the same objective from one side. For a good action it rises with the ratio and stops at $1+\epsilon$. For a bad action it is flat until $1-\epsilon$ and then falls with no floor, which is what keeps pulling a bad action’s probability back down.

The clipped term above is only the policy piece. Written the same way, as one objective to maximize, the full per-update objective bundles three parts:

$$L^{\text{total}} = \underbrace{L^{\text{clip}}}_{\text{policy}} \;-\; c_v\, \underbrace{L^{\text{value}}}_{\text{critic}} \;+\; c_e\, \underbrace{S[\pi]}_{\text{entropy}}.$$

The first term is the clipped surrogate we just built (make good actions likelier, safely), which we push up. The second, weighted $c_v = 1.0$, is the critic’s own loss, the squared error between its prediction $V(s_t)$ and the return target $\hat A_t + V(s_t)$ from the last section. It enters with a minus sign because we want that error small, so the same update that improves the actor also sharpens the coach. The third, weighted $c_e = 0.01$, is an entropy bonus, added to reward keeping the action distribution spread out so the policy does not collapse to a single deterministic choice before it has explored. This is exactly the combined objective from the PPO paper. In code the optimizer minimizes its negation. One more parallel with the policy clip, the critic’s loss is clipped too (use_clipped_value_loss), so once its new value estimate moves more than $\pm\epsilon$ from the old prediction, the clipped term flattens that sample’s gradient. As with the policy clip, that discourages rather than hard-prevents a single wild batch from yanking the value function around.

And we don’t take one step on all this, we take many. The 24,576 transitions are shuffled and split into 4 mini-batches (num_mini_batches = 4), and we sweep over all of them 5 times (num_learning_epochs = 5). That is $5 \times 4 = 20$ gradient steps, all reusing this same rollout, with gradients clipped to a norm of 1.0 for good measure. As a nice safety net on top of all that, the learning rate is adaptive. After each update we measure the KL divergence between the old and new policy and nudge the learning rate down if we moved too far past our target of 0.01, or up if we barely budged.

Step 4: Throw it all away, and repeat

Once the update is done, we discard the entire rollout buffer. All 24,576 transitions are deleted, and the next iteration collects fresh data with the freshly updated policy.

This is what makes PPO an on-policy algorithm. It only ever learns from data generated by the current policy. The moment the weights change, yesterday’s experience is considered stale, because those actions were sampled from a policy that no longer exists, and the whole clipped-ratio machinery above only makes sense while the “old” policy in the denominator is genuinely the one that produced the data.

Contrast this with off-policy methods (think DQN or SAC), which stash millions of past transitions in a replay buffer and reuse them for many updates, even ones generated by long-obsolete policies. Off-policy learning is more sample-efficient, it squeezes more learning out of each transition, but it’s fiddlier to stabilize. On-policy PPO makes the opposite bet. Data is cheap when you have 1,024 robots on a GPU cranking out tens of thousands of transitions per iteration, so just throw it away and keep the learning simple and stable.

And that’s the whole loop. Act for 24 steps → score every action with returns and GAE advantages → nudge the policy with the clipped PPO objective → discard the data → repeat. Do that 1,000 times, about 24.6M environment steps, and a robot that started by flailing and faceplanting learns to stride across flat ground on command.

Parameter sharing: pooling multiple environments into one policy update

The loop above kept invoking 1,024 robots and one shared buffer without dwelling on the trick that makes it work^[11], a single brain learning from all of them at once. That merge deserves its own look. If you watched our training run live, you’d see something delightfully absurd, a thousand-plus Unitree H1 humanoids twitching, stumbling, and eventually striding across a grid of flat ground, all at once, all on a single GPU. It looks like a robot army. But the twist that makes reinforcement learning at this scale actually work is this. There is only one brain. Every one of those robots is being driven by the same policy (the neural network that maps what the robot senses to what it does), and every robot’s experience flows back to improve that single shared network.

Let’s unpack how that works, because it’s the engine behind why we can train a walking policy in under two hours instead of days.

A thousand copies of the same task

We run 1,024 environments (“envs”) in parallel. Each env is an independent copy of the exact same task, one H1 robot on flat ground, trying to track a commanded velocity. Isaac Sim simulates all 1,024 of them together on the GPU’s physics engine, so they step forward in lockstep, thousands of little physics worlds advancing side by side.

They are copies, but they are not clones of each other’s situation. Two things are deliberately made different across the 1,024 envs:

• Different commands. Each robot gets its own random command, a forward speed $v_x$ (0 to 1 m/s) and a random heading to face. The lateral slot $v_y$ is part of the command but held at 0 in this task, so robots differ only in how fast they walk and where they are trying to point. Robot #7 might be asked to stride forward at full speed, robot #500 to amble slowly, and robot #900 to turn in place.
• Slight randomization. Each episode the robot re-spawns within half a meter of its home spot, facing a completely random direction. Its sensor readings are corrupted with a little noise too, so no two robots see exactly the same numbers. This is a thin slice of domain randomization (Tobin et al., 2017^[5], Peng et al., 2018^[6]). The stock locomotion suite also randomizes torso mass and center of mass and shoves the robot at a random interval between 10 and 15 seconds, a separate knob from the command resampling above, but the H1 task switches all three off, and it pins friction to one value instead of a range.

Why bother? Because we want one policy that handles any command under slightly varied conditions, not a policy that memorized a single easy scenario. More on why this buys robustness below.

One policy, shared by everyone (parameter sharing)

All 1,024 robots read from the same neural network weights. This is called parameter sharing. Instead of training 1,024 separate brains, we train one and let every robot use it simultaneously. Robot #7 and robot #900 feed their own observations in and get their own actions out, but the weights doing the computing are identical and shared.

This is exactly what we want. The task is the same task for all of them, track a velocity, stay upright, so a single policy that’s good at the general problem is far more useful (and far cheaper to train) than a thousand specialists. And it means every robot’s experience is evidence about how to improve the same set of weights.

Pooling experience into one averaged update

We already did the per-iteration arithmetic back in the loop. Every robot contributes 24 steps, so the shared buffer fills with $1{,}024 \times 24 = 24{,}576$ transitions before each update. Here is the whole training budget in one place:

The point worth adding here is whose experience fills that batch, and what the averaged update does with it. It is a deliberately mixed bag, forward-walking, turning, standing-still, and stumbling experience jumbled together from a thousand differently-commanded, differently-randomized robots. When PPO shuffles those transitions into mini-batches and averages the loss over them, it pays no attention to which robot each one came from, so the gradient is effectively averaged over every robot’s experience at once. Picture each robot casting a vote on how to nudge the weights, and the update following the average:

$$\nabla_\theta \mathcal{L} \approx \frac{1}{N}\sum_{i=1}^{N} \nabla_\theta \mathcal{L}_i, \qquad N = 24{,}576.$$

That single averaged step improves the one shared policy, which is then copied straight back out to all 1,024 robots for the next iteration. This is the whole reason parameter sharing pays off. A thousand robots are not training a thousand brains, they are collecting a thousand streams of evidence about how to improve the same one.

Why massive parallelism is such a good deal

Big wins fall out of pooling experience this way.

Speed. The GPU simulates all 1,024 robots in parallel, so they cost far less than running them one after another. We gather 24,576 fresh transitions in roughly the time one robot would take to produce 24. Real-world experience would be hopelessly slow (and would break real hardware). Here we collect ~24.6M steps of it in under two hours.

Low-variance gradients. A gradient estimated from one robot’s 24 steps is noisy, it might reflect one lucky stride or one unlucky faceplant. Averaging over 24,576 transitions cancels out that noise. Statistically, the noise in an averaged estimate shrinks like $1/\sqrt{N}$, so a batch this large gives a far steadier, more trustworthy signal about which way to improve. Steadier gradients mean we can learn confidently and quickly instead of lurching around.

Robustness, for free. Because those 24,576 transitions span many commands, many start poses, and noisy sensor readings, the single policy is forced to get good at the general problem rather than one narrow case. That spread across the 1,024 robots is what pushes the learned brain toward something that doesn’t fall over the moment conditions shift, the first step toward a policy that might survive outside the simulator.

The brain: an actor-critic network

We’ve leaned on the phrase “the policy” for several sections now without ever pinning down what it is. So what actually decides which joint targets to send 50 times a second? That’s the job of the policy network, the “brain” of the whole operation. In PPO this brain actually comes in two parts that train together but do very different jobs, the actor and the critic. Think of the actor as the doer and the critic as the coach.

The actor: from what-I-sense to what-I-do

The actor is the part that we actually deploy on the robot. Its job is to look at the current observation and decide on an action.

The observation is the vector we described earlier, proprioception (joint positions and velocities, base linear and angular velocity, and the projected gravity vector that tells the robot which way is down), plus its own previous action, glued together with the current velocity command (the $v_x$, $v_y$, $\omega_z$ the robot is being asked to hit right now). Concretely it is one flat vector:

$$s_t = \Big[\; \underbrace{v^{\text{base}}}_{3},\; \underbrace{\omega^{\text{base}}}_{3},\; \underbrace{g^{\text{body}}}_{3},\; \underbrace{(v_x^{\text{cmd}},\,v_y^{\text{cmd}},\,\omega_z^{\text{cmd}})}_{3},\; \underbrace{q-q^{\text{def}}}_{19},\; \underbrace{\dot q}_{19},\; \underbrace{a_{t-1}}_{19} \;\Big] \in \mathbb{R}^{69}$$

reading left to right, base linear velocity, base angular velocity, the projected-gravity vector, the velocity command, the joint angles (measured relative to the default pose) and joint velocities, and the previous action, $69$ numbers in all for H1’s $19$ actuated joints. (The terrain height-scan input is switched off on flat ground. It is what the rough-terrain variant would add here.) Feeding the policy its last action gives it a sense of what it just did, which helps it produce smooth, continuous motion rather than jerking between unrelated targets. The action is the set of 19 target joint angles that get handed to the PD controllers.

The mapping between them is a small multi-layer perceptron (MLP) - the plainest kind of neural network, just a stack of “multiply by weights, add a bias, apply a nonlinearity” layers. Here it has three hidden layers of 128 units each, with ELU^[3] activations (Exponential Linear Unit, a smooth cousin of the more famous ReLU that lets small negative values through instead of hard-clipping them to zero - this tends to make training a little smoother):

$$\text{obs} \;\rightarrow\; [128] \;\rightarrow\; [128] \;\rightarrow\; [128] \;\rightarrow\; \boldsymbol{\mu}\ (19\text{ joint targets})$$

The subtle part is that the network does not output the action directly. It outputs the mean $\boldsymbol{\mu}$ of a Gaussian (bell-curve) distribution over joint targets. To actually act, we sample from that distribution:

$$a \sim \mathcal{N}(\boldsymbol{\mu}(\text{obs}),\ \sigma^2)$$

Why sample instead of just using the mean? Because a policy that always does exactly the same thing can never discover that something else works better. The random jitter is exploration - it’s how the robot stumbles into a slightly better stepping pattern and PPO gets to notice “hey, that variation earned more reward, do more of that.” The spread of that jitter is controlled by a standard deviation $\sigma$, which here is a separate, learned parameter rather than something the network computes per state. It starts at init_noise_std = 1.0 (nice and wide - explore wildly at first) and the training process is free to shrink it as the policy gets confident. Because $\sigma$ is state-independent, the exploration noise is the same shape no matter what the robot is currently sensing. It is a learned per-joint vector, one standard deviation per action dimension (a diagonal covariance over the 19 actions), all initialized to 1.0 rather than a single global scalar.

At deployment we drop the randomness entirely and just use the mean $\boldsymbol{\mu}$. Exploration was a training-time tool. On the real robot you want the policy’s single best guess, not a dice roll.

The critic: a scorekeeper, not a decider

The critic is a second MLP with the exact same shape - three hidden layers of 128 ELU units - but its output layer produces a single number instead of 19:

$$\text{obs} \;\rightarrow\; [128] \;\rightarrow\; [128] \;\rightarrow\; [128] \;\rightarrow\; V(s)\ \in \mathbb{R}$$

That number is the state-value V(s), roughly, “starting from this situation and behaving the way I currently do, how much total future reward should I expect?” The critic never picks an action. Its only purpose is to give the actor a baseline to compare against, so PPO can compute advantages - a measure of whether an action turned out better or worse than expected from that state. (Those advantages are what steer the actor’s updates, through the clipped objective we walked through in the PPO loop above.) Because the critic is purely a training aid, it’s thrown away at deployment. The shipped robot carries only the actor.

To make that abstract “value” concrete, here is the final policy’s critic scoring states in real time as the robot walks, gets shoved, and topples:

The division of labor is the whole idea behind “actor-critic”. The actor proposes actions, the critic judges how things are going, and the critic’s judgment is exactly what makes the actor’s learning signal less noisy. Two humble networks, one doing, one grading.

Every knob, and what it does

Most of these knobs already showed up earlier, scattered across the loop, the network, and the objective. This section just gathers them in one place.

The full knob list

The auto-LR loop, spelled out

The one knob that behaves differently from the rest is the learning rate, because we don’t hold it fixed, it steers itself. Here’s the feedback loop in words.

Before each gradient step, we measure how far the policy has already drifted from the one that collected the data, using the KL divergence. KL divergence is just a number that says “how different are these two probability distributions?”, zero means identical, larger means they’ve drifted apart. We compare that measured change to our target, desired_kl $= 0.01$:

• If the policy moved too much (measured KL above twice the target), we were being reckless, so we lower the learning rate, dividing it by $1.5$.
• If the policy barely moved (measured KL below half the target), we were being timid and wasting the update, so we raise the learning rate, multiplying it by $1.5$.
• Anywhere between those two marks we leave the learning rate alone. The dead band stops the controller thrashing on ordinary noise.

$$\text{KL} > 2\,\text{target} \;\Rightarrow\; \text{LR} \leftarrow \text{LR}/1.5 \qquad\qquad \text{KL} < \tfrac{1}{2}\,\text{target} \;\Rightarrow\; \text{LR} \leftarrow 1.5\,\text{LR}$$

The effect is a thermostat for learning speed. Early on, when the loss landscape is friendly, the controller cranks the LR up and we make fast progress. Later, near convergence, it eases off so we don’t kick a good policy off a cliff. This is why you rarely need to hand-tune the exact learning rate for these locomotion tasks, you tune the target behavior (desired_kl) instead, and let the loop find the step size. Note that this is a second safety loop layered on top of clip_param. The clip removes the incentive to push past the ratio bound, while the KL controller resizes the step itself. Both run on every mini-batch, so the learning rate can be adjusted up to 20 times within a single iteration.

Designing the reward: telling the robot what good means

The network decides what to do, but it can only chase whatever we pay it to chase, which brings us to the reward. In reinforcement learning (RL), the robot never sees a “correct” set of joint angles the way it would in supervised learning. Instead, at every control step it receives a single number, the reward, and its only mandate is to act so that the sum of rewards over an episode is as large as possible. That means the reward function is the whole job description. If we get it wrong, the robot will do exactly what we asked and nothing like what we wanted, a classic failure mode sometimes called “reward hacking.” So designing the reward is less like writing a loss function and more like writing a contract that a very literal-minded intern will exploit to the letter.

Our contract for the Unitree H1 follows a philosophy you’ll see across most locomotion work. A few positive terms that define the goal, and many small negative terms that define good style and basic safety. The positive terms say what to accomplish (move at the commanded velocity, take real steps). The penalties say how to accomplish it (stay upright, don’t flail, don’t twitch, don’t cheat). Keeping the goal terms few and large while keeping the style terms many and small is deliberate. The robot should be overwhelmingly motivated to do the task, and only gently nudged toward doing it gracefully. If a posture penalty were as large as the tracking reward, the robot might freeze in a pretty pose and never walk.

The goal terms: tracking velocity with an exponential kernel

The two headline rewards ask the robot to match the commanded velocity. Recall the command is a target forward speed $v_x^{\text{cmd}}$, a side speed $v_y^{\text{cmd}}$ that H1 holds at 0, and a target turn rate $\omega_z^{\text{cmd}}$. A naive way to reward “getting close” would be the negative squared error, $-\lVert v^{\text{cmd}} - v \rVert^2$. That works in principle, but it has an ugly property. The penalty grows without bound. A robot that is briefly way off (say, it stumbles and its base lurches at high speed) gets slammed with an enormous negative number, and that one bad instant can swamp thousands of good steps. The gradient is also largest when you’re farthest from the target, which pushes the policy hardest exactly when its behavior is most erratic and least informative.

Instead we wrap the error in an exponential kernel:

$$r_{\text{track,xy}} = \exp\!\left(-\frac{\lVert v^{\text{cmd}}_{xy} - v_{xy}\rVert^2}{\sigma^2}\right), \qquad r_{\text{track,}\omega_z} = \exp\!\left(-\frac{(\omega_z^{\text{cmd}} - \omega_z)^2}{\sigma^2}\right),$$

with $\sigma = 0.5$. A precise note on frames, since it matters for what “the velocity” means. The H1 task binds these two terms to the frame-aware implementations track_lin_vel_xy_yaw_frame_exp and track_ang_vel_z_world_exp. The linear term measures the error in the robot’s gravity-aligned yaw frame (its own forward/side directions, projected level), so “forward” always means where the robot is facing, not a fixed world axis. The yaw term compares the world-frame turn rate against the command.

This little change of kernel buys us three things:

• It is bounded in $[0, 1]$. A perfect match gives exactly $1$. Being wildly off gives something near $0$. No single instant can produce a runaway reward or penalty, so velocity tracking can never bully the other terms into irrelevance. It also makes rewards from different terms comparable in scale, which is why almost every term here lands within an order of magnitude of $1$.
• Its gradient is strongest near the target. The exponential is nearly flat when the error is huge (being “very wrong” and “slightly less very wrong” barely differ), and steepest as you approach zero error. So the learning signal sharpens precisely in the regime where fine-tuning matters, shaving the last $0.1$ m/s off the tracking error. The parameter $\sigma$ sets the width of this sweet spot. At $\sigma = 0.5$, an error of half a meter per second still earns about $e^{-1} \approx 0.37$ of the reward, so the robot is encouraged but not tortured.
• It caps how much raw speed-matching can dominate. Because the term saturates at $1$, once the robot is tracking well it can’t earn more by obsessing over velocity. That frees the optimizer to start caring about the smaller style penalties, which is exactly the progression we want, nail the task first, then clean it up.

Baseline reference (three-motion test). The clip below is the trained policy run through a three-motion test of walking straight, walking a curve, and spinning in place. Each expected curve is shown as a black line pre-drawn on the white floor (a line-follower track). Every reward term is kept in place.

To understand the effect of each term, from here on we retrain with one term surgically removed at a time, that is the ablation, and run the resulting policy through the three-motion test so you can see exactly what its absence cost.

Ablation, without speed tracking (-track_lin_vel_xy_yaw_frame_exp). The robot stops chasing the commanded speed and drifts.

Measured (250-step rollout, 64 robots). velocity-tracking error (xy) 0.57 m/s vs baseline 0.11 m/s (5.1×).

Ablation, without turn tracking (-track_ang_vel_z_world_exp). The robot no longer turns toward the commanded heading.

Measured (250-step rollout, 64 robots). turn-rate error 1.01 rad/s vs baseline 0.12 rad/s (8.3×).

Take real steps, don’t shuffle or slide (feet_air_time, feet_slide)

Here’s a subtle trap. A humanoid can technically satisfy the velocity command by sliding skating forward on stiff legs, or shuffling with tiny, high-frequency micro-steps. Both track the commanded speed while looking nothing like walking. H1 uses the biped-specific variant of this term, feet_air_time_positive_biped, and its logic is worth stating precisely because it is not the naive “reward swing time” you might expect. It looks only at moments of single support, exactly one foot on the ground, and pays out the duration of the current phase (how long the planted foot has been down, or equivalently the swing foot has been up), capped at $0.6$ seconds, and only while the robot is actually commanded to move. Rewarding single-support time is what pushes a clean one-foot-at-a-time cadence. To earn it the robot must lift a foot, carry it, and plant it, rather than scoot on two feet or double-shuffle. The $0.6$ s cap stops a stalled single-support phase from paying out forever, though the trained gait’s swings stay well under half that, so in practice the cap rarely does anything.

Formally, letting $t^{\text{mode}}_f$ be foot $f$‘s time in its current phase (contact time if planted, air time if swinging), the term takes the smaller of the two feet’s phase times, clamps it to $t_{\max}$, gates it on being in single support, and pays it only while a move is commanded:

$$r_{\text{air}} = \min\!\Big(\min_{f\in\text{feet}} t^{\text{mode}}_f,\ t_{\max}\Big)\cdot\mathbb{1}\!\left[\text{single support}\right]\cdot\mathbb{1}\!\left[\lVert v^{\text{cmd}}_{xy}\rVert > 0.1\right], \qquad t_{\max}=0.6\ \text{s}.$$

The companion anti-slide penalty charges any horizontal foot motion while that foot is in contact with the ground:

$$r_{\text{slide}} = -\sum_{f\in\text{feet}} \lVert v^{\text{foot}}_{f,\,xy}\rVert\; \cdot \mathbb{1}\!\left[\text{contact}_f\right].$$

Ablation, without the swing-time reward (-feet_air_time). It shuffles with tiny low-lift steps instead of real strides.

It still scores well (reward 30.9) because shuffling tracks velocity. The gait is the giveaway.

Ablation, without the anti-slide penalty (-feet_slide). Planted feet skate along the ground.

This run scores the highest reward of any ablation here (33.6). Sliding is an efficient cheat that the tracking metrics actually reward, which is exactly why the penalty exists.

What the two foot terms actually pay, moment by moment

This section goes a level deeper, tracing reward step by step against the exact moment a foot lifts, swings, and lands, instead of one end-of-episode score, for a clearer view of what is happening. The clips below show the feet_air_time reward and feet_slide penalty for the trained baseline, two ablations, and the baseline at a faster commanded speed.

The figure below lines every run’s reward and foot-contact trace up on the same axes, so the shuffling policy’s shrunken sawtooth and the skating policy’s swollen slide penalty are both visible in a single glance.

The fall penalty: the dominant shaping signal

By far the largest number in the whole scheme is the termination penalty of $-200$, applied once if the robot falls (which also ends the episode). What counts as “a fall” is concrete, not vibes. A contact sensor watches the torso link, and if the torso touches the ground the episode terminates and the penalty fires. (The only other way an episode ends is reaching the time limit, which is not a fall and carries no penalty. This is the same time-out we bootstrapped through in the advantage calculation.) Everything else lives in roughly the $[-1, +1]$ range per step, so a single fall wipes out the reward of hundreds of good steps. This asymmetry is intentional and it’s arguably the most important design decision here.

As a formula it is just a one-shot indicator on the termination event:

$$r_{\text{term}} = -200\cdot\mathbb{1}\!\left[\text{episode ended by a fall}\right].$$

Ablation, without the fall penalty (-termination_penalty). This clip doesn’t show a fall, but balance is no longer specially prioritized, so the fall rate rises across training as a whole.

Measured (training). fall rate 5.3% vs baseline ~2.0% (2.6×). Metrically it still walks. The cost is safety, not tracking.

Posture penalties: no flailing, natural gait

An upright, velocity-tracking robot can still look deeply wrong, windmilling its arms for balance, twisting its trunk, splaying its hips. The joint_deviation penalties discourage this by charging a small cost for how far a joint strays from its default posture, $\lvert q - q^{\text{def}} \rvert$, applied separately to hips ($-0.2$), arms ($-0.2$), and torso ($-0.1$). One subtlety here is that the hip term penalizes only the hip yaw and roll joints, not hip pitch. That is deliberate, pitch is the joint that swings the leg forward to take a step, so leaving it free lets the robot stride naturally while yaw and roll (which would splay or twist the legs sideways) are pinned near neutral. The arms term is what keeps the H1 from flailing its upper body as a cheap balancing trick. The hip-yaw/roll and torso terms keep the legs squared-up and the trunk composed. The net effect is a natural gait. The robot learns to balance with its legs and stepping rather than with frantic whole-body contortions, because those contortions now cost it reward.

Each deviation term is an L1 distance from the default pose over its joint group, and “stay upright” penalizes how far gravity tilts out of the body-vertical axis (its projection onto the body $xy$ plane):

$$r_{\text{dev},g} = -\sum_{j\in g} \lvert q_j - q^{\text{def}}_j\rvert \quad (g\in\{\text{hip},\text{arms},\text{torso}\}), \qquad r_{\text{upright}} = -\lVert g^{\text{body}}_{xy}\rVert^2.$$

The ankle joint-limit penalty stays at zero until a joint pushes past its soft limit, then grows linearly:

$$r_{\text{lim}} = -\sum_{j\in\text{ankles}} \Big[\operatorname{ReLU}\!\left(q_j - q^{\max}_j\right) + \operatorname{ReLU}\!\left(q^{\min}_j - q_j\right)\Big].$$

Ablation, without the stay-upright penalty (-flat_orientation_l2). The robot walks visibly hunched/pitched forward.

Measured (250-step rollout, 64 robots). trunk tilt off vertical 22.88° vs baseline 3.94° (5.8×).

Ablation, without the calm-arms penalty (-joint_deviation_arms). The arms swing and flail as a free balance crutch.

Measured (250-step rollout, 64 robots). mean arm-joint deviation 0.58 rad vs baseline 0.027 rad (21×).

Ablation, without the neutral-hips penalty (-joint_deviation_hip). The hips drift from their neutral posture.

Measured (250-step rollout, 64 robots). mean hip-joint deviation 0.25 rad vs baseline 0.13 rad (2.0×).

Ablation, without the untwisted-torso penalty (-joint_deviation_torso). The trunk twists more as it walks.

Measured (250-step rollout, 64 robots). mean torso-joint deviation 0.23 rad vs baseline 0.13 rad (1.8×).

What the posture penalties actually charge

Now the same lens turns on posture. The clips below show flat_orientation_l2, joint_deviation_arms, and joint_deviation_hip traced step by step for three ablations, rather than one end-of-episode number.

The figure below lines every run’s penalty and the deviation driving it up on the same axes, so how far each ablation strays and what it costs are both visible in a single glance.

Smoothness penalties: no twitch, no wobble

The last family removes the high-frequency jitter that plagues freshly trained policies. action_rate_l2 penalizes $\lVert a_t - a_{t-1}\rVert^2$, the change in commanded joint targets from one step to the next, discouraging jerky, twitchy commands. dof_acc_l2 penalizes joint accelerations, further favoring smooth, low-energy motion (and, on real hardware, gentler actuators). ang_vel_xy_l2 penalizes roll and pitch rate, i.e. the base wobbling side to side or nodding fore-and-aft, which produces a steadier trunk. These weights are tiny (down to $-1.25\times 10^{-7}$ for accelerations, since raw accelerations are numerically large) precisely because they should only sand off the rough edges, never override the task.

The three smoothness terms are just squared magnitudes, of the step-to-step action change, the joint accelerations, and the base roll/pitch rate:

$$r_{\text{act}} = -\lVert a_t - a_{t-1}\rVert^2, \qquad r_{\text{acc}} = -\lVert \ddot q\rVert^2, \qquad r_{\text{wobble}} = -\lVert \omega_{xy}\rVert^2.$$

Ablation, without the smoothness penalties (-action_rate_l2 + dof_acc_l2). The motion becomes twitchier, which a still frame hides.

Measured (250-step rollout, 64 robots). mean step-to-step action change 0.20 vs baseline 0.11 (1.8×).

Ablation, without the no-wobble penalty (-ang_vel_xy_l2). The trunk rocks and nods more.

Measured (250-step rollout, 64 robots). mean roll/pitch rate 0.85 rad/s vs baseline 0.44 rad/s (2.0×).

Ablation, without the joint-limit penalty (-dof_pos_limits). The ankles ride closer to their mechanical limits.

Measured (250-step rollout, 64 robots). mean ankle deviation (toward end-stops) 0.34 rad vs baseline 0.11 rad (3.0×).

What the wobble and smoothness penalties actually charge

The last two penalties get the same treatment. The clips below show ang_vel_xy_l2 and the smoothness terms traced step by step for two ablations, rather than reduced to one final number.

The figure below uses the same layout, penalty against the motion causing it, so the trunk-rate spike and the smoothness jitter are both easy to spot across all three runs.

Why the H1 task disables the vertical-velocity penalty (lin_vel_z_l2)

Isaac Lab ships a base locomotion reward set with a few extra terms, and the stock H1 config deliberately switches some off. The most instructive is lin_vel_z_l2, which penalizes vertical velocity of the base. That term makes sense for a wheeled robot or a quadruped that should glide at constant height, but it is actively harmful for a biped. Walking on two legs is fundamentally a controlled fall-and-catch. The center of mass necessarily rises and dips with every step as the body vaults over the stance leg. Penalizing vertical motion would fight that natural bob and push the robot toward a stiff, gliding gait that is both unnatural and harder to balance. We also disable dof_torques_l2 (redundant with our smoothness terms and prone to making the robot too timid) and undesired_contacts (a biped’s feet, and sometimes its posture, need contact freedom the base term wouldn’t allow).

The full reward at a glance

The table below sums up the whole reward, a few large positive terms for the goal, one huge penalty for falling, and a long tail of small penalties that shape how the robot moves. Every step the policy optimizes the weighted sum of all of them, and getting the proportions right, goals near $1$, safety dominant, style terms small, is what turns random early flailing into a stable walk.

Results, reading the curves

Reward, velocity-tracking error, and fall rate, plus the three PPO loss terms (policy/surrogate, value, entropy), over 1,000 iterations (bold = smoothed, faint = raw). Hover anywhere to read every metric at that iteration. The crosshair is shared across all six panels.

With the knobs set as above, here is what a full 1,000-iteration run actually produces. The three curves tell one story, and it matches the reward design exactly:

• It learns “don’t fall” first, then “walk well.” Early on the fall-rate climbs to ~100% while the robot topples repeatedly and the huge −200 penalty drags reward negative. Around iteration ~200 the policy discovers how to stay upright, fall-rate collapses toward zero, and only then does reward climb from negative into the mid-+20s as the small (+1) tracking rewards start paying off.
• Massive parallelism makes it fast. 1,024 robots produce ~24.6M steps of experience in under two hours and 1,000 updates.
• Reward shaping is the real design work. Two velocity terms define the goal. The posture and smoothness penalties are what turn the solution into a natural, upright walk instead of a reward-hacking twitch.
• Flat walking converges early (~iteration 400) and is the easy part. Complexity gets added on top for rough terrain, faster gaits, and sim-to-real robustness.

From walking to navigation: driving the policy with a navigation stack

Everything above trained a velocity-tracking controller. Hand the policy a target $(v_x, v_y, \omega_z)$ and it walks that way. That command is a clean, narrow interface, so we can stack a navigation layer on top that decides where to go and emits velocity commands, the exact signal the policy already consumes. The walking controller itself does not change. This is the standard robotics split. A global planner finds a route to the goal, a local planner follows that route, and the locomotion policy does the walking.

To make it concrete we drop the trained H1 into a flat-floor house, three rooms (an entry, a living room, and a bedroom) separated by walls with ordinary doorways, plus a table sitting in the middle of the living room and a bed in the bedroom. The global planner builds an occupancy grid of the house and inflates the walls and furniture by the robot’s radius. That inflation is the trick that lets the planner treat the robot as a single point. Once the obstacles are grown by the robot’s radius, any point-path that steers clear of them also keeps the robot’s whole body clear (this is called configuration-space expansion).

It then runs A*^[13] on that grid from the robot’s cell to the goal. A* explores outward from the start, always expanding the most promising cell next, the one that minimizes distance travelled so far plus a straight-line guess of the distance still to go ($f = g + h$). Since that guess never overestimates, the first time A* reaches the goal it has a shortest route. It may step diagonally but is not allowed to cut across a wall corner. Raw A* returns a jagged staircase of cells, so we simplify it with greedy line-of-sight shortcutting. Wherever two waypoints can “see” each other in a straight line with no obstacle between, the cells in the middle are dropped, which collapses the staircase into the handful of straight legs (five corners here) you actually see.

The local planner then follows that route with pure pursuit^[14]. At each step it looks a fixed distance ahead along the path, aims the robot at that lookahead point, and picks a forward speed. (Strictly, classic pure pursuit follows the arc through that point.) That lookahead distance is the one knob that matters. Set it too short and the robot fixates on the nearest scrap of path and weaves, oscillating from side to side. Set it too long and it cuts corners, aiming so far ahead that it shortcuts across the inside of a turn and clips the wall. We tuned it just long enough to smooth the doorways without cutting them. The planner never touches a joint and never reasons about balance. It sets only a heading and a speed, and the same locomotion policy turns that into a walk.

One honest note about where the robot thinks it is. Because this runs in simulation, we read the robot’s ground-truth pose straight from the simulator, so A* always knows exactly where the robot is and we needed no localization or SLAM. On real hardware that pose is not free. It would come from a SLAM / localization stack, and its drift and noise would be the planner’s problem. Here that whole layer is handed to us.

Test drive the house, then improve the policy

In this section, we’ll test the baseline policy in the house we built and see how it works. And, honestly, it worked surprisingly well right out of the box, which was pretty cool to see. But looking more closely, there were a few things that could be improved.

So we decided to try a few targeted fine-tunes, building on top of the policy we already had rather than starting from scratch. This is an important distinction from the reward ablations we showed earlier, where each policy was trained from scratch. Here, we simply added more training to the existing policy. The motivation was straightforward. We already had a good policy, so rather than recreate it from scratch, we wanted to make a few targeted improvements. This also made the fine-tuning process much faster.

1. Baseline (stock defaults)

Here’s the baseline policy driving the same house and A* route.

Three things stood out on closer look:

• The turns aren’t clean. About $0.5$ to $0.6$ rad/s, wide arcs that swing toward the walls instead of pivoting cleanly through the doorway. It still threads this house only because the rooms are forgiving. A tighter turn is where it actually breaks, which is the failure we isolate in the next step.
• No planted stance at the goal. It never really settles, it just keeps jogging in place.
• The walk could be smoother. Maybe we are over-indexing on that one, but it’s noticeable.

The next three fine-tunes take these one at a time, starting with turning. To see each fine-tune in isolation, every checkpoint below is also run through the same three-motion line-follower test used for the reward ablations, walk straight, walk a curve, spin in place. The baseline clips below are the exact same recordings from the Baseline reference (three-motion test) earlier, not a re-run:

2. Faster turning

We widened the yaw range to $\omega_z\in[-2,2]$ and stiffened the heading controller to $k=1.5$. It now turns at about $2$ rad/s and will spin essentially in place, which is the headroom you want for tighter maps. The side effect is that it marches even harder at rest and sways more.

The same three motions after the turning fine-tune, note how much tighter the spin is:

3. A planted stance

The stock task asks only 2% of training environments to hold a zero command, so standing still was barely learned. Raising that to 15% teaches it to plant its feet, and now it stops dead at the goal instead of jogging. The jog-in-place we waved off at the very top is exactly what we now train away. The walk still sways.

The three motions after the standing fine-tune, note it now plants its feet at the end of the straight and curve runs instead of jogging:

4. A smooth walk (final)

We halved the yaw-tracking reward (from $1.0$ to $0.5$), tripled the body-wobble penalty, and doubled the tilt penalty. Trunk wobble drops below even the baseline while the tight turns and the planted stop are kept. This is the policy in the demo above.

The final policy on the same three motions, tight spin and planted stops kept, now with a calmer trunk:

The numbers behind the videos. Standing still at a hard zero command (16 robots, how far the feet travel, a planted foot barely moves):

Walking quality (straight walk at $0.5$ m/s, lower is calmer):

The turning and standing fine-tunes each fixed one problem while nudging another the wrong way (faster turning made the sway worse, the planted stance did too), which is exactly why the final pass explicitly penalizes wobble and tilt. It lands the best balance of the four, a policy that turns tightly, mostly plants at the goal (11/16), and, on trunk wobble, walks more smoothly than even the baseline while holding tilt about even with it.

Wrapping up

This turned out to be a pretty long write-up, but it was genuinely enjoyable to put together. There was a lot to explore and play with along the way. The post went deep into reward engineering, measuring moment by moment exactly what each reward term contributes to a given gait. From there, the trained policy left the training environment entirely and was put to work navigating an actual house. The navigation stack was built largely from scratch, including the global planner, local controller, and the targeted fine-tuning needed to improve the resulting behavior.

The one deliberate shortcut was localization. Instead of building a full SLAM stack, the robot’s pose was read directly from the simulator. That felt like a reasonable tradeoff, keeping the focus on the walking policy and navigation logic rather than sensor fusion and state estimation.

One takeaway for me from the fine-tuning process was that while it’s genuinely easy to get something reasonable working with reward engineering, getting from working to really good is a lot harder. Tuning the reward functions for a specific behavior can sometimes turn into a bit of a rabbit hole, where a small change improves one thing but introduces another issue somewhere else. It takes patience and quite a bit of iteration. There’s definitely an art to getting these reward functions just right.

Hopefully this was as much fun to read as it was to build.

References

Reinforcement learning

• [1] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. https://arxiv.org/abs/1707.06347 ↩
• [2] Schulman, J., Moritz, P., Levine, S., Jordan, M. I., & Abbeel, P. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation (GAE). arXiv:1506.02438. https://arxiv.org/abs/1506.02438 ↩
• [3] Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). arXiv:1511.07289. https://arxiv.org/abs/1511.07289 ↩
• [4] Pardo, F., Tavakoli, A., Levdik, V., & Kormushev, P. (2018). Time Limits in Reinforcement Learning. ICML 2018; arXiv:1712.00378. https://arxiv.org/abs/1712.00378 ↩
• [5] Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS 2017; arXiv:1703.06907. https://arxiv.org/abs/1703.06907 ↩
• [6] Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018). Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. ICRA 2018; arXiv:1710.06537. https://arxiv.org/abs/1710.06537 ↩

Simulation and training frameworks

• [7] NVIDIA. Isaac Sim: Robotics Simulation and Synthetic Data Generation. https://developer.nvidia.com/isaac/sim ↩
• [8] NVIDIA. Isaac Lab: Unified framework for robot learning built on Isaac Sim (GitHub). https://github.com/isaac-sim/IsaacLab ↩
• [9] Mittal, M., et al. (2023). Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments (predecessor of Isaac Lab). arXiv:2301.04195. https://arxiv.org/abs/2301.04195 ↩
• [10] ETH Zurich Robotic Systems Lab. rsl_rl: RL algorithms for robotics (GitHub). https://github.com/leggedrobotics/rsl_rl ↩
• [11] Rudin, N., Hoeller, D., Reist, P., & Hutter, M. (2021). Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. CoRL 2021; arXiv:2109.11978. https://arxiv.org/abs/2109.11978 ↩
• [15] NVIDIA. H1FlatEnvCfg, the environment config behind the Isaac-Velocity-Flat-H1-v0 task (Isaac Lab source). https://github.com/isaac-sim/IsaacLab/blob/main/source/isaaclab_tasks/isaaclab_tasks/manager_based/locomotion/velocity/config/h1/flat_env_cfg.py ↩

Robot hardware

• [12] Unitree Robotics. Unitree H1 full-size bipedal humanoid robot. https://www.unitree.com/h1 ↩

Navigation

• [13] Hart, P. E., Nilsson, N. J., & Raphael, B. (1968). A Formal Basis for the Heuristic Determination of Minimum Cost Paths (A* search). IEEE Transactions on Systems Science and Cybernetics, 4(2), 100-107. https://doi.org/10.1109/TSSC.1968.300136 ↩
• [14] Coulter, R. C. (1992). Implementation of the Pure Pursuit Path Tracking Algorithm. Technical Report CMU-RI-TR-92-01, Robotics Institute, Carnegie Mellon University. https://www.ri.cmu.edu/publications/implementation-of-the-pure-pursuit-path-tracking-algorithm/ ↩