SHAC and SAPO locomotion with the MuJoCo Warp adjoint

A differentiable simulator returns the derivative of the next state with respect to the current state and control. Short-Horizon Actor-Critic (SHAC) uses these derivatives to train a policy by back-propagating short simulated rollouts, and a learned value function summarizes everything beyond the rollout [Xu et al. 2022]. Compared with reinforcement learning from sampled returns, this can need far fewer simulated steps, but only if the gradients stay informative through contact over the rollout. This report tests SHAC and its maximum-entropy variant SAPO [Xing et al. 2025] on the analytic step adjoint of MuJoCo Warp, the GPU implementation of MuJoCo, as proposed in pull request #1535 at commit 357a75d.

The study has two parts. The first asks whether the pull request's implicit time integration allows larger physics timesteps, and what larger timesteps do to throughput, gradient quality, and the learned policies, on the reference running tasks of the original SHAC work. The second trains velocity-tracking policies on tasks that follow Isaac Lab's locomotion environments, for the Unitree Go1 quadruped and the Unitree H1 and G1 humanoids, and returns to the classic MuJoCo Humanoid's running task. It identifies what prevented learning and what fixed it. Figures 1–4 show four of the resulting policies in a simulator with a finer timestep than the one they were trained in.

The main findings are as follows. Implicit damping removes the explicit-damping instability, but larger timesteps shorten the horizon over which the policy gradient is informative and change the dynamics enough that 20 ms policies fail at 1 ms (Sections 3 and 4). On the velocity-tracking tasks, SHAC's tanh-squashed actions saturate, which multiplies the analytic action gradient by about 0.02; SAPO's entropy terms contain a correction that penalizes saturation, and with them every tested seed learned the full Go1 command set; the correction alone also sufficed, but learned about five times more slowly (Section 4). With SHAC and without a gait prior, H1 needed a target critic that follows the trained critic quickly; the G1 walked cleanly only with SAPO and a differentiable gait prior on foot heights and placement, whereas without the prior it tracked the commands with a limp (Section 5).

Analytic step gradients in MuJoCo Warp

MuJoCo Warp (MJWarp) is a GPU implementation of MuJoCo. Pull request #1535 adds an analytic backward pass for one simulation step. MuJoCo computes accelerations by solving a convex constraint problem for contacts, joint limits, and friction. The pull request differentiates the converged solution with the implicit function theorem instead of differentiating the solver iterations. At commit 357a75d it supports the Newton constraint solver, elliptic and pyramidal friction cones, the Euler and implicitfast integrators, and eleven primitive collision pairs. Other pairs, such as capsule–cylinder, contribute no gradient. The velocity-tracking tasks therefore let robot geometry collide only with the floor, as Isaac Lab does for the robots used here; the final G1 configuration additionally lets its two feet collide with each other (Section 5).

The training harness captures each physics step and its adjoint as CUDA graphs and keeps the completed forward data of every physics step in a window, so that the backward pass applies the step adjoints in reverse order without recomputing the forward steps (Figure 5). The policy, written in PyTorch, and the simulator share one CUDA stream, and an epoch needs one host synchronization. The adjoint of the control input ignores actuator force limits. The harness therefore sets the control derivative of saturated actuators to zero.

Time integration

MuJoCo advances generalized positions \(q\) and velocities \(v\) with a physics timestep \(h\). The integrators considered here are semi-implicit: the new velocity updates the position, \(q_{k+1} = q_k \oplus h\,v_{k+1}\), where \(\oplus\) denotes integration on the configuration manifold. They differ in the acceleration used in \(v_{k+1} = v_k + h\,a_k\):

Here \(M\) is the joint-space inertia matrix, \(f_k = f(q_k, v_k, u_k)\) collects all applied, passive, bias, and constraint forces, \(u_k\) is the actuator control, and \(B\) is the diagonal matrix of joint damping coefficients. MuJoCo's Euler integrator uses the middle form by default. implicitfast also treats actuator velocity gains, such as the derivative gain of a position servo, implicitly, and omits Coriolis derivatives. For a joint with effective inertia \(I\) and damping \(b\), the explicit form multiplies the velocity by \(1 - hb/I\) per step and is unstable for \(h > h_\text{crit} = 2I/b\). Joints with small inertia and large damping therefore bound the timestep of explicit integration. The policy acts every \(\Delta t_c = 20\) ms and holds its control for \(K = \Delta t_c / h\) physics steps, so all objectives and metrics live on the same 20 ms control grid.

Short-Horizon Actor-Critic

SHAC [Xu et al. 2022] trains a stochastic policy \(\pi_\theta\) with parameters \(\theta\) on \(N\) parallel environments. Each epoch continues every environment for a window of \(T\) control steps from the states reached in the previous epoch and minimizes

where \(r_t^n\) is the reward of environment \(n\) at step \(t\), \(\gamma = 0.99\) the discount factor, \(s_T^n\) the state at the end of the window, and \(V_{\bar\phi}\) a target critic. The gradient \(\nabla_\theta L\) is obtained by back-propagating through the policy and through every simulation step of the window. If an environment terminates inside the window, its sum stops there without a value term. The critic \(V_\phi\) regresses TD(\(\lambda\)) targets with \(\lambda = 0.95\), and the target critic follows it once per epoch with the Polyak update \(\bar\phi \leftarrow \alpha\,\bar\phi + (1 - \alpha)\,\phi\). The factor \(\alpha\) sets how quickly the bootstrapped value at the window end reflects the current critic: the original humanoid setting \(\alpha = 0.995\) gives the target a half-life of 138 epochs, \(\alpha = 0.9\) a half-life of 6.6 epochs. A longer window \(T\) exposes more consequences of an action to the gradient, but back-propagating through contact over many steps produces large and inconsistent gradients (Section 3).

The policy samples a pre-squash action \(u = \mu_\theta(s) + \sigma_\theta \odot \epsilon\) with \(\epsilon \sim \mathcal{N}(0, I)\) and applies \(a = \tanh(u) \in (-1, 1)^d\), where \(d\) is the action dimension. For the position-controlled robots, \(a\) sets the targets \(q_\text{target} = \bar q + s_a\,a\) of proportional-derivative servos around a default pose \(\bar q\), with action scale \(s_a = 0.5\) rad. SHAC learns a state-independent standard deviation \(\sigma_\theta\), uses ELU networks with layer normalization, Adam, gradient-norm clipping at 1, and learning rates that decay linearly to \(10^{-5}\).

SAPO: entropy in the returns

Soft analytic policy optimization (SAPO) [Xing et al. 2025] keeps SHAC's window objective but replaces each reward by an entropy-augmented reward,

Here \(\pi_\theta\) is evaluated at the sampled action \(a_t\) in state \(s_t\), \(\mathcal{N}_j\) is the Gaussian density with mean \(\mu_j\) and standard deviation \(\sigma_j\), and the second term accounts for the squashing. The one-sample entropy estimate \(-\log \pi_\theta\) is shifted by the target entropy \(\bar{\mathcal{H}} = -d/2\) and scaled by \(d\), so that \(\hat h_t\) lies roughly in \([0, 1]\) and is comparable across robots. The temperature \(\alpha_H\) starts at 1 and is adjusted by gradient steps on \(-\log \alpha_H \,\big(\mathbb{E}[\log \pi_\theta] + \bar{\mathcal{H}}\big)\), which lowers it when the policy's entropy exceeds the target. The augmented reward \(\tilde r_t\) enters both the actor objective \(L(\theta)\) and the critic's TD targets. SAPO further uses a state-dependent standard deviation \(\sigma_\theta(s)\) with \(\log \sigma \in [-5, 2]\), two critics whose mean bootstraps the actor and whose minimum enters the targets, no target network, SiLU activations, AdamW, gradient clipping at 0.5, and learning rates of \(2 \cdot 10^{-3}\) (actor) and \(5 \cdot 10^{-4}\) (critic). All other settings are those of SHAC.

Saturated actions and the analytic gradient

Every analytic gradient that reaches the policy passes through the squashing function:

When an action component saturates, \(|a_j| \to 1\), the factor \(1 - a_j^2\) goes to zero and the simulator gradient no longer moves that component, however informative it is. The squash term of the log-density does the opposite in SAPO: since \(-\log \pi_\theta\) contains \(+\sum_j \log(1 - a_j^2)\), which tends to \(-\infty\) at saturation, the entropy reward penalizes saturated actions and its gradient pushes them back. To test this mechanism in isolation, a squash penalty adds only this term to SHAC's reward,

with a fixed weight \(w_\text{sq} = 0.3\). Unlike the entropy reward, it does not reward a larger standard deviation.

Tasks

Table 1 lists the robots. Section 3 and the first part of Section 4 use the SHAC reference tasks: forward velocity plus uprightness, heading, and height terms, with the Go1's velocity reward capped at 1.5 m/s. The velocity-tracking tasks of Sections 4 and 5 follow Isaac Lab's flat-terrain locomotion environments. The policy observes the base linear and angular velocities and the gravity direction in the body frame, the command \(c = (c_x, c_y, c_\omega)\) of forward, lateral, and yaw velocity, the joint positions relative to \(\bar q\), the joint velocities, and the previous action. Isaac Lab's tracking rewards are the kernels \(\exp(-\lVert c_{xy} - v_{xy}\rVert^2/\sigma_c^2)\) on the planar base velocity \(v_{xy}\) and \(\exp(-(c_\omega - \omega_z)^2/\sigma_c^2)\) on the yaw rate \(\omega_z\), with \(\sigma_c = 0.5\), plus Isaac Lab's penalties on vertical velocity, roll and pitch rates, tilt, torques, accelerations, action changes, joint limits, hip deviations, and a foot air-time reward. Episodes last 20 s and commands are resampled every 10 s. The humanoids terminate when the torso touches the floor; initial states come from a bank of states checked for penetration, with random heading and base velocities of up to 0.5 m/s.

Table 1. Robot models. DoF counts include the six floating-base coordinates. Menagerie is DeepMind's collection of MuJoCo robot models. The Isaac Lab equivalent Go1 has Isaac Lab's default pose, servo gains of its Go2 motor model, armature 0.02 kg·m², no joint damping or friction loss, and sliding-only foot friction.

For the Go1, a second reward follows mjlab's velocity task as used in a Go2 environment built on Rewarped: Gaussian kernels on the planar velocity error plus the squared vertical velocity (weight 2, variance 0.25 (m/s)²) and on the yaw-rate error plus the squared roll and pitch rates (weight 2, variance 0.5 (rad/s)²), an uprightness kernel and a default-pose kernel (weight 1 each), and penalties on action changes (weight 0.1) and soft joint-limit violations (weight 1). It is called the mjlab reward below.

Gait prior

The analytic gradient treats the contact set as fixed within a step, so it cannot value a foot that is about to lift off or touch down. A gait clock with period \(P = 0.8\) s gives foot \(i\) at control step \(t\) the phase \(\psi_i = (t\,\Delta t_c/P + \delta_i) \bmod 1\), with offsets \(\delta = (0, 0.5)\) for the left and right foot. A foot is in stance while \(\psi_i < D\), with duty factor \(D = 0.6\), and otherwise in swing with progress \(\rho_i = (\psi_i - D)/(1 - D) \in [0, 1)\). With \(p_i\) a point under the sole of foot \(i\), computed by differentiable forward kinematics, three terms are added to the reward:

The first term asks the foot to follow a swing arc of height \(z_\text{sw} = 0.08\) m above its standing height \(z_0\); \([\cdot]\) is 1 when its condition holds. The second penalizes the horizontal speed \(\dot p_{i,xy}\) of feet in clock stance, computed from the displacement over one control step. The third is a Raibert-type placement rule for the second half of the swing, with placement error \(e_i\): \(\tilde p_{i,xy}\) is the foot position relative to the base in the heading frame, \(\bar p_{i,xy}\) its position in the default pose, \(v_b\) the planar base velocity in that frame, and \(\kappa_v = \kappa_e = 0.1\) s. Placing the foot ahead in proportion to the velocity arrests a fall; the term in \(v_b - c_{xy}\) shifts the step to accelerate or brake. The policy observes \(\sin(2\pi t\Delta t_c/P)\) and \(\cos(2\pi t\Delta t_c/P)\). H1 uses \(w_h = 40\), \(w_s = 2\), \(w_r = 10\); the G1 uses \(w_h = 120\) and \(w_r = 20\) (units m⁻² for \(w_h\) and \(w_r\), s²/m² for \(w_s\)). In the SAPO configurations of the G1, commands with planar speed and yaw rate below 0.1 hold both feet in clock stance, so the prior asks the robot to stand rather than step in place.

Evaluation

Policies are evaluated deterministically (\(a = \tanh(\mu_\theta)\)) in a reference simulator, implicitfast with \(h = 1\) ms and the robot's original contact parameters, from 256 randomized starts for 10 s, with commands drawn as in training. The tracking error is the mean norm of the difference between the commanded and the measured planar base velocity; a policy walks when this error is below 0.3 m/s, since standing still gives 0.5–1.0 m/s. Gait quality is measured per foot under fixed commands from the simulator's foot–floor contacts and the foot positions: the duration and clearance of airborne phases, the stride between touchdowns, and the share of a foot's horizontal travel that happens while it is in contact, called the drag share. The alternation is the share of consecutive touchdowns made by different feet.

Stability and accuracy

For five robots (Ant, Go1, Go1 with its damping moved into the servos as an actuator velocity gain, the MuJoCo Humanoid, and H1 with its original torque motors) and each timestep \(h \in \{1, 2, 4, 5, 10, 20\}\) ms, 1024 worlds received the same temporally correlated random actions, \(u_t = \operatorname{clip}(0.9\,u_{t-1} + 0.45\,\xi_t, -1, 1)\) with standard normal \(\xi_t\), for 10 s. A world diverged when its state became non-finite, a velocity exceeded \(10^3\), or the root left the height band from −1 to 20 m. The explicit-damping bound \(h_\text{crit}\) predicts every damping-driven divergence (Table 2): the Go1 (\(h_\text{crit} = 14.7\) ms) is stable at 10 ms and diverges at 20 ms, and the Humanoid (\(h_\text{crit} = 10.1\) ms) diverges at 10 ms. Implicit damping removes these divergences. It does not make 20 ms safe in general: 1.9% of Humanoid worlds and about 70% of H1 worlds diverge at 20 ms with either implicit form, from stiff dynamics other than damping.

Table 2. Share of 1024 worlds that diverged within 10 s under random actions. “Implicit” is implicitfast; Euler with implicit damping gives the same shares except for H1 at 20 ms (68%). Shares above 1% are marked.

Euler with implicit damping and implicitfast give identical trajectories for these robots, because their only velocity-dependent smooth force is joint damping. When the Go1's damping is moved into its servos, only implicitfast treats it implicitly and halves the joint error. Accuracy degrades in proportion to the timestep (Figure 6): the median joint error after one 20 ms control step, measured against a 0.25 ms reference from 4096 sampled states, grows for the Go1 from 0.066° at 1 ms to 0.90° at 10 ms and 2.1° at 20 ms.

Throughput

These measurements use the first version of the harness, whose backward pass recomputes each step before applying its adjoint (Section 6). Replaying captured CUDA graphs is 2.7–4.7 times faster than launching each kernel from Python for a forward and backward step. A recomputed step with its adjoint costs 5–6 forward steps, and implicitfast costs 11–17% more than Euler, mostly in the backward pass. Between 2 and 10 ms the cost of one physics step barely changes, so 10 ms produces 4.8–4.9 times more simulated time per second than 2 ms. A SHAC epoch, however, also pays for the policy, the rewards, and the critic update, which do not shrink with the timestep: with 256 environments an epoch at 10 ms is only 2.3 (Ant) and 2.4 (Go1) times faster than at 2 ms, and 2.6–2.7 times with 1024 environments.

Gradient horizon

The step adjoint agrees with central finite differences where the dynamics are smooth: over five control steps it is within 5% of the finite difference in 73% (Humanoid) to 89% (Go1) of worlds, with median relative errors of 0.05–0.3%. In the other worlds the finite difference does not converge as the perturbation shrinks, which marks a contact activating, a stick–slip transition, or a joint limit engaging inside the perturbation. What SHAC needs is the gradient of a multi-step return. For a fixed policy and 256 start states, the return over \(T\) control steps is

with \(\chi_t^n \in \{0, 1\}\) equal to 1 until environment \(n\) terminates. The half-batch cosine compares \(\nabla_\theta J_T\) computed on two disjoint halves of the batch. It is near 1 when the gradient direction is a property of the policy and the task, and near 0 when a few chaotic or discontinuous rollouts dominate. Figure 7 shows that the timestep dominates beyond 8 control steps. For the Go1 at 2 ms the gradient stays consistent (cosine ≥ 0.97) to 32 steps, with the analytic slope equal to the finite-difference slope; at 20 ms the cosine is 0.41 at 8 steps and near zero at 16. On the fall-dominated states visited by a learning MuJoCo Humanoid, the gradient is consistent for only about four steps. Softer contacts and a fourfold solver budget did not restore consistency at 10 ms. A plausible explanation, not confirmed here, is that contact sets and stick–slip states are fixed per physics step, so a larger step changes the discrete dynamics by a larger jump that the local gradient misses. Most training therefore uses windows of 8 or 16 control steps.

Reference tasks and the timestep

Ant and Go1 were trained with SHAC on the reference tasks with explicit Euler at 2 ms and with implicitfast at 2, 5, 10, and 20 ms, three seeds each, 1000 epochs of 256 environments and an 8-step window. Only the simulator differs between configurations. Per epoch, the 2 ms configurations learned best, and explicit and implicit integration at 2 ms gave similar results (Figure 8, Table 3). The Ant's speed in the reference simulator fell from 6.3–6.4 m/s at 2 ms to 2.7 m/s at 10 ms, in the order of gradient quality. The Go1's reward saturates at 1.5 m/s, and its 2, 5, and 10 ms policies reached that speed and kept it in the reference simulator.

Highlighted + bold = best value per robot at the displayed precision. † = lost more than a quarter of its training-simulator speed in the reference simulator; not ranked. Go1 speed is not ranked because its reward targets 1.5 m/s.

Table 3. Reference tasks, deterministic evaluation over 20 s from 256 randomized starts; mean ± standard deviation over three seeds. The reference simulator is implicitfast at 1 ms. Training time is the number of epochs times the isolated median epoch time.

The 20 ms Go1 policies did not transfer: 1.55 m/s in their training simulator, 0.84 m/s with 2.6 m of lateral drift in the reference simulator, where the seed-0 policy settles into a crouch (Figure 11). A larger timestep also does not buy accuracy with more epochs: the Ant trained for 4000 epochs at 10 ms, 1.8 times the wall-clock time of a 2 ms run, reached 5.6 m/s in its training simulator but only 2.7 m/s in the reference simulator. Go1 policies trained at 10 ms or less behaved almost identically at every evaluation timestep up to 10 ms (Table A2), so below this limit the simulated dynamics are effectively converged; above it, training optimizes a different physical system.

Velocity tracking: the standing optimum

On Isaac Lab's Go1 task, which commands forward, lateral, and yaw velocities from \([-1, 1]\) (m/s or rad/s) and the Isaac Lab equivalent Go1 of Table 1, SHAC learned to stand: 95% survival and a tracking error of 0.77 m/s, close to the 0.75 m/s expected magnitude of the commanded velocity. Restricting the commands, changing the robot model, the tracking kernel, the action range, or the window, adding a trot-pattern gait prior, or a command curriculum did not help (Table A3). Proximal policy optimization (PPO), a reinforcement-learning method that uses no simulator gradients, learned the task with Isaac Lab's settings, including unbounded Gaussian actions scaled by 0.25 rad; with SHAC's scale of 0.5 rad, PPO also stood still.

Two problems were found. The first is in the robot model. The analytic adjoint occasionally gave a single world a gradient 10 to more than 100 times the median over worlds, which can dominate a batch update. The spikes come from the impedance of the Menagerie Go1 feet, which rises to its maximum \(d_\text{max} = 1\) at 23 mm of penetration. MuJoCo regularizes a contact in proportion to \((1 - d)/d\) and clamps \(d\) at 0.9999, so the constraint becomes nearly rigid. With \(d_\text{max} = 0.95\) in training (evaluation keeps the original feet), the spikes disappeared (Table A4), and SHAC learned the forward-command variant with all four seeds, against none without the change. It still stood on the full command set, with 512 and with 2048 environments (Table 4).

The second problem is in the policy parameterization. Figure 12 follows the training tracking error and the saturation of the sampled actions. Within 50 epochs, SHAC drove 52% of the action components to \(|a| > 0.95\), and 97% by the end. The mean squash derivative \(1 - a^2\), which multiplies every analytic action gradient, fell to 0.02: the joint targets sit at the ends of their ±0.5 rad range, where the gradient that could move the policy away from standing nearly vanishes. SAPO kept saturation at 2% or less and the derivative between 0.72 and 0.80, and learned the full task with both seeds (tracking error 0.05 m/s).

Table 4 separates the causes. Removing SAPO's entropy terms brought back the standing optimum, and SAPO's optimizer settings alone (activations, AdamW, clipping, learning rates, no target network) did not escape it. The state-dependent standard deviation and the second critic were not needed. Adding only the entropy terms to SHAC, or only the squash penalty, made SHAC learn the task with both seeds. The squash penalty does not reward a larger standard deviation, and with it the standard deviation fell from 0.37 to 0.11 during training, below SHAC's 0.30. Saturation therefore explains the standing optimum. The noise scale may explain the speed: with the squash penalty alone, the training tracking error fell below 0.3 m/s only after 1090 and 1217 epochs, against 200 and 215 epochs with the entropy terms, whose Gaussian part raised the standard deviation to about 0.4 early in training, and 181 with SAPO (Figure 12). Removing the tanh did not help: the unbounded actions grew to a mean magnitude of 9–29, so the joint targets lay beyond the joint ranges, where MuJoCo clamps the control and its gradient is zero.

Highlighted + bold = largest share of seeds that walk (tracking error below 0.3 m/s).

Table 4. Go1 on the full Isaac Lab command set after 1600 epochs (5 ms physics, 512 environments and a 0.16 s window unless stated, foot impedance maximum 0.95 in training unless stated), evaluated in the reference simulator. “SHAC + entropy terms” and “SHAC + squash penalty” keep all other SHAC settings.

The SAPO policy that was swept followed every tested command. Held fixed for 10 s, eleven commands, including 1 m/s backward, 0.5 m/s sideways, diagonal motion, and turning in place at 1 rad/s, were tracked with a planar error of at most 0.02 m/s and a yaw-rate error of at most 0.01 rad/s (Table A5; Figure 1). One of 64 rollouts fell at 1 m/s forward.

Reward and window

The reference reward that rewards forward speed up to a cap trained the Go1 to move, but not to walk well. Under a fixed 1 m/s command, Table 5 compares it with the mjlab reward. With the capped reward the robot ran at 1.73 m/s, above the cap, with a vertical base speed of 0.19 m/s and roll and pitch rates of 1.54 rad/s; with the mjlab reward, SHAC tracked the command with an error of 0.09 m/s at a vertical speed of 0.06 m/s and roll and pitch rates of 0.11 rad/s, and SAPO walked as steadily (Figure 13). Despite the five times stronger action-change penalty, SHAC's action change rose; SAPO lowered it by a factor of five and the tracking error from 0.09 to 0.05 m/s, at 1.1 times SHAC's epoch time. A 0.54 s window instead of 0.16 s, the closest match to the 0.53 s window of the Go2 environment that motivated the comparison, gave SAPO the same tracking error at three times the epoch time; with SHAC, one of two seeds stood still at this window after its actor gradient norm reached 2247. On the full command set, the mjlab reward gave SAPO's policies lower roll and pitch rates than the Isaac Lab reward (0.06 against 0.21 rad/s).

Highlighted + bold = best value among policies that walk.

Table 5. Go1 after 400 epochs, two seeds per row, evaluated in the reference simulator over the alive steps of 256 rollouts: base speeds in the body frame, the norm of the roll and pitch rates, the root mean square of the yaw-rate error, the action change \(\sum_j (\Delta a_j)^2\) per control step, and the root mean square joint deviation from the default pose. The capped reward has no command, so its tracking error is not defined. Windows of 0.16 and 0.54 s are 8 and 27 control steps.

Why the reference tasks fail

On the SHAC reference task, no tested configuration taught the MuJoCo Humanoid to balance. Seventeen variants, with timesteps of 2, 5, and 10 ms, windows of 8 to 32 steps, stronger actuators, a standing task, softer contacts, lower learning rates, clipped adjoints, and full-episode back-propagation, ended with mean episodes of 0.77–1.21 s, barely longer than the 0.66–0.82 s of the idle robot (Table A1). With the velocity term, the policies converged to a forward dive (Figure 14). The critic was informative, and the step adjoint was accurate, but on the falling states these policies visit, the policy gradient stays consistent for only about four control steps (Section 3). A humanoid also cannot stand by stiffening: holding H1 upright as an inverted pendulum about the ankle would need an ankle stiffness near \(mgl \approx 500\) N·m/rad, where \(m\) is the mass, \(g\) gravitational acceleration, and \(l\) the height of the center of mass, and H1's ankle provides at most 40 N·m. Balance therefore requires stepping, which changes the set of active contacts. The adjoint differentiates the solution for the current contact set and cannot represent the benefit of a foot contact that does not exist yet.

What made the humanoids learn

The Isaac Lab tasks of Section 2 changed four things for H1 and G1 together: torso–floor contact instead of a height threshold as termination, floor-only collisions (the Menagerie H1 has 127 self-collision pairs without adjoint support), initial states without penetration (half of the reference task's Humanoid resets started up to 14 cm inside the floor), and Isaac Lab's servo gains. With SHAC's original target critic (\(\alpha = 0.995\)), H1 then walked with the gait prior, and not without it (one seed, 600 epochs). With \(\alpha = 0.9\), the bootstrapped value at the window end follows the critic within a few epochs instead of about a hundred, and H1 learned 1.7–2.3 times as fast per epoch with the prior and also walked without it (Table 6, Figure 15). The MuJoCo Humanoid ran only when two changes were combined (Figure 4, Table A8): \(\alpha = 0.9\) and a joint damping of 10 N·m·s/rad, which the original SHAC simulator applies to every joint. Damping raised the half-batch cosine of a random policy's gradient at 32 control steps from 0.09 to 0.82 (Table A9).

SAPO walked on H1 without the prior as accurately as SHAC (0.10 ± 0.04 against 0.12 ± 0.02 m/s). On the G1, SHAC balanced but did not walk, both without the prior and with H1's prior weights; with stronger prior weights (\(w_h = 120\), \(w_r = 20\)) and without lateral commands it walked with two of three seeds. SAPO without the prior tracked the full command set with errors of 0.21 / 0.14 m/s. Tracking error, however, does not show how the robot walks.

Highlighted + bold = best value per robot among policies that walk (mean tracking error below 0.3 m/s). † = not ranked: its feet cross or pass into each other (Table 7).

Table 6. H1 and G1 on the Isaac Lab velocity tasks (5 ms physics, 16-step window, 512 environments), evaluated in the reference simulator from 256 randomized starts with commands drawn as in training; mean ± standard deviation over seeds. \(\alpha\) is the Polyak factor of the target critic; SAPO has no target network. All G1 rows except the SHAC row with the gait prior use the full Isaac Lab G1 command set, including lateral commands of ±0.5 m/s and 10% standing commands. H1 is commanded forward and yaw velocities only, as in Isaac Lab.

Gait quality

Under fixed commands, the per-foot measurements of Section 2 expose the difference (Table 7, Figure 16). Without the prior, both SAPO policies limp: one foot swings for 0.24 s and clears the floor by 3 cm, while the other leaves the floor for only 0.06 s at 0.4–0.5 cm and is dragged for 25–31% of its travel; at zero command their feet slide or shuffle instead of standing still (foot speed in contact up to 0.09–0.19 m/s). Without collisions between the legs, they also crossed their feet: the left foot was not to the left of the right foot in up to 38–83% of the steps of a command. The SHAC policy with the prior alternated its feet but lifted one foot lower and shorter than the other, and fell under lateral commands, on which it was not trained.

With the gait prior and SAPO on the full command set, all three seeds of the final configuration alternate their feet (alternation 1.00) with symmetric swings (symmetry 0.88–0.94), clearances of 7–8 cm, and drag shares of at most 5%. Over the fixed commands, including 0.3 m/s sideways and turning while walking, the largest planar error was 0.09 m/s; one of three seeds lost one of 16 rollouts in at least one command. At zero command, two of three seeds stand without stepping, while the other keeps tapping one foot; over 128 standing rollouts per seed, 383 of 384 stayed up. With commands drawn as in training, the tracking error was 0.10 / 0.09 / 0.10 m/s at 98.4–99.6% survival (Table 6). The same configuration with floor-only collisions, as for all other robots (two seeds), let the feet pass into each other while stepping sideways, in up to 11–13% of the steps and by up to 2 cm. Letting the foot capsules of the two feet collide, a capsule–capsule pair that the adjoint supports, reduced this to at most 0.3% without changing the gait (Table 7). With randomly drawn commands, the floor-only policies survived 100% of the rollouts at a tracking error of 0.08 m/s; physical contact between the feet thus costs a little survival and accuracy. Zeroing the clock observation for standing commands, which was also tried, made one of two seeds fall in 4% of the standing rollouts, so the final configuration keeps the clock running. Figure 2 shows a final policy (seed 2) under a command sequence, and Figure 17 compares it with a policy trained without the prior.

Highlighted + bold = best value in the column. † = not ranked: its feet cross or pass into each other. ‡ = evaluated on forward and turning commands only, which it was trained on; not ranked in the feet columns.

Table 7. Gait under a fixed forward command of 0.5 m/s in the reference simulator, 16 rollouts of 10 s measured after a 2 s transient. “Prior” is the gait prior of Section 2; “colliding feet” lets the G1's two feet collide with each other, whereas otherwise robot geometry collides only with the floor. Alternation is the share of consecutive touchdowns made by different feet; clearance is the mean peak height of the foot body above its stance height during airborne phases; the drag share is the share of a foot's horizontal travel that happens while it is in contact; symmetry is the smaller of the left/right ratios of swing duration and clearance (1 is symmetric). Crossed and overlapping feet are the largest shares, over 300 control steps of 8 rollouts per command, of steps in which the left foot is not to the left of the right foot in the heading frame, and in which the foot collision capsules of the two feet interpenetrate; the commands are 0.3 m/s to either side, 1 m/s forward, and 0.5 m/s forward while turning at 0.5 rad/s (H1: 0.5 and 1 m/s forward, turning while walking and in place; G1 with SHAC: forward and turning only).

The first version of the harness recomputed every physics step during the backward pass before applying its adjoint. The step adjoint needs only the step's input and its completed forward data, so the final trainer keeps one simulator state buffer per physics step of the window, each with its own captured forward and backward graph. On identical inputs, the action gradients of a 4-step window then differ from the recomputing backward by a relative norm of \(1.1 \cdot 10^{-2}\) for the Go1 and \(4.9 \cdot 10^{-6}\) for H1; two recomputing runs differ by as much, because the GPU's contact detection is not deterministic. The backward pass no longer runs the constraint solver, which saves the most when many physics steps make up one control step (Table 8). Together with a captured critic update, resets without host synchronization, and batched forward kinematics, the final trainer runs an epoch of the reference-task Ant and Go1 configurations 1.75 times faster at 2 ms.

Table 8. Median epoch time with this study's other jobs paused, including the critic update; 512 environments, 5 ms physics timestep, 8-step (Go1, Ant) or 16-step (humanoids) windows unless stated. The SAPO comparison is the fastest of three interleaved repeats. Rows describe different configurations, so no value is ranked.

SAPO's epoch costs 1.1 times SHAC's with the 0.16 s window and 1.3 times with the 0.54 s window; with the 0.16 s window most of the difference is its second critic, without which a SAPO epoch takes 89 ms. Proximal policy optimization needs no backward pass through the simulator and runs eight times as many environments, so it processes about 5–7 times more environment steps per second than SHAC. It also needs far more of them: 29.5 million steps for the Go1 task and 98 million for H1 walking, against 4.9 million for SHAC with the gait prior on H1 (2.3 minutes of training) and 6.6 million for SAPO on the full Go1 task.

Two trainer variants reduced the number of epochs on the H1 task. Reusing the stored per-sample action gradients for four further actor updates without simulator work, following reparameterization proximal policy optimization [Zhong et al. 2025], reached a fixed training-quality threshold after 181, 169, and 213 epochs, against 388, 315, and 285 without reuse, at the same epoch cost and with similar final policies. A 10 ms timestep with 1024 environments and reuse reached it in 0.6 minutes (Table A6).

• Seeds. The reference-task results use three seeds per configuration, and the velocity-tracking comparisons one to four; several screening rows in the appendix use one seed. Shares such as “two of two seeds” separate configurations that always or never walk, but they are not reliable success rates for configurations in between, such as SHAC with the mjlab reward (one of two) or per-world clipping on the full Go1 task (one of four).
• Mechanism. The saturation measurements of Figure 12 use one seed per configuration. The squash penalty was tested with one weight (0.3). That saturation is the cause of the standing optimum is supported by the ablation, but other effects of the entropy terms, for example on the critic targets, were not separated.
• Stability and accuracy protocol. Divergence was measured under random actions from near-standing states, and accuracy over one control step. Trained policies visit fewer states, and long-horizon differences between timesteps are dominated by chaotic divergence.
• Task fidelity. The tasks follow Isaac Lab's definitions but omit observation noise, mass randomization, pushes, and the heading controller. The Go1 servo model approximates Isaac Lab's learned actuator network. The G1 policy controls only the legs; its waist and arms hold their default pose. The Go2 environment that motivated the mjlab reward and the 0.54 s window was not reproduced: it uses a different robot, servo gains, control rate (60 Hz), and Newton's Featherstone solver.
• Robot models. The running MuJoCo Humanoid uses a joint damping of 10 N·m·s/rad, as in the original SHAC work but not in MuJoCo's model. The Go1 was trained with a foot impedance maximum of 0.95 instead of the Menagerie value of 1 and evaluated with the original feet. On the velocity-tracking tasks, except for the final G1's feet, robot geometry collides only with the floor, so legs can pass through each other; the H1 policy's feet almost never overlapped (in less than 0.1% of steps), and the Go1's legs were not checked. The gait prior fixes the step period (0.8 s) and duty factor, so the G1 cannot adapt its cadence to the commanded speed. With colliding feet, the G1 policies survived 98.4–99.6% of rollouts with random commands, and one of three seeds taps a foot when commanded to stand; three seeds do not establish a failure rate.
• Adjoint of the tested commit. The control derivative ignores actuator force limits (corrected in the harness); mesh collisions, tendons, and equality constraints are unsupported; collision pairs outside the eleven supported primitive pairs contribute no gradient without a warning; and the commit has no adjoint tests. For Euler with implicit joint damping, the velocity derivative with respect to joint positions omits the configuration dependence of the mass matrix (56% and 430% relative error on a damped two-link pendulum); the training runs used implicitfast or explicit Euler, which are not affected.
• Hardware and timing. All timings come from one MIG 1g.24gb partition, a quarter of an RTX PRO 6000 Blackwell GPU, with four CPU cores. Processes outside this study were not controlled; two timing runs gave bimodal epoch times (about 90 or 190 ms for the same configuration) and were discarded.

The environment is the pull request's locked Python environment (uv sync at commit 357a75d) with PyTorch 2.11.0 for CUDA 12.8 added. Go1 and H1 come from MuJoCo Menagerie at 2cd6b7b; the Ant, Humanoid, Go1-kv, and H1-PD models are included under implicit/models/, and the G1 model from mjlab under tuning/models/g1_mjlab/. The main entry points are listed below; the velocity-tracking commands write to runs/loop/ and results/loop/, whose published copies are under tuning/checkpoints/ and tuning/results/.

# integrator stability and one-step accuracy (Section 3), per robot
python integrator_sweep.py --robot go1 --nworld 1024 --duration 10 --out results/e1_go1.json
# simulator and SHAC epoch throughput (Sections 3 and 6)
python bench.py --eager --out results/bench_dt10.json
python bench.py --integrators implicitfast --nworlds 1024 --dt 0.002 --substeps 10 --out results/bench_dt2.json
python bench.py --integrators implicitfast --nworlds 1024 --dt 0.02 --substeps 1 --out results/bench_dt20.json
python time_epochs.py --robots ant go1 --num-envs 256 1024 --out results/epoch_timing.json
# policy-gradient quality for a fixed trained policy (Section 3)
python run_e3.py --robots ant go1 --checkpoint-config explicit_2ms
python run_e3.py --robots humanoid --checkpoint checkpoints/HA_ref_2ms_T16.pt --warmup 10
# reference-task training matrix, time-matched runs, and evaluation (Section 4)
python run_matrix.py --robots ant go1 --epochs 1000 --horizon 8 --tag e4
python run_matrix.py --robots ant go1 --configs implicit_10ms --seeds 0 1 --epochs 4000 --tag e5
python eval_matrix.py --tag e4 && python eval_matrix.py --tag e5
python summarize_e4.py --tag e4 && python summarize_e4.py --tag e5
python transfer_matrix.py
# velocity-tracking tasks (Sections 4 and 5), run from tuning/: training, reference evaluation, sweeps, gait, saturation
python shac2.py --task go1_vel --out runs/loop/ilfull_sapo_T8_s0 --set num_envs=512 horizon=8 epochs=1600 dt=0.005 seed=0 sapo=true geom_solimp=[0.015,0.95,0.023,0.5,2]
python shac2.py --task go1_vel --out runs/loop/abl_squash03_s0 --set num_envs=512 horizon=8 epochs=1600 dt=0.005 seed=0 squash_reg=0.3 geom_solimp=[0.015,0.95,0.023,0.5,2]
python shac2.py --task go1_rw_full --out runs/loop/rwfull_sapo_T8_s0 --set num_envs=512 horizon=8 epochs=1600 dt=0.005 seed=0 sapo=true geom_solimp=[0.015,0.95,0.023,0.5,2]
python shac2.py --task h1_vel --out runs/loop/h1vel_sapo_s0 --set num_envs=512 horizon=16 epochs=1500 dt=0.005 seed=0 sapo=true
python shac2.py --task g1_walk --out runs/loop/g1walk3_sapo_s2 --set num_envs=512 horizon=16 epochs=2000 dt=0.005 seed=2 sapo=true nconmax=32 njmax=160 model_edit=floor_only+g1_feet3+g1_il+g1_footpair
python gait_eval.py --ckpt runs/loop/g1walk3_sapo_s2.pt --out results/loop/gait/g1walk3_sapo_s2.json
./foot_sep_eval.sh g1walk3_sapo_s2 floor_only+g1_feet3+g1_il+g1_footpair
./make_g1_videos.sh runs/loop/g1walk3_sapo_s2.pt runs/loop/g1vel_sapo_s0.pt
python shac2.py --task h1_gait --out runs/loop/a11_h1gait_ta09 --set num_envs=512 horizon=16 epochs=600 dt=0.005 target_alpha=0.9
python shac2.py --task humanoid_clean --out runs/loop/hd3_humref_d10_2ms_T16_ta09 --set num_envs=512 horizon=16 epochs=800 dt=0.002 model_edit=floor_only+damp10 nconmax=24 njmax=96 target_alpha=0.9 --tset robot=humanoid_ref
python eval2.py --sim ref --ckpt runs/loop/ilfull_sapo_T8_s0.pt --out results/loop/eval/ilfull_sapo_T8_s0.json
python cmd_sweep.py --ckpt runs/loop/rwfull_sapo_T8_s0.pt --out results/loop/cmd_rwfull_sapo_T8_s0.json
python gait_eval.py --ckpt runs/loop/h1vel_sapo_s0.pt --out results/loop/gait/h1vel_sapo_s0.json
python sat_check.py --runs s95_go1vel_s0 abl_noent_s0 abl_shacent_s0 abl_squash03_s0 ilfull_sapo_T8_s0 --epochs 0 50 100 150 200 300 400 600 800 1000 1200 1400 1550 --out results/loop/sat_curve.json
python ppo.py --task go1_vel --out runs/loop/ppo_go1vel_5ms_s025 --set iterations=300 dt=0.005 --tset action_scale=0.25
python time_suite.py --suite suites/timing9.json --out results/loop/timing9.json --epochs 40 --skip 10

• Source: simulator wrapper, environments, SHAC, robot definitions, and the experiment drivers listed in the artifact manifest, which gives the size and SHA-256 hash of every published file.
• Results: integrator sweep, simulator throughput, SHAC epoch timing, training summary, cross-evaluation, humanoid variants, and gradient checks under implicit/results/e3/. Per-epoch training histories are stored gzip-compressed under implicit/results/runs/.
• Checkpoints of the seed-0 policies and of the humanoid runs are under implicit/checkpoints/ (see the manifest).
• Velocity-tracking study (Sections 4–6): trainer, tasks, forward kinematics, simulator wrapper with stored forward steps, robot models and edits, PPO baseline, evaluation scripts, results under tuning/results/, training histories under tuning/results/runs/, and selected checkpoints under tuning/checkpoints/, all listed in the manifest.
• Figure data, humanoid training curves, and Go1 and gait figure data used by the interactive plots.
• Earlier version of this report (hybrid policies, pull request commit 02d09b1, explicit damping), with its artifacts under pr1535/.