Videos to Forces: Physics Based Climbing Control
From Online Videos

Anonymous submission

Full Motion Rollouts

Finals of the Boulder Japan Cup 2024, where many of the climbers rank in the world's top 20.

Each climb is trained and evaluated in segments to save compute, though nothing prevents running it whole. The motion can jump at segment boundaries, which are colored in orange on the video timeline. Overlapping segments by about a second would smooth them.

Bodies are drawn as their separate collision shapes, so hand and foot contacts stay visible through the gaps from any angle. Drawing a skinned body instead would be a simple swap.

The wall meshes are under copyright and cannot be published, so the 3D viewer shows each wall as images rendered every 7.5°. Dragging steps between these views instead of rotating freely, within 60° to either side and 30° above; zooming and panning work as usual.

Men's final · Boulder 1

climber 3
Broadcast video
Simulation · drag to rotate, two-finger drag to moveSegment boundary
0:00

Men's final · Boulder 2

climber 2
Broadcast video
Simulation · drag to rotate, two-finger drag to moveSegment boundary
0:00

Men's final · Boulder 3

climber 4
Broadcast video
Simulation · drag to rotate, two-finger drag to moveSegment boundary
0:00

Women's final · Boulder 1

climber 1
Broadcast video
Simulation · drag to rotate, two-finger drag to moveSegment boundary
0:00

Women's final · Boulder 2

climber 5
Broadcast video
Simulation · drag to rotate, two-finger drag to moveSegment boundary
0:00

Women's final · Boulder 3

climber 5
Broadcast video
Simulation · drag to rotate, two-finger drag to moveSegment boundary
0:00

Women's final · Boulder 4

climber 6
Broadcast video
Simulation · drag to rotate, two-finger drag to moveSegment boundary
0:00

Reprojection Errors

Overlays of each step of our pipeline against the broadcast video.

Click on each tab to see the overlays on different walls.

Source video
CoMotion
Cleaned reference
Ours
frame 103

Comparisons

Each climb runs in segments (marks on the timeline). A segment whose training reaches the 48 h limit is evaluated at its last checkpoint before the limit, marked “timed out”. If a segment completes training but still falls, it will be marked “fell”, in which case the frame is frozen until the next segment and starts again from that segment's first pose.

Residual Force Control (RFC) was trained on one segment per climb; switch to that segment to see all three methods. Unlike our method, RFC has collision and residual force on from the start; it has the same curriculum that drives the residual force to zero as our method.

Click on each tab to see the comparisons on different walls.

Ours
fell
DeepMimic
fell
RFC
0:00

Force Validation

We use the LAAS Parkour dataset, a mocap dataset with measured end-effector forces, to validate our policy's contact forces.

Kong vault

Ground truth
Ours, no guidance
Ours, with guidance
Kong vault: contact forces per hand and toe, and in total, measured vs ours without and with guidance
Contact forces (red line: current frame)
frame 0
4-point contact (hands and feet)2-point contact (hands) measured (ground truth)ours, no guidanceours, with guidance

Safety vault

Ground truth
Ours, no guidance
Ours, with guidance
Safety vault: contact forces per hand and toe, and in total, measured vs ours without and with guidance
Contact forces (red line: current frame)
frame 0
1-point contact2-point contactflight measured (ground truth)ours, no guidanceours, with guidance