PyBullet MCP Server

A Model Context Protocol (MCP) server that enables AI assistants to interact with PyBullet physics simulations. Build physics-based projects through natural language interactions with AI agents.

Features

• 37 MCP Tools: Comprehensive API for physics simulation control including robot joint control
• Simulation Management: Create and manage multiple independent physics simulations with configurable gravity
• Object Manipulation: Add primitive shapes (box, sphere, cylinder, capsule) and URDF models with full property control
• Robot Control: Query joint information, control motors (position/velocity/torque), and calculate inverse kinematics
• Physics Control: Apply forces, torques, and step through simulations with configurable timesteps
• State Persistence: Save and load complete simulation states to/from JSON files
• Constraints: Create joints between objects (fixed, prismatic, spherical)
• Collision Detection: Query contact points with detailed collision information
• Visualization: Optional GUI mode with debug visualization and camera control
• Error Handling: Comprehensive validation with descriptive error messages

Important Notes

• Coordinate Requirements: All vectors must be provided as complete 3D coordinates [x, y, z]
  • Gravity: [0, 0, -9.81] (not [-9.81])
  • Positions: [x, y, z] (not [x, y] or [x])
  • Forces/Torques: [fx, fy, fz] (not [fx])
  • Orientations: [x, y, z, w] quaternion (not [w])
• Mass Constraint: Object mass must be positive (mass > 0). Use large mass (e.g., 1000) for static objects
• GUI Limitation: Only one GUI simulation can be active at a time (PyBullet limitation)
• URDF Paths: Use absolute paths or paths relative to the server's working directory
• Revolute Joints: create_constraint does not support "revolute" — use generate_revolute_joint + load_urdf instead. Always pass an explicit output_path inside the workspace to generate_revolute_joint, otherwise the generated URDF lands in /tmp/ and load_urdf will reject it.

Installation

Prerequisites

• Python 3.9 or higher
• pip package manager
• Virtual environment (recommended)

Install Dependencies

1. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install FastMCP and PyBullet:

pip install fastmcp pybullet

3. For development (includes testing tools):

pip install fastmcp pybullet pytest hypothesis pytest-cov

Verify Installation

Check that the required packages are installed:

python -c "import pybullet; import mcp; print('Installation successful!')"

Quick Start

Running the Server

See QUICKSTART.md for detailed instructions on:

• Starting the server
• Configuring Claude Desktop
• Example prompts to explore all features
• Common workflows and use cases

Quick command:

source venv/bin/activate  # On Windows: venv\Scripts\activate
python -m src.server

The server will start and listen for MCP protocol connections from AI assistants.

Example Usage with MCP Client

Once connected to an MCP client (like Claude Desktop), you can interact through natural language:

1. Create a simulation:

Create a new physics simulation with Earth gravity

This calls create_simulation with gravity [0, 0, -9.81].

Important: Gravity must be a 3D vector. The server accepts shorthand like [-9.81] and expands it to [0, 0, -9.81].

2. Add objects:

Add a red box at position (0, 0, 1) with dimensions 0.5x0.5x0.5 and mass 1.0

This calls add_box to create a box object. Position is automatically expanded to [0, 0, 1].

Add a sphere at (2, 0, 1) with radius 0.3

This calls add_sphere to create a sphere.

Note: Mass must be positive. For static objects (like ground planes), use large mass values (e.g., 1000).

3. Run the simulation:

Step the simulation forward 100 times

This calls step_simulation with steps=100.

4. Query object state:

What is the position and velocity of object 0?

This calls get_object_state to retrieve position, orientation, and velocities.

5. Apply forces:

Apply a force of [10, 0, 0] to object 0

This calls apply_force to push the object.

6. Save the simulation:

Save the current simulation state to simulation.json

This calls save_simulation to persist the state.

7. Load a simulation:

Load the simulation from simulation.json

This calls load_simulation to restore the saved state.

MCP Configuration

To use this server with Cursor or any other MCP-compatible client, add the following to your MCP configuration file:

{
  "mcpServers": {
    "pybullet": {
      "url": "http://localhost:8000/mcp",
      "disabled": false
    }
  }
}

The server runs HTTP transport by default. You can change the transport method by editing the entry point in server.py:

if __name__ == "__main__":
    mcp.run(transport="http", port=8000)

Restart your MCP client after updating the configuration.

Available Tools

The server exposes 37 tools through the MCP protocol:

Simulation Management (5 tools)

• create_simulation: Initialize a new physics simulation with configurable gravity and optional GUI

  • Parameters: gravity (list[float], default: [0, 0, -9.81]), gui (bool, default: false)
  • Returns: simulation_id, gravity, gui_enabled

• list_simulations: Get all active simulation IDs

  • Returns: list of simulation IDs

• destroy_simulation: Clean up and remove a simulation

  • Parameters: sim_id (str)
  • Returns: confirmation message

• step_simulation: Advance simulation by one or more timesteps

  • Parameters: sim_id (str), steps (int, default: 1)
  • Returns: simulation_id, steps_taken, current_time

• set_timestep: Configure the timestep duration for a simulation

  • Parameters: sim_id (str), timestep (float)
  • Returns: confirmation message

Object Operations (9 tools)

• add_box: Add a box shape to the simulation

  • Parameters: sim_id, dimensions (list[float]), position (list[float]), mass (float, default: 1.0), color (list[float], optional)
  • Returns: object_id, shape, position

• add_sphere: Add a sphere shape to the simulation

  • Parameters: sim_id, radius (float), position (list[float]), mass (float, default: 1.0), color (list[float], optional)
  • Returns: object_id, shape, position

• add_cylinder: Add a cylinder shape to the simulation

  • Parameters: sim_id, radius (float), height (float), position (list[float]), mass (float, default: 1.0), color (list[float], optional)
  • Returns: object_id, shape, position

• add_capsule: Add a capsule shape to the simulation

  • Parameters: sim_id, radius (float), height (float), position (list[float]), mass (float, default: 1.0), color (list[float], optional)
  • Returns: object_id, shape, position

• load_urdf: Load a robot model from a URDF file

  • Parameters: sim_id, file_path (str), position (list[float]), orientation (list[float], optional)
  • Returns: object_id, file_path, position

• set_object_pose: Update object position and orientation

  • Parameters: sim_id, object_id (int), position (list[float]), orientation (list[float])
  • Returns: confirmation message

• get_object_state: Query complete object state

  • Parameters: sim_id, object_id (int)
  • Returns: position, orientation, linear_velocity, angular_velocity

• apply_force: Apply a force vector to an object

  • Parameters: sim_id, object_id (int), force (list[float]), position (list[float], optional)
  • Returns: confirmation message

• apply_torque: Apply rotational force to an object

  • Parameters: sim_id, object_id (int), torque (list[float])
  • Returns: confirmation message

• set_object_velocity: Set an object's linear and/or angular velocity directly

  • Parameters: sim_id, object_id (int), linear_velocity (list[float], optional), angular_velocity (list[float], optional)
  • Returns: confirmation message

• change_dynamics: Modify object physics properties at runtime

  • Parameters: sim_id, object_id (int), link_index (int, default: -1), mass (float, optional), lateral_friction (float, optional), spinning_friction (float, optional), rolling_friction (float, optional), restitution (float, optional), linear_damping (float, optional), angular_damping (float, optional), contact_stiffness (float, optional), contact_damping (float, optional)
  • Returns: confirmation message

• get_dynamics_info: Query current dynamic properties of an object

  • Parameters: sim_id, object_id (int), link_index (int, default: -1)
  • Returns: mass, lateral_friction, local_inertia_diagonal, restitution, rolling_friction, spinning_friction, contact_damping, contact_stiffness, body_type, collision_margin

Ray Casting (2 tools)

• ray_test: Cast a single ray to detect obstacles and measure distances

  • Parameters: sim_id, ray_from (list[float]), ray_to (list[float])
  • Returns: hit (bool), object_id, link_index, hit_fraction, hit_position, hit_normal

• ray_test_batch: Cast multiple rays efficiently for lidar/sensor simulation

  • Parameters: sim_id, rays_from (list[list[float]]), rays_to (list[list[float]])
  • Returns: list of hit results (same fields as ray_test per ray)

Camera Rendering (4 tools)

• compute_view_matrix: Compute view matrix from camera eye/target/up vectors

  • Parameters: camera_eye_position (list[float]), camera_target_position (list[float]), camera_up_vector (list[float])
  • Returns: view matrix as list of 16 floats

• compute_view_matrix_from_yaw_pitch: Compute view matrix from spherical coordinates (orbit camera)

  • Parameters: distance (float), yaw (float), pitch (float), target_position (list[float]), up_axis_index (int, default: 2)
  • Returns: view matrix as list of 16 floats

• compute_projection_matrix: Compute projection matrix from camera parameters

  • Parameters: fov (float), aspect (float), near_plane (float), far_plane (float)
  • Returns: projection matrix as list of 16 floats

• get_camera_image: Render RGB, depth, and segmentation images from a virtual camera

  • Parameters: sim_id, width (int), height (int), view_matrix (list[float]), projection_matrix (list[float]), renderer (str, default: "ER_BULLET_HARDWARE_OPENGL")
  • Returns: width, height, rgb (base64 PNG), depth (list[float]), segmentation (list[int])

Constraint Management (2 tools)

• create_constraint: Create a joint between two objects

  • Parameters: sim_id, parent_id (int), child_id (int), joint_type (str), joint_axis (list[float], optional), parent_frame_position (list[float], optional), child_frame_position (list[float], optional)
  • Joint types: "fixed", "prismatic", "spherical"
  • Returns: constraint_id, joint_type
  • Note: "revolute" is NOT supported here — use generate_revolute_joint + load_urdf instead.

• remove_constraint: Remove a constraint from the simulation

  • Parameters: sim_id, constraint_id (int)
  • Returns: confirmation message

Revolute (Hinge) Joints — Workaround

PyBullet's runtime constraint API does not support revolute joints. The workaround is a two-step process using these two tools:

Step 1 — Generate a URDF with the revolute joint:

• generate_revolute_joint: Generates a URDF file containing two shapes connected by a revolute joint
  • Parameters: parent_shape (str: "box"/"sphere"/"cylinder"), child_shape (str), parent_dimensions (list[float]), child_dimensions (list[float]), parent_mass (float), child_mass (float), joint_axis (list[float]), joint_origin (list[float], optional), joint_lower_limit (float, default: -π), joint_upper_limit (float, default: π), max_effort (float), max_velocity (float), output_path (str)
  • Returns: urdf_path, parent_shape, child_shape, joint_type, joint_axis, joint_limits
  • output_path must be set to a path inside the server's working directory. If omitted, the file is written to the system temp directory (/tmp/) which is outside the allowed path and will cause load_urdf to fail with an access denied error.

Step 2 — Load the generated URDF:

• Call load_urdf with the urdf_path returned from generate_revolute_joint
• Then use set_joint_motor_control to drive the joint

Collision Detection (2 tools)

• get_all_collisions: Query all contact points in the simulation

  • Parameters: sim_id
  • Returns: list of contact points with positions, normals, forces

• get_collisions_for_pair: Query contact points between specific objects

  • Parameters: sim_id, obj_a (int), obj_b (int)
  • Returns: list of contact points for the pair

Visualization (2 tools)

• enable_debug_visualization: Enable debug rendering of contact points and frames

  • Parameters: sim_id, show_contacts (bool, default: true), show_frames (bool, default: false)
  • Returns: confirmation message

• set_camera: Configure camera position and target for GUI mode

  • Parameters: sim_id, distance (float), yaw (float), pitch (float), target (list[float])
  • Returns: confirmation message

Persistence (2 tools)

• save_simulation: Save simulation state to a JSON file

  • Parameters: sim_id, file_path (str)
  • Returns: confirmation with file path

• load_simulation: Load simulation state from a JSON file

  • Parameters: file_path (str), gui (bool, default: false)
  • Returns: new simulation_id, file_path

Robot Control (5 tools)

• get_num_joints: Query number of joints in a URDF model

  • Parameters: sim_id (str), object_id (int)
  • Returns: number of joints (int)

• get_joint_info: Get detailed joint properties

  • Parameters: sim_id (str), object_id (int), joint_index (int)
  • Returns: joint_name, joint_type, lower_limit, upper_limit, max_force, max_velocity, joint_axis

• get_joint_state: Get current joint state

  • Parameters: sim_id (str), object_id (int), joint_index (int)
  • Returns: joint_position, joint_velocity, reaction_forces, motor_torque

• set_joint_motor_control: Control robot joints

  • Parameters: sim_id (str), object_id (int), joint_index (int), control_mode (str), target_position (float, optional), target_velocity (float, optional), force (float, optional), position_gain (float, optional), velocity_gain (float, optional)
  • Control modes: "POSITION_CONTROL", "VELOCITY_CONTROL", "TORQUE_CONTROL"
  • Returns: confirmation message

• calculate_inverse_kinematics: Calculate joint angles for target end-effector pose

  • Parameters: sim_id (str), object_id (int), end_effector_link_index (int), target_position (list[float]), target_orientation (list[float], optional), lower_limits (list[float], optional), upper_limits (list[float], optional), joint_ranges (list[float], optional), rest_poses (list[float], optional)
  • Returns: list of joint positions

Example Workflows

Basic Falling Box

Create a simple simulation with a box falling under gravity:

# Through MCP client (natural language):
"Create a simulation with Earth gravity"
"Add a box at position (0, 0, 5) with dimensions 1x1x1"
"Step the simulation 200 times"
"What is the position of object 0?"

Stacked Objects

Create a stack of objects:

"Create a simulation"
"Add a box at (0, 0, 0.5) with dimensions 10x10x1 and mass 1000"  # Ground
"Add a box at (0, 0, 1.5) with dimensions 1x1x1"
"Add a sphere at (0, 0, 3) with radius 0.5"
"Step the simulation 300 times"
"Get all collisions"

Constrained Objects

Create objects connected by a joint:

"Create a simulation"
"Add a box at (0, 0, 2) with dimensions 1x1x1"  # Object 0
"Add a sphere at (2, 0, 2) with radius 0.5"     # Object 1
"Create a fixed constraint between object 0 and object 1"
"Apply a torque of [0, 0, 10] to object 1"
"Step the simulation 200 times"

Save and Load

Persist a simulation:

"Create a simulation"
"Add a box at (0, 0, 1)"
"Add a sphere at (1, 0, 1)"
"Step the simulation 50 times"
"Save the simulation to my_sim.json"

# Later...
"Load the simulation from my_sim.json"
"Step the simulation 50 more times"

Robot Simulation

Load and control a URDF robot model:

"Create a simulation with GUI enabled"
"Load URDF from /path/to/robot.urdf at position (0, 0, 1)"
"How many joints does object 0 have?"
"Get information about joint 0 of object 0"
"Get the current state of joint 0"
"Set joint 0 to position 1.57 with position control and force 100"
"Step the simulation 100 times"
"Get the state of joint 0 again to see it moved"
"Calculate inverse kinematics for object 0 end-effector link 6 at position [0.5, 0, 0.5]"

Robot Control Features:

• Query number of joints and joint properties (type, limits, axis)
• Read joint states (position, velocity, forces, torque)
• Control joints with three modes:
  • Position control: Move joint to target angle/position
  • Velocity control: Spin joint at target speed
  • Torque control: Apply direct torque to joint
• Calculate inverse kinematics for end-effector positioning

Persistence File Format

Simulation states are saved as JSON files with the following structure:

{
  "gravity": [0.0, 0.0, -9.81],
  "timestep": 0.01,
  "objects": [
    {
      "object_id": 0,
      "type": "primitive",
      "shape": "box",
      "dimensions": [0.5, 0.5, 0.5],
      "position": [0.0, 0.0, 1.0],
      "orientation": [0.0, 0.0, 0.0, 1.0],
      "linear_velocity": [0.0, 0.0, -0.98],
      "angular_velocity": [0.0, 0.0, 0.0],
      "mass": 1.0,
      "color": [1.0, 0.0, 0.0, 1.0]
    },
    {
      "object_id": 1,
      "type": "urdf",
      "urdf_path": "/path/to/model.urdf",
      "position": [2.0, 0.0, 0.5],
      "orientation": [0.0, 0.0, 0.0, 1.0],
      "linear_velocity": [0.0, 0.0, 0.0],
      "angular_velocity": [0.0, 0.0, 0.0]
    }
  ],
  "constraints": [
    {
      "constraint_id": 0,
      "parent_id": 0,
      "child_id": 1,
      "joint_type": "fixed"
    }
  ]
}

Compatibility Notes

• Object IDs: Object IDs are reassigned when loading (may differ from saved IDs)
• URDF Files: URDF file paths must be valid when loading. Use absolute paths for reliability
• Simulation ID: A new simulation ID is generated when loading
• Format Version: Current format is compatible with PyBullet 3.2.5+
• Constraints: Constraints are fully serialized and restored

Common Issues and Solutions

Coordinate Errors

Problem: "force needs a 3 coordinates [x,y,z]" or similar errors

Solution: Always provide complete 3D vectors:

# Wrong
gravity = [-9.81]
position = [1, 2]
force = [10]

# Correct
gravity = [0, 0, -9.81]
position = [1, 2, 0]
force = [10, 0, 0]

Mass Validation Error

Problem: "Mass must be positive, got 0.0"

Solution: PyBullet requires positive mass. For static objects, use large mass:

# Wrong
add_box(mass=0)  # Error!

# Correct
add_box(mass=1000)  # Heavy static object

GUI Window Not Appearing

Problem: GUI window doesn't show when gui=true

Solution:

• Only one GUI simulation is allowed per server instance
• Destroy the existing GUI simulation before creating a new one
• Some environments (Docker, SSH without X11) don't support GUI
• On Linux, ensure X11 is available: echo $DISPLAY

URDF Loading Fails

Problem: "Failed to load URDF: File not found"

Solution:

• Use absolute paths: /full/path/to/robot.urdf
• Or use paths relative to the server's working directory
• Verify the file exists: ls -la /path/to/robot.urdf
• Check that mesh files referenced in the URDF are also accessible

Objects Fall Through Ground

Problem: Objects pass through the ground plane

Solution:

• Ensure you're stepping the simulation: step_simulation(steps=100)
• Use an appropriate timestep (default 0.01 is usually good)
• Give the ground plane large dimensions and high mass (e.g., 1000)
• Verify objects have positive mass

Troubleshooting

Common Issues

1. Import errors when starting the server

ModuleNotFoundError: No module named 'mcp' or 'pybullet'

Solution: Ensure your virtual environment is activated and dependencies are installed:

source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install fastmcp pybullet

2. Python version errors

SyntaxError or version compatibility issues

Solution: Check Python version (must be 3.9+):

python --version

If needed, create a venv with a specific Python version:

python3.9 -m venv venv

3. PyBullet GUI not showing

GUI window doesn't appear when gui=true

Solution:

• GUI mode must be explicitly enabled: create_simulation(gui=true)
• Some systems (servers, Docker) don't support GUI mode - use headless mode instead
• On Linux, ensure X11 is available: echo $DISPLAY

4. File permission errors

PermissionError when saving/loading simulations

Solution:

• Ensure write permissions in the target directory
• Use absolute paths: /full/path/to/simulation.json
• Check disk space: df -h

5. MCP client can't connect to server

Server not responding

Solution:

• Verify the server is running: python -m src.server
• Check the URL in your MCP config matches the server port (default: http://localhost:8000/mcp)
• Restart your MCP client after config changes
• Check the server terminal output for error messages

6. Simulation behaves unexpectedly

Objects fall through the ground or constraints don't work

Solution:

• Ensure you're stepping the simulation: step_simulation(sim_id, steps=100)
• Check timestep value (default 0.01 is usually good)
• Verify object masses are positive
• For ground planes, use a box with large dimensions and mass=1000

7. URDF loading fails

ToolError: Failed to load URDF

Solution:

• Verify the URDF file path is correct and accessible
• Use absolute paths for URDF files
• Check that mesh files referenced in URDF are also accessible
• Validate URDF syntax with PyBullet directly

Debug Mode

To see detailed error messages, check the server output in your terminal. The server logs all operations and errors.

Getting Help

If you encounter issues not covered here:

1. Check the server logs for detailed error messages
2. Verify your MCP client configuration
3. Test the server directly with Python to isolate MCP vs. PyBullet issues
4. Review the PyBullet documentation for physics-specific questions

Architecture

The server follows a layered architecture:

MCP Client (Claude Desktop)
         ↓
   MCP Protocol
         ↓
FastMCP Server (37 tools)
         ↓
Manager Classes (helpers)
         ↓
PyBullet Physics Engine

Key Components:

• FastMCP Server (src/server.py): Exposes 37 MCP tools using @mcp.tool decorators
• SimulationManager: Manages PyBullet physics clients and simulation lifecycle
• ObjectManager: Handles object creation, manipulation, and state queries
• ConstraintManager: Creates and manages joints between objects
• PersistenceHandler: Serializes/deserializes simulation state to JSON
• CollisionQueryHandler: Queries contact points and collision information

Each MCP tool validates inputs, calls the appropriate manager, and returns Python objects (dict/list/str/int) directly - FastMCP handles the MCP protocol conversion automatically.

Contributing

Contributions are welcome! Please ensure:

• All tests pass: pytest
• Code is formatted: black src tests
• Code passes linting: ruff check src tests
• Type hints are correct: mypy src
• New features include tests (unit + property-based)
• Documentation is updated

Acknowledgments

• Built with FastMCP by Jeremiah Lowin
• Physics simulation powered by PyBullet
• Property-based testing with Hypothesis

Support

For issues, questions, or contributions:

• Review this README and troubleshooting section
• Review PyBullet documentation for physics-specific questions