Skip to main content
Version: ROS 2 Jazzy

Vision Pack Integration

This tutorial will guide you through the process of connecting the Vision Pack to your Leo Rover. The Vision Pack is a hardware bundle that includes a ZED camera and a Jetson Orin Nano, designed to enhance the visual capabilities of your Leo Rover. By following this tutorial, you will learn how to integrate these components with your rover system, enabling advanced vision processing and perception capabilities.

What to expect?​

After completing this tutorial, you will be able to run a Docker container with ROS Jazzy and ZED wrapper on your Jetson Orin Nano, and have the camera data published on ROS topics.

To see full information about the additional available ROS interfaces and configuration options, check the official ZED ROS wrapper documentation.

Prerequisites​

📄Connect to Leo Rover via SSH
Guide on how to connect to your Leo Rover using SSH for remote access
📄Connect to network
Guide on how to connect your Leo Rover to a Wi-Fi network, enabling internet access
📄ROS Development
Detailed guide on ROS development for Leo Rover, covering topics like adding additional functionalities, building ROS packages and more.

Setting up the Jetson Orin Nano​

info

It is recommended to set up the Jetson before mounting it on the rover. In that case you can use the Jetson with a regular monitor, keyboard and mouse, which makes the setup process easier.

In order to set up the Jetson Orin Nano, you will need to flash the official NVIDIA image on it and update the firmware. Follow the steps below to complete the setup:

  1. Go through the official Jetson Orin Nano setup guide.
  2. During flashing, install JetPack 6.2 with L4T firmware 36.4.0 (or newer).
  3. Ensure the Jetson has internet access (either via Ethernet or Wi-Fi) before proceeding.
  4. Finish the initial setup and create a user account when prompted.
  5. Ensure that your Jetson is running in MAXN SUPER mode for optimal performance.
warning

JetPack 7.0 and 7.1 are not supported by Jetson Orin Nano, so make sure to select a supported version for flashing.

After the initial setup is complete, install the latest packages and the NVIDIA Container Toolkit runtime on the Jetson. Run the following commands directly on the Jetson (either with an attached display or over SSH):

sudo apt update && sudo apt upgrade
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Now you can turn off Wi-Fi on Jetson since later on it will be connected to the rover's network.

At this point you can proceed with the hardware integration and mounting the Jetson on the rover.

Hardware integration​

Mounting​

Wiring​

Software integration​

In order to use the ZED camera with ROS, we will be using the official ZED ROS wrapper in a custom Docker container running on the Jetson.

Follow the steps below to bring the Vision Pack online and publish ZED data to ROS.

Sharing Leo Rover network with the Jetson​

Since Jetson ethernet port is connected to the rover via a USB-to-Ethernet adapter, a new ethernet interface is created and it needs to be configured to bridge the rover's network to the connected device.

To do that, log in to the Raspberry Pi via ssh and open the Netplan configuration file that manages the ethernet bridge:

sudo nano /etc/netplan/10-eth-bridge.yaml

Add the eth1 interface configuration. The final file should look like this:

/etc/netplan/10-eth-bridge.yaml
# Creates a bridge interface and adds ethernet interface to it.
network:
  version: 2
  ethernets:
    eth0:
      renderer: networkd
      optional: true
    eth1:
      renderer: networkd
      optional: true
  bridges:
    br0:
      renderer: networkd
      interfaces: [eth0, eth1]
      addresses: [10.0.0.1/24]
      link-local: []

Apply the new configuration:

sudo netplan apply

Now your Jetson should receive an IP address in the 10.0.0.x range and be accessible from the rover network.

Creating ZED camera ROS configuration​

Before launching the ZED wrapper container, you need to create a configuration file for the camera. This file will contain the parameters for the ZED camera and will be mounted into the container to configure the ROS node.

On Jetson, create folders for configs and logs on the host:

sudo mkdir -p /opt/zed_ros/{config,logs}
sudo chown -R $USER /opt/zed_ros

Add a parameter file under /opt/zed_ros/config:

/opt/zed_ros/config/zed2i.yaml
/**:
  ros__parameters:
    general:
      camera_model: 'zed2i'
      camera_name: 'zed2i'
      grab_resolution: 'HD1080'
      grab_frame_rate: 15
    video:
      brightness: 4
      contrast: 4
      hue: 0
    depth:
      min_depth: 0.01
      max_depth: 15.0
    pos_tracking:
      publish_tf: false
note

These are just example parameters - tune them as needed and refer to the ZED ROS wrapper documentation for the full list of options.

Setting up the ZED wrapper container​

Pull the prebuilt image with ROS Jazzy and ZED wrapper from our registry onto the Jetson:

sudo docker pull ghcr.io/fictionlab/jetson_ros_jazzy_zed:latest
note

Ensure that your Jetson has internet access to pull the image. Further internet access might be required on camera startup if the ZED wrapper needs to download additional assets like calibration files or detection models.

Start the ZED wrapper container whenever you need the camera online. This run command exposes the GPU, host networking, and the ZED calibration assets to the container:

sudo docker run --rm --runtime nvidia -it --privileged --network=host --ipc=host --pid=host \
  -e NVIDIA_DRIVER_CAPABILITIES=all -e DISPLAY=$DISPLAY \
  -v /tmp/.X11-unix/:/tmp/.X11-unix \
  -v /dev:/dev \
  -v /dev/shm:/dev/shm \
  -v /usr/local/zed/resources/:/usr/local/zed/resources/ \
  -v /usr/local/zed/settings/:/usr/local/zed/settings/ \
  -v /opt/zed_ros/config:/data:ro \
  ghcr.io/fictionlab/jetson_ros_jazzy_zed:latest

Once you see the container shell prompt, source both the ROS installation and the workspace overlay, then launch the wrapper for the zed2i camera using the host configuration mounted at /data/zed2i.yaml:

source /opt/ros/jazzy/install/setup.bash
source /root/ros2_ws/install/setup.bash
ros2 launch zed_wrapper zed_camera.launch.py camera_model:=zed2i ros_params_override_path:=/data/zed2i.yaml

At this point the rover computer should see the ZED ROS interfaces.

note

When launching ZED for the first time it might need some time to initialize the camera and generate the calibration files. Because of that, the topics might not be available immediately.

Modifying the URDF to include the Vision Pack​

To ensure proper TF frames and integration with the rest of the rover system, you will need to add the ZED camera to the URDF model of the rover.

To do that log in to the Raspberry Pi and install the zed_description package that contains the URDF macros for ZED cameras:

sudo apt update
sudo apt install ros-${ROS_DISTRO}-zed-description

Then create new xacro macro for the vision pack in /etc/ros/urdf directory:

/etc/ros/urdf/zed2i.urdf.xacro
<?xml version="1.0"?>
<robot xmlns:xacro="http://www.ros.org/wiki/xacro">

  <xacro:include filename="$(find zed_description)/urdf/zed_macro.urdf.xacro" />

  <xacro:zed_camera name="zed2i" model="zed2i">
    <gnss_origin xyz="0 0 0" rpy="0 0 0" />
  </xacro:zed_camera>

  <joint name="base_to_zed2i_joint" type="fixed">
    <parent link="base_link"/>
    <child link="zed2i_camera_link"/>
    <origin xyz="0.144 0 0.023" rpy="0 0 0"/>
  </joint>

</robot>
note

If you have mounted the ZED camera in a different position or orientation than the one specified in the hardware integration section, make sure to update the origin tag in the joint definition accordingly.

Now, include this file in the main rover URDF before the closing </robot> tag:

/etc/ros/urdf/robot.urdf.xacro
<xacro:include filename="/etc/ros/urdf/zed2i.urdf.xacro"/>

After saving both files, apply the updated robot description:

ros-nodes-restart

You should now be able to see the ZED camera frames in the robot's TF tree.

(Optional) Launching the container on boot​

If you want the ZED camera to be available immediately after the Jetson boots up, you can follow the steps below to set up a systemd service to run the container and mount a host directory for configs and logs.

note

Perform these steps on the Jetson outside of the container.

Save the following docker-compose.yml to previously created /opt/zed_ros (adjust environment overrides as needed):

/opt/zed_ros/docker-compose.yml
services:
  zed-wrapper:
    image: ghcr.io/fictionlab/jetson_ros_jazzy_zed:latest
    container_name: zed-ros-wrapper
    restart: unless-stopped
    network_mode: host
    ipc: host
    privileged: true
    runtime: nvidia
    environment:
      - RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
      - ROS_DOMAIN_ID=0
    volumes:
      - /opt/zed_ros/config:/data:ro
      - /opt/zed_ros/logs:/root/.ros/log
      - /dev/bus/usb:/dev/bus/usb
      - /etc/localtime:/etc/localtime:ro
    command: >
      ros2 launch zed_wrapper zed_camera.launch.py camera_model:=zed2i
      ros_params_override_path:=/data/zed2i.yaml

Add the systemd unit /etc/systemd/system/zed-ros-wrapper.service:

/etc/systemd/system/zed-ros-wrapper.service
[Unit]
Description=ZED ROS 2 Wrapper container
After=network-online.target docker.service
Requires=docker.service

[Service]
Type=exec
WorkingDirectory=/opt/zed_ros
ExecStart=/usr/bin/docker compose up --remove-orphans
ExecStop=/usr/bin/docker compose down
Restart=always
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl enable --now zed-ros-wrapper
info

You can use journalctl -u zed-ros-wrapper -f to follow the logs and ensure the container starts correctly.

To restart the container manually, use sudo systemctl restart zed-ros-wrapper.

Now the ZED wrapper container will start automatically on boot, and you can manage your configurations and logs easily from the host filesystem.

What's next?​

ZED camera can be used in projects involving autonomous navigation. You might be interested in a tutorial about it.

You can also use the ZED camera for object detection using the ZED ROS wrapper's built-in detection capabilities.