FisheyeStitcher

../../_images/fisheye_stitcher.jpg

Generate panorama image by stitching dual-fisheye camera images. This is ROS wrapper of drNoob13/fisheyeStitcher

Subscribing Topic

  • ~input (sensor_msgs/Image)

    Input dual-fisheye image.

Publishing Topic

  • ~output (sensor_msgs/Image)

    Output panorama image.

Parameters

  • ~light_compen (Bool, default: false)

    Light Fall-off Compensation. The outer edge of the fisheye image has low brightness.

  • ~refine_align (Bool, default: false)

    Adjusts any discontinuities caused by objects with varying depth in the stitching boundaries. Note that this process takes a large CPU usage.

  • ~fovd (Double, default: 195.0)

    Field of view of fisheye camera [degree].

  • ~save_unwarped (Bool, default: false)

    Save unwarped (rectified) fisheye images under ~/.ros. These images are useful to update panorama parameter.

  • ~mls_map_path (String, default: “”)

    Path to .yml.gz file which contains MLS grids information. How to generate this file is explained below.

Sample

Before running the sample, please catkin build jsk_perception.

rosrun jsk_perception install_sample_data.py
rosbag play $(rospack find jsk_perception)/sample/data/insta360_air.bag --loop --clock
roslaunch jsk_perception sample_dual_fisheye_to_panorama.launch mls_map_path:=$(rospack find jsk_perception)/sample/data/fisheye_stitcher_grid_xd_yd_3840x1920_fetch15.yml.gz

Update panorama parameter

This section describes how to update the panorama parameter for generating a panorama image. Note that different cameras basically require different parameters, even if they are of the same type. However, if you’re lucky, you can use the parameters of another camera, so you can try that first.

Here is an example using insta360.

  1. Unwarp and save the left and right fisheye images. The images are $HOME/.ros/l_img_crop.jpg and $HOME/.ros/r_img_crop.jpg

roslaunch jsk_perception sample_insta360_air.launch use_usb_cam:=true save_unwarp:=true
  1. Annotate the corresponding points of the left and right images. We use labelme as a GUI tool. When annotating, it is easier to start with l_img_crop.jpg, which has a narrow angle of view than r_img_crop.jpg. $HOME/.ros/l_img_crop.json and $HOME/.ros/r_img_crop.json should be outputted.

sudo pip install labelme==4.5.7
labelme $HOME/.ros/l_img_crop.jpg
labelme $HOME/.ros/r_img_crop.jpg

../../_images/left_annotation.pngLeft image with annotation ../../_images/right_annotation.pngRight image with annotation

  1. Save the correspondence points to a matlab file.

rosrun jsk_perception create_mls_correspondence.py
  1. Run mls_rigid_example2.m to generate the MLS grids file. matlab can be installed from the University of Tokyo license. You also need to install the image processing toolbox and the parallel computing toolbox. Please check the official wiki

  2. Convert the MLS grid file to a yaml file so that OpenCV can read it.

rosrun jsk_perception mls_matlab2opencv.py
  1. Compress the yaml file to gz and place it under jsk_perception.

gzip ~/.ros/fisheye_stitcher_grid_xd_yd_3840x1920.yml
mv ~/.ros/fisheye_stitcher_grid_xd_yd_3840x1920.yml.gz $(rospack find jsk_perception)/config/fisheye_stitcher_grid_xd_yd_3840x1920.yml.gz