Refer to the article for links :

https://blog.csdn.net/zhang970187013/article/details/81098175
<https://blog.csdn.net/zhang970187013/article/details/81098175>

https://github.com/IFL-CAMP/easy_handeye/tree/master
<https://github.com/IFL-CAMP/easy_handeye/tree/master>

https://blog.csdn.net/sinat_23853639/article/details/80276848
<https://blog.csdn.net/sinat_23853639/article/details/80276848>

I ran for a week , A little bit forward, but there's always a problem , All kinds of messy node conflicts No topics were posted
, The coordinate system always jumps , The page appears to flash back , Just write the article launch The first few steps alone to run out , I didn't expect that the problem would be solved like this ...

  IntelRealsense ZR300 Calibration of camera external parameters

 
first , The principle of calibration is : Base coordinate system (base_tree) And the camera (camera_tree) The two coordinate systems are different tree, By attaching a label to your hand , The camera recognized the tag position
and
orention, And passed hand_eye The calibration package is obtained marker_frame( manipulator ), Further, we get the relative base Location relationship of . That is, the sub coordinate system (camera_rgb_optical_frame) To parent coordinate system (base_link) The relationship between . After that, if the camera recognizes the location of the object ( stay camera Under the coordinate system ), You can pass the transform( This transformation relationship ), Into base( That is, the robot knows its own position coordinate system ) Position in coordinate system , In this way, the robot can get the actual position in space recognized by the camera through the transformation relationship .

The overall steps are as follows :

1. Clear irrelevant nodes
#rosnode cleanup
2. Start the camera driver
#roslaunch realsense_camera zr300_nodelet_rgbd.launch
3. Connecting robots
#roslaunch ur_control ur10_control.launch robot_ip:=192.168.2.24
4. start-up moveit and rviz
#roslaunch ur10_rg2_moveit_config demo.launch
5. Start the calibration procedure launch node
#roslaunch ur10_realsense_handeyecalibration.launch
launch The document has been modified :( A single terminal will solve the problem rviz Always back off , Coordinate system runout , Acquisition errors and other issues )

among :start robot/robot222 And the front start realsense Carry it out alone and run in the node , This solves the problem .
<launch> <arg name="namespace_prefix"
default="ur10_realsense_handeyecalibration" /> <arg name="robot_ip" doc="The IP
address of the UR10 robot" /> <!--<arg name="marker_frame"
default="aruco_marker_frame"/>--> <arg name="marker_size" doc="Size of the
ArUco marker used, in meters" default="0.1" /> <arg name="marker_id" doc="The
ID of the ArUco marker used" default="100" /> <!-- start the realsense --> <!--
<include file="$(find kinect2_bridge)/launch/kinect2_bridge.launch" /> --> <!--
<group ns="namespace1"> --> <!-- <include file="$(find
realsense_camera)/launch/zr300_nodelet_rgbd.launch" /> --> <!-- <arg
name="depth_registration" value="true" /> --> <!-- </group> --> <!-- </include>
--> <!-- start ArUco --> <node name="aruco_tracker" pkg="aruco_ros"
type="single"> <!-- <remap from="/camera_info" to="/kinect2/hd/camera_info" />
--> <remap from="/camera_info" to="/camera/rgb/camera_info" /> <!-- <remap
from="/image" to="/kinect2/hd/image_color_rect" /> --> <remap from="/image"
to="/camera/rgb/image_rect_color" /> <param name="image_is_rectified"
value="true"/> <param name="marker_size" value="$(arg marker_size)"/> <param
name="marker_id" value="$(arg marker_id)"/> <!-- <param name="reference_frame"
value="kinect2_rgb_optical_frame"/> <param name="camera_frame"
value="kinect2_rgb_optical_frame"/> --> <param name="reference_frame"
value="camera_rgb_optical_frame"/> <param name="camera_frame"
value="camera_rgb_optical_frame"/> <param name="marker_frame"
value="camera_marker" /> </node> <!-- start the robot --> <!-- <include
file="$(find ur_modern_driver)/launch/ur10_bringup.launch"> <arg name="limited"
value="true" /> <arg name="robot_ip" value="192.168.2.24" /> </include> -->
<!-- <include file="$(find
ur10_rg2_moveit_config)/launch/planning_context.launch"> <arg
name="load_robot_description" value="false"/> <arg
name="load_robot_description" value="true"/> --> <!-- <include file="$(find
ur5_moveit_config)/launch/ur5_moveit_planning_execution.launch"> --> <!--
</include> --> <!-- <include file="$(find
ur10_rg2_moveit_config)/launch/ur10_moveit_planning_execution.launch"> <arg
name="limited" value="true" /> </include> --> <!-- start the robot222 --> <!--
<group ns="namespace2"> <include file="$(find
ur_control)/launch/ur10_control.launch"> <arg name="limited" value="true" />
<arg name="robot_ip" value="192.168.2.24" /> </include> </group> <group
ns="namespace3"> <include file="$(find
ur10_rg2_moveit_config)/launch/demo_norviz.launch"> --> <!-- <include
file="$(find ur10_rg2_moveit_config)/launch/demo.launch"> --> <!-- <arg
name="limited" value="true" /> </include> </group> --> <!-- start easy_handeye
--> <include file="$(find easy_handeye)/launch/calibrate.launch" > <arg
name="namespace_prefix" value="$(arg namespace_prefix)" /> <arg
name="eye_on_hand" value="false" /> <!-- <arg name="tracking_base_frame"
value="kinect2_rgb_optical_frame" /> --> <arg name="tracking_base_frame"
value="camera_rgb_optical_frame" /> <arg name="tracking_marker_frame"
value="camera_marker" /> <arg name="robot_base_frame" value="base_link" /> <!--
<arg name="robot_effector_frame" value="wrist_3_link" /> --> <arg
name="robot_effector_frame" value="rg2_eef_link" /> <arg
name="freehand_robot_movement" value="false" /> <arg
name="robot_velocity_scaling" value="0.5" /> <arg
name="robot_acceleration_scaling" value="0.2" /> </include> </launch>
In addition, the label content should be modified , stay ur10_realsense_handeyecalibration.launch In the file , Yes

<arg name="marker_size" doc="Size of the ArUco marker used, in meters"
default="0.0765" />
 <arg name="marker_id" doc="The ID of the ArUco marker used" default="582" />

there default The values are
Specific printed label ( stay ~/catkin_ws/src/aruco_ros/aruco_ros/etc There are corresponding labels under the folder that can be identified , use ps Change to pdf And shrink print can be recognized , of course aruco Download is also available on the official website
The corresponding new tag can only be used after changing the configuration file ) Size of ( Note that the accuracy is m), the second default The value is the tag number , You can run the program and let it go rqt_image_view Check the identified tags inside , My tags ID=582.



In the end, the right one can be achieved camera and base Calibration of .

So the revised ur10_realsense_handeyecalibration.launch The calibration program file is :
<launch> <arg name="namespace_prefix"
default="ur10_realsense_handeyecalibration" /> <arg name="robot_ip" doc="The IP
address of the UR10 robot" /> <!--<arg name="marker_frame"
default="aruco_marker_frame"/>--> <arg name="marker_size" doc="Size of the
ArUco marker used, in meters" default="0.0765" /> <arg name="marker_id"
doc="The ID of the ArUco marker used" default="582" /> <!-- start the realsense
--> <!-- <include file="$(find kinect2_bridge)/launch/kinect2_bridge.launch" />
--> <!-- <group ns="namespace1"> --> <!-- <include file="$(find
realsense_camera)/launch/zr300_nodelet_rgbd.launch" /> --> <!-- <arg
name="depth_registration" value="true" /> --> <!-- </group> --> <!-- </include>
--> <!-- start ArUco --> <node name="aruco_tracker" pkg="aruco_ros"
type="single"> <!-- <remap from="/camera_info" to="/kinect2/hd/camera_info" />
--> <remap from="/camera_info" to="/camera/rgb/camera_info" /> <!-- <remap
from="/image" to="/kinect2/hd/image_color_rect" /> --> <remap from="/image"
to="/camera/rgb/image_rect_color" /> <param name="image_is_rectified"
value="true"/> <param name="marker_size" value="$(arg marker_size)"/> <param
name="marker_id" value="$(arg marker_id)"/> <!-- <param name="reference_frame"
value="kinect2_rgb_optical_frame"/> <param name="camera_frame"
value="kinect2_rgb_optical_frame"/> --> <!-- <param name="reference_frame"
value="camera_rgb_frame"/> <param name="camera_frame"
value="camera_rgb_frame"/> --> <param name="reference_frame"
value="camera_link"/> <param name="camera_frame"
value="camera_rgb_optical_frame"/> <param name="marker_frame"
value="camera_marker" /> </node> <!-- start the robot --> <!-- <include
file="$(find ur_modern_driver)/launch/ur10_bringup.launch"> <arg name="limited"
value="true" /> <arg name="robot_ip" value="192.168.2.24" /> </include> -->
<!-- <include file="$(find
ur10_rg2_moveit_config)/launch/planning_context.launch"> <arg
name="load_robot_description" value="false"/> <arg
name="load_robot_description" value="true"/> --> <!-- <include file="$(find
ur5_moveit_config)/launch/ur5_moveit_planning_execution.launch"> --> <!--
</include> --> <!-- <include file="$(find
ur10_rg2_moveit_config)/launch/ur10_moveit_planning_execution.launch"> <arg
name="limited" value="true" /> </include> --> <!-- start the robot222 --> <!--
<group ns="namespace2"> <include file="$(find
ur_control)/launch/ur10_control.launch"> <arg name="limited" value="true" />
<arg name="robot_ip" value="192.168.2.24" /> </include> </group> <group
ns="namespace3"> <include file="$(find
ur10_rg2_moveit_config)/launch/demo_norviz.launch"> --> <!-- <include
file="$(find ur10_rg2_moveit_config)/launch/demo.launch"> --> <!-- <arg
name="limited" value="true" /> </include> </group> --> <!-- start easy_handeye
--> <include file="$(find easy_handeye)/launch/calibrate.launch" > <arg
name="namespace_prefix" value="$(arg namespace_prefix)" /> <arg
name="eye_on_hand" value="false" /> <!-- <arg name="tracking_base_frame"
value="kinect2_rgb_optical_frame" /> --> <!-- <arg name="tracking_base_frame"
value="camera_rgb_frame" /> --> <arg name="tracking_base_frame"
value="camera_link" /> <arg name="tracking_marker_frame" value="camera_marker"
/> <arg name="robot_base_frame" value="base_link" /> <!-- <arg
name="robot_effector_frame" value="wrist_3_link" /> --> <arg
name="robot_effector_frame" value="rg2_eef_link" /> <arg
name="freehand_robot_movement" value="false" /> <arg
name="robot_velocity_scaling" value="0.5" /> <arg
name="robot_acceleration_scaling" value="0.2" /> </include> </launch>
6. start-up rqt
#rqt_image_view
7. print label Prepare hand eye calibration

There are various problems :value value Conversion was not successful , And then it was -nan -nan -nan -nan.
[ERROR] [1515404258.018207679]: Ignoring transform for child_frame_id
"tool0_controller" from authority "unknown_publisher" because of a nan value in
the transform (0.000000
26815622274503241629349378993175553962225540122973894056168551426931042827562994218784229639762337459109057520690886188494020583679371347120178163255083008.000000
26815622274206272770731577112262473438093890782345820683584369763133472687942565341004009969841415624372630858128332080466218773763999513725165301565227008.000000)
(-nan -nan -nan -nan)
Refer to the blog for revision :

https://github.com/ThomasTimm/ur_modern_driver/issues/128
<https://github.com/ThomasTimm/ur_modern_driver/issues/128>

Here's how , about tool0_controller Wrong reporting :

https://blog.csdn.net/coldplayplay/article/details/79106134
<https://blog.csdn.net/coldplayplay/article/details/79106134>

But I changed the program and found that it didn't solve the problem , And when running the original program ( Reference yourself UR10
RG2 That blog post ), This is also a problem . When crying for nothing , Change your mind . hold ur10_rg2_ros( Before all UR10 RG2 And the definition of grab
control program ) Delete all , Re import the previously backed up original for compilation , no problem . Then modify it step by step again ( Go through the whole process ) This problem is solved ...

( be careful : modify .cpp After the file is saved, it must be renewed cmake, Otherwise, no usable files will be generated . This and py File in ros Different .)


The latest solution is : In the revision communication.cpp and realtime_communication.cpp After the file of , If the above errors still exist, it may be due to network reasons ur10 Disconnect from the machine , Such consequences will not only lead to the above bug
It may also cause the manipulator to shake , Wrong emergency stop , The connection process is often reconnect. The solution is to modify the two documents , Then test it wirelessly or wirelessly , Avoid network problems roslaunch A compile error is caused when .

Calibrate again , The total execution steps are as follows :
#roslaunch realsense_camera zr300_nodelet_rgbd.launch #roslaunch ur_control
ur10_control.launch robot_ip:=192.168.2.24 #roslaunch easy_handeye
static_transform_publisher.launch #roslaunch ur10_rg2_moveit_config demo.launch
#roslaunch easy_handeye ur10_realsense_handeyecalibration.launch
Be careful to open it rqt #rqt_image_view, Otherwise, the acquisition will fail



The acquisition steps are as follows :1( If not 17( Mine is 8) The words of )--2--3--4--5--2345( Repeat until all the data are collected )--6--7( Save the results )





The next step is to display the calibration results in real time :


I changed it to a real-time one (100hz) For publishing location conversion relationships launch file , Put in ~/catkin_ws/src/easy_handeye/easy_handeye/launch( And before ur10_realsense_handeyecalibration.launch In a bag ), there args The following is the collected transformation relationship position(x
y z) and orention(x y z
w) Value of , Notice the sub coordinate system behind it (camera_rgb_optical_frame) Point to parent coordinate system (base_link), The frequency is usually set to 100Hz.
<launch> <node pkg="tf" type="static_transform_publisher"
name="link1_broadcaster" args="0.244332058703 -0.0155924006203 -0.0434545849503
0.0144077350003 -0.0295559697343 -0.0183787903043 0.999290289101 base_link
camera_link 100" /> </launch>


stay rviz Calibration results appear in it , But there's always been that before , Moreover, the correct calibration result coordinate system only flickers . This is because in the start
easy_handeye( It was written before ur10_realsense_handeyecalibration.launch The package calls this package to realize the calibration function ) A fixed original coordinate system position is published by default , The default coordinate system of this rule is always displayed in the calibration.launch In the file . stay calibration The booted node contains <!--
Dummy calibration to have a fully connected TF tree and see all frames
--> In the following definition, there are related fixed coordinate system position marks , So if you want to remove it, it will not be displayed .( This is the wrong coordinate system , The real can only flicker ).



 



After modification :



The preliminary calibration work is completed , Then the integration of the program , And previous depth cameras as well as linemod Run at the same time ( Of course, it's hard for computers to display in real time ).
Operation steps camera + distinguish +listener #roslaunch realsense_camera zr300_nodelet_rgbd.launch
#rosrun object_recognition_core detection -c `rospack find
object_recognition_linemod`/conf/detection.ros.ork #rosrun
ur10_rg2_moveit_config listener.py (#rosrun rqt_reconfigure rqt_reconfigure)
Coordinate system transformation + Start the robot #roslaunch ur_control ur10_control.launch robot_ip:=192.168.2.24
#roslaunch easy_handeye static_transform_publisher.launch #roslaunch
ur10_rg2_moveit_config demo.launch #roslaunch easy_handeye
ur10_realsense_handeyecalibration.launch Robot control #rosrun ur10_rg2_moveit_config
moveit_controller.py // Limit #rosrun ur10_rg2_moveit_config
new_moveit_controller.py // control program
there lisntener.py It's a dig that I made myself camera_frame Position of object in coordinate system , Convenient for future calls :
#!/usr/bin/env python #coding=utf-8 import rospy # from std_msgs.msg import
String from object_recognition_msgs.msg import RecognizedObjectArray def
callback(data): if len(data.objects)>0: print "coke
detected~~~~~~~~~~~~~~~~~~~~" rospy.loginfo(rospy.get_caller_id() + 'heard %s',
data.objects[0].pose.pose.pose) else: print "nothing detected................."
def listener(): # In ROS, nodes are uniquely named. If two nodes with the same
# name are launched, the previous one is kicked off. The # anonymous=True flag
means that rospy will choose a unique # name for our 'listener' node so that
multiple listeners can # run simultaneously. rospy.init_node('listener',
anonymous=True) rospy.Subscriber('recognized_object_array',
RecognizedObjectArray, callback) # spin() simply keeps python from exiting
until this node is stopped rospy.spin() if __name__ == '__main__': listener()
Show results :



such , The overall program came down , stay rviz Inside the observation results :




Of course, it is too laggy. , And the problem is that camera_rgb_frame and camera_depth_frame Not between transform, Although it does not affect , Because this is the coordinate system inside the camera , and IntelRealsense
ZR300 The advantage is that there is no need to calibrate the camera's internal parameters .

expectation :


A lot of time is wasted in the calibration , However, the calibration of external parameters was successfully completed . The next step is to use TF, Broadcast monitoring TF, utilize lookupTransform(),transformPoint() Equifunction , The location information of coke cans collected by the camera ( stay camera_rgb_frame lower ) Convert to base coordinate system (base_link), In this way, the manipulator can be indicated to reach the corresponding position .

The previous calibration of external parameters is only to base_link and camera_frame Two tree It's connected (static_transform
100Hz release ), in other words tf You can use two tree Sub coordinate system of , and tf To store the transformation relationship , And the corresponding points in different tree No coordinate system is used to convert to the same one . The next step is to continue the work .

Of course, this is all engineering practice , Start looking at some deep learning models , Finally, the innovation of thesis publishing is also based on this platform , Innovation in image recognition . take a heavy burden and embark on a long road , keep trying (#^.^#)~