Katsushi Ikeuchi

ICRA Conference 2024 Conference Paper

LiDAR-camera Calibration using Intensity Variance Cost

• Ryoichi Ishikawa
• Shuyi Zhou
• Yoshihiro Sato
• Takeshi Oishi
• Katsushi Ikeuchi

We propose an extrinsic calibration method for LiDAR-camera fusion systems using variations in intensities projected from camera images to the LiDAR point cloud. As the input, the proposed method uses a sequence of LiDAR data and camera images captured while moving the system. Once the camera motion is calculated, camera images are projected onto the point cloud. The variations in the projected intensities at each point are large in the presence of errors in the estimated motion or calibration parameters. Consequently, the extrinsic parameters are optimized for cost minimization based on the intensity variance. In addition, a suitable geometry is proposed for the calibration and verified using simulations. Our experimental results showed that the proposed method accurately performed calibrations using a camera and a sparse multi-beam LiDAR or one-dimensional LiDAR.

IROS Conference 2022 Conference Paper

Deep Gesture Generation for Social Robots Using Type-Specific Libraries

• Hitoshi Teshima
• Naoki Wake
• Diego Thomas
• Yuta Nakashima
• Hiroshi Kawasaki
• Katsushi Ikeuchi

Body language such as conversational gesture is a powerful way to ease communication. Conversational gestures do not only make a speech more lively but also contain semantic meaning that helps to stress important information in the discussion. In the field of robotics, giving conversational agents (humanoid robots or virtual avatars) the ability to properly use gestures is critical, yet remain a task of extraordinary difficulty. This is because given only a text as input, there are many possibilities and ambiguities to generate an appropriate gesture. Different to previous works we propose a new method that explicitly takes into account the gesture types to reduce these ambiguities and generate human-like conversational gestures. Key to our proposed system is a new gesture database built on the TED dataset that allows us to map a word to one of three types of gestures: “Imagistic” gestures, which express the content of the speech, “Beat” gestures, which emphasize words, and “No gestures. ” We propose a system that first maps the words in the input text to their corresponding gesture type, generate type-specific gestures and combine the generated gestures into one final smooth gesture. In our comparative experiments, the effectiveness of the proposed method was confirmed in user studies for both avatar and humanoid robot.

ICRA Conference 2021 Conference Paper

Verbal Focus-of-Attention System for Learning-from-Observation

• Naoki Wake
• Iori Yanokura
• Kazuhiro Sasabuchi
• Katsushi Ikeuchi

The learning-from-observation (LfO) framework aims to map human demonstrations to a robot to reduce programming effort. To this end, an LfO system encodes a human demonstration into a series of execution units for a robot, which are referred to as task models. Although previous research has proposed successful task-model encoders, there has been little discussion on how to guide a task-model encoder in a scene with spatio-temporal noises, such as cluttered objects or unrelated human body movements. Inspired by the function of verbal instructions guiding an observer’s visual attention, we propose a verbal focus-of-attention (FoA) system (i. e. , spatiotemporal filters) to guide a task-model encoder. For object manipulation, the system first recognizes the name of a target object and its attributes from verbal instructions. The information serves as a where-to-look FoA filter to confine the areas in which the target object existed in the demonstration. The system then detects the timings of grasp and release that occurred in the filtered areas. The timings serve as a when-to-look FoA filter to confine the period of object manipulation. Finally, a task-model encoder recognizes the task models by employing the FoA filters. We demonstrate the robustness of the verbal FoA in attenuating spatio-temporal noises by comparing it with an existing action localization network. The contributions of this study are as follows: (1) to propose a verbal FoA for LfO, (2) to design an algorithm to calculate FoA filters from verbal input, and (3) to demonstrate the effectiveness of a verbal FoA in localizing an action by comparing it with a state-of-the-art vision system.

IROS Conference 2018 Conference Paper

LiDAR and Camera Calibration Using Motions Estimated by Sensor Fusion Odometry

• Ryoichi Ishikawa
• Takeshi Oishi
• Katsushi Ikeuchi

This paper proposes a targetless and automatic camera-LiDAR calibration method. Our approach extends the hand-eye calibration framework to 2D-3D calibration. The scaled camera motions are accurately calculated using a sensor-fusion odometry method. We also clarify the suitable motions for our calibration method. Whereas other calibrations require the LiDAR reflectance data and an initial extrinsic parameter, the proposed method requires only the three-dimensional point cloud and the camera image. The effectiveness of the method is demonstrated in experiments using several sensor configurations in indoor and outdoor scenes. Our method achieved higher accuracy than comparable state-of-the-art methods.

ICRA Conference 2014 Conference Paper

Detecting potential falling objects by inferring human action and natural disturbance

• Bo Zheng 0001
• Yibiao Zhao
• Joey C. Yu
• Katsushi Ikeuchi
• Song-Chun Zhu

Detecting potential dangers in the environment is a fundamental ability of living beings. In order to endure such ability to a robot, this paper presents an algorithm for detecting potential falling objects, i. e. physically unsafe objects, given an input of 3D point clouds captured by the range sensors. We formulate the falling risk as a probability or a potential that an object may fall given human action or certain natural disturbances, such as earthquake and wind. Our approach differs from traditional object detection paradigm, it first infers hidden and situated “causes (disturbance) of the scene, and then introduces intuitive physical mechanics to predict possible “effects (falls) as consequences of the causes. In particular, we infer a disturbance field by making use of motion capture data as a rich source of common human pose movement. We show that, by applying various disturbance fields, our model achieves a human level recognition rate of potential falling objects on a dataset of challenging and realistic indoor scenes.

IROS Conference 2014 Conference Paper

Extraction of person-specific motion style based on a task model and imitation by humanoid robot

• Takahiro Okamoto
• Takaaki Shiratori
• M. Glisson
• K. Yamane
• Shunsuke Kudoh
• Katsushi Ikeuchi

In this paper, we present a humanoid robot which extracts and imitates the person-specific differences in motions, which we will call style. Synthesizing human-like and stylistic motion variations according to specific scenarios is becoming important for entertainment robots, and imitation of styles is one variation which makes robots more amiable. Our approach extends a learning from observation (LFO) paradigm which enables robots to understand what a human is doing and to extract reusable essences to be learned. The focus is on styles in the domain of LFO and the representation of them using the reusable essences. In this paper, we design an abstract model of a target motion defined in LFO, observe human demonstrations through the model, and formulate the representation of styles in the context of LFO. Then we introduce a framework of generating robot motions that reflect styles which are automatically extracted from human demonstrations. To verify our proposed method we applied it to a ring toss game, and generated robot motions for a physical humanoid robot. Styles from each of three random players were extracted automatically from their demonstrations, and used for generating robot motions. The robot imitates the styles of each player without exceeding the limitation of its physical constraints, while tossing the rings to the goal.

ICRA Conference 2013 Conference Paper

Representation and mapping of dexterous manipulation through task primitives

• Phongtharin Vinayavekhin
• Shunsuke Kudoh
• Jun Takamatsu
• Yoshihiro Sato
• Katsushi Ikeuchi

The goal of this work is to teach a robot to regrasp an object using knowledge obtained from human demonstration. This paper presents a task model that represents a human regrasping movement. The task model is based on the topological information and comprised of four task primitives. Human regrasping movement is recognised and represented as a sequence of these task primitives by the proposed recognition algorithm. The proposed method then maps each task primitive to the target robot hand using knowledge obtained from human demonstration. The experimental result verified the proposed task model by executing the regrasping movement on the real robot hand.

ICRA Conference 2011 Conference Paper

Towards an automatic robot regrasping movement based on human demonstration using tangle topology

• Phongtharin Vinayavekhin
• Shunsuke Kudoh
• Katsushi Ikeuchi

This paper introduces a novel method to teach a robot to regrasp an object based on the Programming by Demonstration paradigm. In this paradigm, a robot observes a human performing a regrasping task via various sensors to recognise crucial information in order to reproduce the task using its own hand. The main contribution is in the proposal of a representation technique that can analyse a human regrasping movement and reproduce this movement in a robot hand. The technique is based on tangle topology where both hand and manipulated object are considered as strands. This allows a regrasping movement to be considered as an alteration of the tangle relationship between the strands (hand and the object) over time. Human regrasping movements are analysed and reproduced in multi-fingered robot hands in a grasp simulation to demonstrate the efficiency of the proposed method.

IROS Conference 2010 Conference Paper

Detecting dance motion structure using body components and turning motions

• Bjoern Rennhak
• Takaaki Shiratori
• Shunsuke Kudoh
• Phongtharin Vinayavekhin
• Katsushi Ikeuchi

This paper presents a novel method for robust dance motion structure detection. In the japanese folk dance domain, teachers created illustrations of dance poses. These poses characterize the most important movements of a dance. So far there is no simple and reliable extraction method which can extract all poses as shown in these drawings. We use these poses for the Task Model (TM) in the context of Learning from Observation (LFO). LFO which is a well known technique for successful human to robot motion mapping, consists of tasks (what to do) and skills (how to do). We propose a novel approach, to extract special motions from a dance, called turning motions useful for skill mapping in the LFO paradigm. Furthermore, we use a modified version of this approach, to detect all poses as shown in the drawings, called turning poses. To achieve this we observe both forearms at the same time and analyze their movement in different 2-D coordinate planes. We evaluate the parameters with and without a weighting function where we minimize acceleration, velocity and power. We successfully demonstrate this novel method using two very different japanese folk dances and discuss further implications of this work in respect to the LFO paradigm and dances of other domains.

IROS Conference 2010 Conference Paper

Temporal scaling of leg motion for music feedback system of a dancing humanoid robot

• Takahiro Okamoto
• Takaaki Shiratori
• Shunsuke Kudoh
• Katsushi Ikeuchi

In this paper, we propose a method to achieve temporal scaling of leg motions as a fundamental technique for a music feedback system of a dancing humanoid robot. We asked dancers to perform dance motion at normal musical tempo and faster musical tempos and observed how dancers modified performance for given musical tempos. The obtained insights from the observation are 1) a dancer needs to preserve leg postures that are important to emphasize dance expression, 2) there is a priority to determine what features of leg motion can be adjusted, and 3) stylistic leg motion resembles normal step motion if dancers cannot follow fast musical tempo completely. Based on these insights, we generate leg motion appropriately adjusted for changing musical tempo while maintaining balance. We validated our method via simulation experiments with a humanoid robot HRP-2.

ICRA Conference 2007 Conference Paper

Humanoid Robot Painter: Visual Perception and High-Level Planning

• Miti Ruchanurucks
• Shunsuke Kudoh
• Koichi Ogawara
• Takaaki Shiratori
• Katsushi Ikeuchi

This paper presents visual perception discovered in high-level manipulator planning for a robot to reproduce the procedure involved in human painting. First, we apply a technique of 2D object segmentation that considers region similarity as an objective function and edge as a constraint with artificial intelligent used as a criterion function. The system can segment images more effectively than most of existing methods, even if the foreground is very similar to the background. Second, we propose a novel color perception model that shows similarity to human perception. The method outperforms many existing color reduction algorithms. Third, we propose a novel global orientation map perception using a radial basis function. Finally, we use the derived model along with the brush's position- and force-sensing to produce a visual feedback drawing. Experiments show that our system can generate good paintings including portraits.

ICRA Conference 2007 Conference Paper

Marker-less Human Motion Estimation using Articulated Deformable Model

• Koichi Ogawara
• Xiaolu Li
• Katsushi Ikeuchi

This paper presents a novel whole body motion estimation method by fitting a deformable articulated model of the human body into the 3D reconstructed volume obtained from multiple video streams. The advantage of the proposed method is two fold: (1) combination of a robust estimator and ICP algorithm with Kd-tree search in pose and normal space make it possible to track complex and dynamic motion robustly against noise and interference between limb and torso, (2) the hierarchical estimation and backtrack re-estimation algorithm enable accurate estimation. The power to track challenging whole body motion in real environment is also presented.

IROS Conference 2007 Conference Paper

Multilinear analysis for task recognition and person identification

• Manoj Perera
• Takaaki Shiratori
• Shunsuke Kudoh
• Atsushi Nakazawa
• Katsushi Ikeuchi

This paper introduces a Multi Factor Tensor(MFT) model to recognize motion styles and person identities in dance sequences. We apply a musical information analysis method in segmenting the motion sequence relevant to the key poses and the musical rhythm. We define a task model considering the repeated motion segments, where the motion is decomposed into person invariant factor task and person dependant factor style. We capture the motion data of different people for a few cycles, segment it using the musical analysis approach, normalize the segments using a vectorization method, and realize our MFT model. The experiments are conducted according to two approaches. Various experiments that we conduct to evaluate the potential of the recognition ability of our proposed approaches and the results demonstrate the high accuracy of our model. The recognition results and the motion decomposition will be used in further extending the motion generation process in various styles and for different tasks.

IROS Conference 2007 Conference Paper

Robot painter: from object to trajectory

• Miti Ruchanurucks
• Shunsuke Kudoh
• Koichi Ogawara
• Takaaki Shiratori
• Katsushi Ikeuchi

This paper presents visual perception discovered in high-level manipulator planning for a robot to reproduce the procedure involved in human painting. First, we propose a technique of 3D object segmentation that can work well even when the precision of the cameras is inadequate. Second, we apply a simple yet powerful fast color perception model that shows similarity to human perception. The method outperforms many existing interactive color perception algorithms. Third, we generate global orientation map perception using a radial basis function. Finally, we use the derived foreground, color segments, and orientation map to produce a visual feedback drawing. Our main contributions are 3D object segmentation and color perception schemes.

IROS Conference 2007 Conference Paper

Temporal scaling of upper body motion for Sound feedback system of a dancing humanoid robot

• Takaaki Shiratori
• Shunsuke Kudoh
• Shinichiro Nakaoka
• Katsushi Ikeuchi

This paper proposes a method to model the modification of upper body motion of dance performance based on the speed of played music. When we observed structured dance motion performed at a normal music playback speed and motion performed at a faster music playback speed, we found that the detail of each motion is slightly different while the whole of the dance motion is similar in both cases. This phenomenon is derived from the fact that dancers omit the details and perform the essential part of the dance in order to follow the faster speed of the music. To clarify this phenomenon, we analyzed the motion differences in the frequency domain, and obtained two insights on the omission of motion details: (1) High frequency components are gradually attenuated depending on the musical speed, and (2) important stop motions are preserved even when high frequency components are attenuated. Based on these insights, we modeled our motion modification considering musical speed and joint limitations that a humanoid robot has. We show the effectiveness of our method via some applications for humanoid robot motion generation.

ICRA Conference 2006 Conference Paper

Humanoid Robot Motion Generation with Sequential Physical Constraints

• Miti Ruchanurucks
• Shinichiro Nakaoka
• Shunsuke Kudoh
• Katsushi Ikeuchi

This paper presents a method to optimize and filter trajectories generated from recorded human motion for a humanoid robot with physical limits. The objective function is responsible for mimicking human trainers, enhancing the possibility for fast convergence, while constraints are used to transform motion within the limit of the capabilities of the humanoid robot. Those constraints for angle, velocity, and dynamic force are represented as B-spline coefficients. An iterative soft-constraint paradigm is proposed to enhance the quality of velocity and force constraints. Collision avoidance is also considered as a constraint. For precision refinement, in regions of high-frequency motion not adequately modeled by an initial splining, B-spline is extensible into a hierarchy so that optimization that meets global criteria can be performed locally. Furthermore, all of the constraints can be used solely as filtering. To use these filters, an effective method to directly decompose a trajectory to a B-spline is also presented

ICRA Conference 2006 Conference Paper

Stepping Motion for a Human-like Character to Maintain Balance against Large Perturbations

• Shunsuke Kudoh
• Taku Komura
• Katsushi Ikeuchi

We propose a method of maintaining balance for a human-like character against large perturbations. The method enables a human-like model to maintain its balance with active whole-body motion, such as rotating its arms, bending down, and taking a step, if necessary. First, we capture the human motions of maintaining balance and abstract essential mechanisms from these motions. Next, we construct a model of maintaining balance that has a simple structure, such as an inverted pendulum. This model has two modes of maintaining balance: keeping the feet on the ground, and stepping. In this paper, the stepping mode is mainly described. Finally, we generate whole-body motion based on the model against several perturbations, and we discuss the validity of our method

ICRA Conference 2006 Conference Paper

Synthesizing Dance Performance using Musical and Motion Features

• Takaaki Shiratori
• Atsushi Nakazawa
• Katsushi Ikeuchi

This paper proposes a method for synthesizing dance performance synchronized to played music and our method presents a system that imitates dancers' skills in performing their motion while they listen to the music. Our method consists of a motion analysis, a music analysis, and a motion synthesis based on results of the analyses. In these analysis steps, motion and music features are acquired. These features are derived from motion keyframes, motion intensity, music intensity, musical beats, and chord changes. Our system also constructs a motion graph to search similar poses from given dance sequences and to connect them as possible transitions. In the synthesis step, the trajectory that provides the best correlation between music and motion features is selected from the motion graph, and the resulting motion is generated. Our experimental results indicate that our proposed method actually creates dance as the system "hears" the music

IROS Conference 2005 Conference Paper

Generation of humanoid robot motions with physical constraints using hierarchical B-spline

• Miti Ruchanurucks
• Shinichiro Nakaoka
• Shunsuke Kudoh
• Katsushi Ikeuchi

From recorded human motion, trajectory optimization of a humanoid robot with physical limits being the key constraint is presented. The optimization's objective function preserves the salient characteristics of the original motion, while constraints are used to transform the motion to the limit of the capabilities of the humanoid robot. It is shown that using constraints to limit generated trajectories to physically realizable motion ensures that limits are being met more precisely than by reducing them using a standard objective function. The use of wavelets vs. B-spline is compared, and it is shown that B-spline data representation has pronounced advantages. Furthermore, in regions of high-frequency motion not adequately modeled by an initial splining, B-spline is extensible into a hierarchy so that optimizations can be performed locally which still meet global criteria. It is shown how to use B-spline coefficients in angle-, velocity-, acceleration-, and dynamic force-constraints. To detect trajectory subsections with excessive error in need of control point adjustment and local re-optimization, not only is a traditional error detector used, but also a B-spline density detector is also presented. Generated motions were tested using our simulation program and the robot HRP-2.

IROS Conference 2005 Conference Paper

Motion estimation of a moving range sensor by image sequences and distorted range data

• Atsuhiko Banno
• Kazuhide Hasegawa
• Katsushi Ikeuchi

For a large scale object, scanning from the air is one of the most efficient methods of obtaining 3D data. In the case of large cultural heritage objects, there are some difficulties in scanning them with respect to safety and efficiency. To remedy these problems, we have been developing a novel 3D measurement system, the floating laser range sensor (FLRS), in which a rage sensor is suspended beneath a balloon. The obtained data, however, have some distortion due to the intra-scanning movement. In this paper, we propose a method to recover 3D range data obtained by a moving laser range sensor; this method is applicable not only to our FLRS, but also to a general moving range sensor. Using image sequences from a video camera mounted on the FLRS enables us to estimate the motion of the FLRS without any physical sensors such as gyros and GPS. At first, the initial values of camera motion parameters are estimated by perspective factorization. The next stage refines camera motion parameters using the relationships between camera images and the range data distortion. Finally, by using the refined parameter, the distorted range data are recovered. We applied this method to an actual scanning project and the results showed the effectiveness of our method.

IROS Conference 2005 Conference Paper

Task model of lower body motion for a biped humanoid robot to imitate human dances

• Shinichiro Nakaoka
• Atsushi Nakazawa
• Fumio Kanehiro
• Kenji Kaneko
• Mitsuharu Morisawa
• Katsushi Ikeuchi

The goal of this study is developing a biped humanoid robot that can observe a human dance performance and imitate it. To achieve this goal, we propose a task model of lower body motion, which consists of task primitives (what to do) and skill parameters (how to do it). Based on this model, a sequence of task primitives and their skill parameters are detected from human motion, and robot motion is regenerated from the detected result under constraints of a robot. This model can generate human-like lower body motion including various waist motions as well as various stepping motions of the legs. Generated motions can be performed stably on an actual robot supported by its own legs. We used improved robot hardware HRP-2, which has superior features in body weight, actuators, and DOF of the waist. By using the proposed method and HRP-2, we have realized a dance performance of Japanese folk dance by the robot, which is synchronized with a performance of a human grand master on the same stage.

IROS Conference 2005 Conference Paper

The climbing sensor: 3-D modeling of a narrow and vertically stalky space by using spatio-temporal range image

• Ken Matsui
• Shintaro Ono
• Katsushi Ikeuchi

In this paper, we propose a novel type of 3D scanning system named 'climbing sensor'. This system has been designed for scanning narrow and vertically stalky spaces, which are hard or extremely inefficient to scan by commercial laser range scanners due to their dimensions and limitation of FOVs. The climbing sensor equips a platform with two line scanners on a lift, and they scan through the whole target while the lift moves downwards along a ladder. One scanner is for scanning the target, which scans horizontally as the lift moves vertically, and the other scanner is for localizing the platform, which scans vertically. By using spatio-temporal range image acquired from the vertical scanning, we can accurately calculate the speed of the moving platform, with which a correct 3D model can be constructed from horizontal scans. We applied this scanning system to the Bayon Temple in Cambodia as a part of our digital archiving project of cultural assets. The scanning results proved that the system gives a sufficiently accurate 3D model and the effectiveness of our proposed system and speed estimating process.

ICRA Conference 2004 Conference Paper

Detection of Vehicles in Panoramic Range Image

• Kiyotaka Hirahara
• Katsushi Ikeuchi

It is important to assess street-parking vehicles causing traffic problems in urban areas. However, assessments are performed manually and at high cost. Developing a detection system of those vehicles is a top priority for reducing cost and avoiding human error. We propose a detection method using a laser-range finder and a line-scan camera. In the detection method, two kinds of cluster analyses are applied to laser-range points: one is for clustering laser-range points at each scan, and the other for clustering laser-range points over several scans. Each cluster of laser-range points indicates a vehicle. As a result of verification experiments in real roads, a detection rate reached 90%.

IROS Conference 2004 Conference Paper

Flexible cooperation between human and robot by interpreting human intention from gaze information

• Kenji Sakita
• Koichi Ogawara
• Shinji Murakami
• Kentaro Kawamura
• Katsushi Ikeuchi

This paper describes a method to realize flexible cooperation between human and robot which reflects the intention and state of human by using gaze information. This physiological information expresses the process of thinking directly, so it enables us to read the internal condition such as hesitation or search in decision making process. We propose a method to interpret the intention and condition from the latest history of gaze movement and determine an appropriate cooperative action of a robot based on it so that the task proceeds smoothly. Finally, we show experimental results by using a humanoid-type robot.

ICRA Conference 2004 Conference Paper

Flying Laser Range Finder and its Data Registration Algorithm

• Yuichiro Hirota
• Tomohito Masuda
• Ryo Kurazume
• Koichi Ogawara
• Kazuhide Hasegawa
• Katsushi Ikeuchi

Scanning from the air is one of the most efficient methods for obtaining 3D data of large-scale objects. For this purpose, we have been developing a flying laser range finder that is suspended under a balloon. Even though the scanning speed of the finder is quite rapid, it is difficult to eliminate the influence of the swing of a balloon. As a result the scanned data have some distortion due to the intra-scanning movement. In order to compensate this intra-scanning movement, we propose a evolutional registration algorithm which not only aligns multiple range images to determine inter-scanning movement parameters, but also rectifies distortion of range image by determining intra-scanning movement parameters. In this paper, we describe our aerial scanning system especially focusing on the design of the flying laser range finder and deformation registration algorithm. To show the effectiveness of our method, we evaluate its performance using synthesized and real data.

ICRA Conference 2004 Conference Paper

Leg Motion Primitives for a Dancing Humanoid Robot

• Shinichiro Nakaoka
• Atsushi Nakazawa
• Kazuhito Yokoi
• Katsushi Ikeuchi

The goal of the study described In this work is to develop a total technology for archiving human dance motions. A key feature of this technology is a dance replay by a humanoid robot. Although human dance motions can be acquired by a motion capture system, a robot cannot exactly follow the captured data because of different body structure and physical properties between the human and the robot. In particular, leg motions are too constrained to be converted from the captured data because the legs must interact with the floor and keep dynamic balance within the mechanical constraints of current robots. To solve this problem, we have designed a symbolic description of leg motion primitives in a dance performance. Human dance actions are recognized as a sequence of primitives and the same actions of the robot can be regenerated from them. This framework is more reasonable than modifying the original motion to adapt the robot constraints. We have developed a system to generate feasible robot motions from a human performance, and realized a dance performance by the robot HRP-1S.

IROS Conference 2004 Conference Paper

Matching and blending human motions temporal scaleable dynamic programming

• Atsushi Nakazawa
• Shinichiro Nakaoka
• Katsushi Ikeuchi

This paper presents a method for matching the frames of the human motions acquired by a motion capture system, and then creating blended (interpolated) motions according to the matching result. This matching method is basically a variation of a dynamic programming (DP) matching but we enhanced it to enable it to detect the timescale parameters. This scaleable dynamic programming (scaleable-DP) can match and evaluate the same class of motions such as walking, running, stepping and their timescale parameters. This approach is adaptable for differences in individuals, such as body sizes and timing of the stop-frames. On the blending pipeline, we first generate the keyframes according to the matching result. The keyframes are generated by considering the spatial and temporal difference of individual motions. After that, transition motions are synthesized between the keyframes. We experimented with our approach by using 15 gait motions and 5 dance motions. The results of these demonstrations show the validity of the proposed algorithm.

ICRA Conference 2003 Conference Paper

Calculating possible local displacement of curve objects using improved screw theory

• Jun Takamatsu
• Koichi Ogawara
• Hiroshi Kimura
• Katsushi Ikeuchi

Various methods to recognize assembly tasks using possible local displacement of objects have been proposed. To calculate this displacement, the screw theory is employed. It is equivalent to the first order Taylor expansion of the displacement. However, such methods can treat polyhedral objects only. Because the screw theory cannot treat curvature information of objects. In this paper, we propose a method to calculate possible local displacement of curve objects using improved screw theory, which is equivalent to the second order Taylor expansion of the displacement, and verify the validity of the proposed method.

ICRA Conference 2003 Conference Paper

Estimation of essential interactions from multiple demonstrations

• Koichi Ogawara
• Jun Takamatsu
• Hiroshi Kimura
• Katsushi Ikeuchi

To learn a new everyday task under the "Learning from Observation" framework, the system needs to detect which parts of the demonstration are essential to complete the task without task-dependent knowledge. In the previous research, we proposed a technique to estimate essential interactions in a task by integrating multiple demonstrations which represent virtually the same task. Although, the technique could automatically segment the essential interactions and determine the number of the interactions, the segmentation algorithm depends on some heuristics and only stationary interactions could be obtained. In this paper, a novel technique is proposed, which overcomes this limitation and can estimate almost any types of interactions. In this approach, a demonstrator needs to give a explicit signal once during each essential interaction as a hint on the occurrence of the essential interaction. From visual information and these signals, the system automatically analyzes the essential parts of the task and their periods, and also detects which environmental objects are interacted with the manipulated object. These information is hard to be obtained from a single demonstration, because of the ambiguity in interpreting the interaction especially in cluttered environment. The proposed method is evaluated in a simulation and also in a real world by using a humanoid robot.

ICRA Conference 2003 Conference Paper

Generating whole body motions for a biped humanoid robot from captured human dances

• Shinichiro Nakaoka
• Atsushi Nakazawa
• Kazuhito Yokoi
• Hirohisa Hirukawa
• Katsushi Ikeuchi

The goal of this study is a system for a robot to imitate human dances. This paper describes the process to generate whole body motions which can be performed by an actual biped humanoid robot. Human dance motions are acquired through a motion capturing system. We then extract symbolic representation which is made up of primitive motions: essential postures in arm motions and step primitives in leg motions. A joint angle sequence of the robot is generated according to these primitive motions. Then joint angles are modified to satisfy mechanical constraints of the robot. For balance control, the waist trajectory is moved to acquire dynamics consistency based on desired ZMP. The generated motion is tested on OpenHRP dynamics simulator. In our test, the Japanese folk dance, 'Jongara-bushi', was successfully performed by HRP-1S.

IROS Conference 2003 Conference Paper

Grasp recognition using a 3D articulated model and infrared images

• Koichi Ogawara
• Jun Takamatsu
• Kentaro Hashimoto
• Katsushi Ikeuchi

A technique to recognize the shape of a grasping hand during manipulation tasks is proposed; which utilizes 3D articulated hand model and a reconstructed 3D volume from infrared cameras. Vision-based recognition of a grasping hand is a tough problem, because a hand may be partially occluded by a grasped object and the ratio of occlusion changes along the progress of the task. To recognize the shape in a single time frame, a robust recognition method of an articulated object is proposed. In this method, 3D volumetric representation of a hand is reconstructed from multiple silhouette images and 3D articulated object model is fitted to be reconstructed data to estimate the pose and the joint angles. To deal with large occlusion, a technique to simultaneously estimate time series reconstructed volumes with the above method is proposed, which can automatically suppress the effect form badly reconstructed volumes. The proposed techniques are verified in simulation as well as in a real world.

ICRA Conference 2003 Conference Paper

Knot planning from observation

• Takuma Morita
• Jun Takamatsu
• Koichi Ogawara
• Hiroshi Kimura
• Katsushi Ikeuchi

Learning from Observation (LFO) has been widely applied in various types of robot system. It helps reduce the work of the programmer. But the available systems have application limited to rigid objects. Deformable objects are not considered because: 1) it is difficult to describe their state, and 2) too many operations are possible on them. In this paper, we choose the knot tying as case study for operating on nonrigid bodies, because a "knot theory" is available and the type of operations is limited. We describe the Knot Planning from Observation (KPO) paradigm, KPO theory and KPO system.

ICRA Conference 2003 Conference Paper

Synthesize stylistic human motion from examples

• Atsushi Nakazawa
• Shinichiro Nakaoka
• Katsushi Ikeuchi

The human body motion synthesis is highly necessary for humanoid robots' motion planning and computer animations. In this paper, new method for generating human-like natural motions based on the motion database acquired by motion capture systems is described. On the analysis step, the acquired motions are divided into some motion segments, and then the characteristic poses and motions are archived as 'motion styles'. The motion style is a kind of the human skill, and it's unique to the motions' scenario, such as the different kinds of dances. On the synthesis step, users direct the key poses of human figures. The system generates the characteristic motions according to the user's directions and motion style database. The experiment result shows that this method can synthesize the realistic 'stylized' motions with this framework.

IROS Conference 2002 Conference Paper

Calculating optimal trajectories from contact transitions

• Jun Takamatsu
• Hiroshi Kimura
• Katsushi Ikeuchi

The learning-from-demonstration method is considered for a novel robot-programming style. It consists of two parts: 1) to recognize human performance from observation as sequential motion primitives; and 2) to execute the same performance. We (2000) have proposed a method to recognize assembly tasks. However, the execution requires the ability to convert motion primitives to collision free paths. In this paper, we describe a method to calculate collision free paths. Many researchers have proposed to calculate collision free paths using analytical methods, potential fields or probabilistic methods. Potential and probabilistic methods are very powerful tools on a computer, but their solutions are not optimal. We propose a method to calculate optimal collision free paths analytically.

IROS Conference 2002 Conference Paper

Correcting observation errors for assembly task recognition

• Jun Takamatsu
• Koichi Ogawara
• Hiroshi Kimura
• Katsushi Ikeuchi

The completion of robot programs requires long development time and much effort. To shorten the programming time and to minimize the effort, we have been developing a system which we refer to as the "assembly-plan-from-observation (APO) system. " This system requires assembly task recognition from observing human performance. Observation data by a robot's vision system is usually error contaminated, and, thus, we cannot use those data directly. This paper proposes two methods to clean up those errors by using contact relations and their transitions. The first one corrects the observed configuration from contact relations observed. The second one identifies wrongly determined contact relations from an analysis of configuration space (C-space). We have implemented both methods on our test bed and have verified their effectiveness.

ICRA Conference 2002 Conference Paper

Generation of a Task Model by Integrating Multiple Observations of Human Demonstrations

• Koichi Ogawara
• Jun Takamatsu
• Hiroshi Kimura
• Katsushi Ikeuchi

This paper describes a new approach on how to teach a robot everyday manipulation tasks under the "learning from observation" framework. Most of the approaches so far assume that a demonstration can be well understood from a single demonstration. However, a single demonstration contains ambiguity, in that interactions which are essential to complete a task cannot be discerned without prior task dependent knowledge, which should be obtained from observation. To address these issues, we propose a technique to integrate multiple observations of demonstrations. The demonstrations differ, but are virtually the same task. The shared interactions among all the demonstrations are considered to be essential and we form a task model from their symbolic representations. Then the relative trajectories corresponding to each essential interaction are generalized by calculating their mean and variance and are also stored in the task model, which is used to reproduce a skilled behavior. We examine this approach by using a human-form robot, which successfully imitates human demonstrations of everyday tasks.

IROS Conference 2002 Conference Paper

Imitating human dance motions through motion structure analysis

• Atsushi Nakazawa
• Shinichiro Nakaoka
• Katsushi Ikeuchi
• Kazuhito Yokoi

This paper presents a method for importing human dance motion into humanoid robots through visual observation. The human motion data is acquired from a motion capture system consisting of 8 cameras and 8 PC clusters. Then the whole motion sequence is divided into motion elements and clustered into groups according to the correlation of end-effector trajectories. We call these segments 'motion primitives'. New dance motions are generated by concatenating these motion primitives. We are also trying to make a humanoid dance these original or generated motions using inverse-kinematics and dynamic balancing techniques.

IROS Conference 2002 Conference Paper

Improved screw theory using second order terms

• Jun Takamatsu
• Hiroshi Kimura
• Katsushi Ikeuchi

The local displacement of an object is very useful for deciding grasp stability, generating trajectories, recognizing assembly tasks, and so on. To calculate this displacement, the screw theory is employed. It is equivalent to the first order Taylor expansion of the displacement. The screw theory is very convenient, because the displacement is formulated as simultaneous linear inequalities, and a powerful tool to calculate such inequalities, the theory of the polyhedral convex cones, has already been established. However, truncation errors introduced by first order approximations sometimes cause mistaken results. In this paper, we improve the screw theory by using 2nd order terms and verify the validity of the result.

IROS Conference 2002 Conference Paper

Iterative refinement of range images with anisotropic error distribution

• Ryusuke Sagawa
• Takeshi Oishi
• Atsushi Nakazawa
• Ryo Kurazume
• Katsushi Ikeuchi

We propose a method which refines the range measurement of range finders by computing correspondences of vertices of multiple range images acquired from various viewpoints. Our method assumes that a range image acquired by a laser rangefinder has anisotropic error distribution which is parallel to the ray direction. Thus, we find the corresponding points of range images along with the ray direction. We iteratively converge range images to minimize the distance of corresponding points. We demonstrate the effectiveness of our method by presenting the experimental results of artificial and real range data. Also, we show that our method refines a 3D shape more accurately as opposed to that achieved by using the Gaussian filter.

IROS Conference 2002 Conference Paper

Modeling manipulation interactions by hidden Markov models

• Koichi Ogawara
• Jun Takamatsu
• Hiroshi Kimura
• Katsushi Ikeuchi

This paper describes a new approach on how to teach everyday manipulation tasks to a robot under the "Learning from Observation" framework. In our previous work, to acquire low-level action primitives of a task automatically, we proposed a technique to estimate essential interactions to complete a task by integrating multiple observations of similar demonstrations. But after many demonstrations are performed, there may be interactions which are the same in nature. These identical interactions should be grouped so that each action primitive becomes unique. For this purpose, a Hidden Markov Model based clustering algorithm is presented which automatically determines the number of independent interactions. We also show that the obtained interactions can be used as discriminators of human behavior. Finally, simulation and experimental results in which a real humanoid robot learns and recognizes essential actions by observing demonstrations are presented.

IROS Conference 2002 Conference Paper

Task analysis based on observing hands and objects by vision

• Yoshihiro Sato
• Keni Bernardin
• Hiroshi Kimura
• Katsushi Ikeuchi

In order to transmit, share and store human knowledge, it is important for a robot to be able to acquire task models and skills by observing human actions. Since vision plays an important role for observation, we propose a technique for measuring the position and posture of objects and hands in 3-dimensional space at high speed and with high precision by vision. Next, we show a framework for the analysis and description of a task by using an object functions as elements.

IROS Conference 2002 Conference Paper

The dynamic postural adjustment with the quadratic programming method

• Shunsuke Kudoh
• Taku Komura
• Katsushi Ikeuchi

The postural balance system is one of the most fundamental functions for humanoid robot control. In this paper, we propose a new feedback balance control system for the human body. This system can manipulate large perturbations. It finds the optimal motion for maintaining balance in the 3D space without receiving any feed-forward input beforehand. Two different strategies are adopted for the optimization: the quadratic programming method and the PD control. Simulation results are compared with real human motion; many common features such as rotating arms are observed.

ICRA Conference 2001 Conference Paper

Acquiring Hand-action Models by Attention Point Analysis

• Koichi Ogawara
• Tomikazu Tanuki
• Hiroshi Kimura
• Katsushi Ikeuchi

This paper describes our current research on learning task level representations by a robot through observation of human demonstrations. We focus on human hand actions and represent such hand actions in symbolic task models. We propose a framework of such models by efficiently integrating multiple observations based on attention points; we then evaluate the model by using a human-form robot. We propose a two-step observation mechanism. At the first step, the system roughly observes the entire sequence of the human demonstration, builds a rough task model and extracts attention points (APs). The attention points indicate the time and position in the observation sequence that requires further detailed analysis. At the second step, the system closely examines the sequence around the APs and the obtained attribute values for the task model, such as what to grasp, which hand to be used, or what is the precise trajectory of the manipulated object. We implemented this system on a human form robot and demonstrated its effectiveness.

IROS Conference 2001 Conference Paper

Parallel processing of range data merging

• Ryusuke Sagawa
• Ko Nishino
• Mark D. Wheeler
• Katsushi Ikeuchi

This paper describes a volumetric view-merging algorithm that generates a consensus surface of an object from its range images. Our original method merges a set of range images into a volumetric implicit-surface representation, which is converted to a surface mesh by using a variant of the marching-cubes algorithm. We propose a method that increases the computation and memory efficiency for computing signed distances and the method of parallel computing on a PC cluster Since our method permits a reduction in the data amount allocated in memory, the closest point is searched efficiently; this allows us to increase the number of parallel traversals and to reduce the computation time. In this paper, we describe the following two algorithms which are complementary in terms of the efficiency of CPU and memory usage: distributed allocation of range data and parallel traversal of partial octrees. By adjusting them according to the system specifications, we can build the model efficiently by a PC cluster We have implemented this system and evaluated its performance.

IROS Conference 2001 Conference Paper

Refining hand-action models through repeated observations of human and robot behavior by combined template matching

• Koichi Ogawara
• Hiroshi Kimura
• Katsushi Ikeuchi

Describes research on learning task level representations by a robot through observation of human demonstrations. We focus on human hand actions and develop a construction method of a human task model which integrates multiple observations to solve ambiguity based on attention points (AP). So far, this analysis constructs a symbolic task model efficiently in a coarse-to-fine way through two steps. However, to represent delicate motion appearing in a task, the system must incorporate the information about precise motion of the manipulated objects into an abstract task model. We propose a method to identify the manipulated object through repeated observations of both human and robot behavior. To this end, we present a method which combines 2D and 3D template matching techniques to localize an object in 3D space generated from a depth and an intensity image. We apply this technique to recognition of human and robot behavior by obtaining the precise trajectory of the manipulated objects. We also present the experimental results achieved through the use of a human-form robot equipped with a 9-eye stereo vision system.

IROS Conference 2000 Conference Paper

Extracting manipulation skills from observation

• Jun Takamatsu
• Hirohisa Tominaga
• Koichi Ogawara
• Hiroshi Kimura
• Katsushi Ikeuchi

The completion of robot programs requires a long development time and much effort. To shorten this programming time and to minimize the effort, we have been developing a system which we refer to as the "Assembly Plan from Observation (APO) system". This system provides the ability for a robot to observe a human performing an assembly task, to recognize the task and to then generate a program to perform that same task. One of the necessary tasks in APO is to create the trajectory of a robot hand movement after having observed a human's performance. The previous system developed a direct observation method based on the trajectory of a human's movement. Although it was simple and handy, the system was susceptible to noise. This paper proposes a method to make the observation robust against noise by analyzing topological contact relations. The system divides the trajectory into small segments based on this contact analysis, and then allocates an operation element, referred to as a "sub-skill", to those segments. The result is a robust, trajectory-based APO system.

IROS Conference 2000 Conference Paper

Recognition of human task by attention point analysis

• Koichi Ogawara
• Soshi Iba
• Tomikazu Tanuki
• Hiroshi Kimura
• Katsushi Ikeuchi

This paper presents a novel method of constructing a human task model by attention point (AP) analysis. The AP analysis consists of two steps: at the first step, it broadly observes human task, constructs rough human task model and finds APs which require detailed analysis; and at the second step, by applying time-consuming analysis on APs in the same human task, it can enhance the human task model. This human task model is highly abstracted and is able to change the degree of abstraction adapting to the environment so as to be applicable in a different environment. We describe this method and its implementation using data gloves and a stereo vision system. We also show an experimental result in which a real robot observed a human task and performed the same human task successfully in a different environment using this model.

ICRA Conference 2000 Conference Paper

Robust Localization for 3D Object Recognition Using Local EGI and 3D Template Matching with M-Estimators

• Kentaro Kawamura
• Kiminori Hasegawa
• Yasuyuki Someya
• Yoichi Sato 0001
• Katsushi Ikeuchi

A teleoperated system in a robot greatly reduces the demands on the human operator, although some human intervention is still required to perform such tasks as insulator recognition, positional adjustments of the robot, and guidance toward electric lines and insulators. In order to automate some of the robot's capabilities, we have developed a 3D object-localization method for the robot's positional adjustment. The method is designed to be insensitive to noise, outliers and occlusions while, at the same time, it has optimal run-time efficiency. The main contribution of our algorithm is the use of an objective function which is specified to reduce the effect of noise and outliers in the range image and a method for minimizing this function. The objective function is efficiently minimized by dynamically recomputing correspondences as the pose improves. Our algorithm is general enough to be applied not only to our dual-armed mobile robots but also to other teleoperation robots. This algorithm should greatly reduce the burden of operators when applied. This paper first describes our algorithm, and then presents a performance evaluation.

ICRA Conference 2000 Conference Paper

Symbolic Representation of Trajectories for Skill Generation

• Hirohisa Tominaga
• Jun Takamatsu
• Koichi Ogawara
• Hiroshi Kimura
• Katsushi Ikeuchi

The completion of robot programs requires long development time and much effort. To shorten this programming time and minimize the effort, we have been developing a system which we refer to as "assembly-plan-from-observation (APO) system; " this system provides the ability for a robot to observe a human performing an assembly tasks, understand the tasks, and subsequently generate a program to perform that same task. One of the necessary tasks in APO is to create a trajectory of robot hand movement from observing human performance. The previous system developed a direct observation method based on the trajectory of a human movement. Though simple and handy, the system was susceptible to noise. This paper proposes a method to make the observation robust against noise by using symbolic representations of a trajectory based on contact analysis. The system divides the trajectory into small segments based on the contact analysis, then allocates an operation element referred to as a sub-skill to those segments; the result is a robust trajectory-based APO system.

IROS Conference 1999 Conference Paper

Local-feature based vehicle recognition in infra-red images using parallel vision board

• Masataka Kagesawa
• Shinichi Ueno
• Katsushi Ikeuchi
• Hiroshi Kashiwagi

The paper describes a method for vehicle recognition, in particular, for recognizing a vehicle's make and model. Our system employs infra-red images so that we can use the same algorithm both day and night. Originally, the algorithm was the eigen-window method based on local features, but it has been changed to a vector quantization based algorithm which was originally proposed by J. Krumm (1997), to implement on an IMAP parallel image processing board. Any of these systems, based on both the eigen-window method and the vector quantization method, make a compressed database of local features for the algorithm of a target vehicle from given training images in advance; the system then matches a set of local features in the input image with those in training images for recognition. This method has the following three advantages: (1) it can detect even if part of the target vehicle is occluded; (2) it can detect even if the target vehicle is translated due to running out of lanes; (3) it does not require us to segment a vehicle from input images. The above advantages have been confirmed by performing outdoor experiments.

IROS Conference 1999 Conference Paper

Task-model based human robot cooperation using vision

• Hiroshi Kimura
• Tomoyuki Horiuchi
• Katsushi Ikeuchi

In order to assist a human, the robot must recognize human motion in real time by vision, and must plan and execute the needed assistance motion based on the task purpose and the context. In this research, we tried to solve such problems. We defined the abstract task model, analyzed the human demonstration by using events and an event stack, and automatically generated the task models needed in the assistance by the robot. The robot planned and executed the appropriate assistance motions based on the task: models according to the human motions in the cooperation with the human. We implemented a 3D object recognition system and a human grasp recognition system by using trinocular stereo color cameras and a real time range finder. The effectiveness of these methods was tested through an experiment in which the human and the robotic hand assembled toy parts in cooperation.

IROS Conference 1998 Conference Paper

Localization of insulators in electric distribution systems by using 3D template matching from multiple range images

• Kentaro Kawamura
• Mark D. Wheeler
• Osamu Yamashita
• Yoichi Sato 0001
• Katsushi Ikeuchi

Kyushu Electric has developed a dual-armed mobile-robot for use in electricity distribution systems. Although some human intervention is still required, the robot greatly reduces the demands on the human operator. In order to automate some of the robot's capabilities, we have developed a 3D object-localization method for robot positional adjustment. The method is designed to be insensitive to noise and outliers while, at the same time, it has optimal run-time efficiency. The paper first describes our algorithm, and then presents a performance evaluation.

IROS Conference 1997 Conference Paper

A quasi-linear method for computing and projecting onto c-surfaces: general case

• George V. Paul
• Katsushi Ikeuchi

This paper presents a general method to compute configuration space (c-space) obstacle surfaces (c-surfaces) in dual quaternion space and for projecting points onto them. We parameterize the c-surface using the rotation angles of the object and the vector of translation parameters of the individual contacts. Once we compute the domain of the rotation parameters, we can setup the translation parameters in a linear equation. The singular value decomposition of this equation gives us with the exact parameters of translation. We extend the theory to find the projection of a point in c-space onto the c-surface. We implement our theory on the assembly plan from observation (APO) system. The APO observes discrete instants of an assembly task and reconstructs the compliant motion plan employed in the task. We compute the contacts at each observed instant and the corresponding c-surface. We then interpolate the path on each c-surface to obtain segments of the path. The complete motion plan will be the concatenation of the connected path segments.

ICRA Conference 1997 Conference Paper

A quasi-linear method for computing and projecting onto c-surfaces: planar case

• George V. Paul
• Katsushi Ikeuchi

This paper presents a general method to compute configuration space (c-space) obstacle surfaces (c-surfaces) in planar quaternion space. We extend the method to find the projection of a given point in c-space onto the c-surface We parameterize the general c-surface using a rotation angle and the vector of translation parameters of the individual contacts. We first compute the domain of the rotation parameter. Then, we can setup the translation parameters in a linear equation. The solution of this equation using singular value decomposition gives us the exact parameters of translation. We can extend this quasi-linear method to project a point in c-space onto the c-surface. We implement our theory on the assembly plan from observation (APO) system. The APO observes discrete instants of an assembly task and reconstructs the compliant motion plan employed in the task. We compute the contacts at each observed instant and the corresponding c-surface. We then interpolate the path on each c-surface to obtain segments of the path. The complete motion plan will be the concatenation of the connected path segments.

IROS Conference 1997 Conference Paper

Visual learning and object verification with illumination invariance

• Kohtaro Ohba
• Yoichi Sato 0001
• Katsushi Ikeuchi

This paper describes a method for recognizing partially occluded objects to realize a bin-picking task under different levels of illumination brightness by using the eigenspace analysis. In the proposed method, a measured color in the RGB color space is transformed into the HSV color space. Then, the hue of the measured color, which is invariant to change in illumination brightness and direction, is used for recognizing multiple objects under different levels of illumination conditions. The proposed method was applied to real images of multiple objects under different illumination conditions, and the objects were recognized and localized successfully.

IROS Conference 1996 Conference Paper

Hand action perception for robot programming

• Yunde Jiar
• Mark D. Wheeler
• Katsushi Ikeuchi

This paper presents a general and robust approach to hand action perception for automatic robot programming using depth image sequences. The human instructor must simply demonstrate an assembly task in front of a vision system in the human world; no dataglove or special markings are necessary. The recorded image sequences are used to recover a depth image sequence for model-based human hand and object tracking to form the perceptual data stream. The data stream is then segmented and interpreted for generating a task sequence which describes the human hand action and the relationship between the manipulated object and the hand. The task sequence might be composed of a series of subtasks and each subtask involves four phases: approaching, pre-manipulating, manipulating and departing. In this paper we also discuss a robot system that replicates the observed task and automatically validates the replication results in the robot world.

IROS Conference 1996 Conference Paper

Modeling planar assembly paths from observation

• George V. Paul
• Katsushi Ikeuchi

This paper describes a system for obtaining the motion plan for a planar assembly task, given a sequence of observations of a human performing the task. The motion plan in configuration space is a series of connected path segments lying outside and on the configuration space obstacle. We use the observed configurations of the assembled objects to selectively compute the features of the c-space obstacle on which the path lies. We project the observed configurations onto these features and reconstruct the path segments. The connected path segments form the model of the observed task and can be used to program a robot to repeat the task. We demonstrate the system using the planar peg in hole task.

IROS Conference 1996 Conference Paper

Recognition of the multi specularity objects for bin-picking task

• Kohtaro Ohba
• Katsushi Ikeuchi

This paper describes a method for recognizing partially occluded objects for bin-picking tasks using the eigen-space analysis. Although effective in recognizing an isolated object, as was shown by Murase and Nayar (1995), the current method can not be applied to piratically occluded objects that are typical in bin-picking tasks. The analysis also requires that the object is centered in an image before recognition. These limitations of the eigen-space analysis are due to the fact that the whole appearance of an object is utilized as a template for the analysis. We propose a new method, referred to as the "eigen-window" method, that stores multiple partial appearances of an object in the eigen-space. Such partial appearances require a large number of memory space. To reduce the memory requirement by avoiding redundant windows and to select only effective windows to be stored, a similarity measure among windows is developed. Using a pose clustering method among windows, the method determines the pose of an object and the object type of itself. We have implemented the method and verify the validity of the method.

ICRA Conference 1995 Conference Paper

An Illumination Planner for Lambertian polyhedral Objects

• Fredric Solomon
• Katsushi Ikeuchi

The measurement of shape is a basic object inspection task. We use a noncontact method to determine shape called photometric stereo. The method uses three light sources which sequentially illuminate the object under inspection and a video camera for taking intensity images of the object. A significant problem with using photometric stereo is determining where to place the 3 light sources and the video camera. In order to solve this problem, we have developed an illumination planner that determines how to position the three light sources and the video camera around the object. The planner determines how to position light sources around an object so that we illuminate a specified set of faces in an efficient manner and so that we obtain an accurate measurement. From a high level, our planner has three major inputs: the CAD model of the object to be inspected, a noise model for our sensor, and a reflectance model for the object to be inspected. We have experimentally verified that the plans generated by the planner are valid and accurate.

IROS Conference 1995 Conference Paper

Assembly of flexible objects without analytical models

• Jun Miura
• Katsushi Ikeuchi

The ability of manipulating flexible objects, such as rubber belts and paper sheets, is important in automated manufacturing systems. This paper describes a novel approach to assembly of flexible objects. The operation dealt with in this paper is to assemble a rubber belt with fixed pulleys. By analyzing possible states of the belt based on the empirical knowledge of the belt, one can derive a method to have not only the action planning but also the visual verification planning. The authors have implemented a belt assembly system using two manipulators and a laser range finder as the sensor, and succeeded in performing the belt-pulley assembly. Extension of the authors' approach to other kinds of assembly of flexible objects is also discussed.

ICRA Conference 1995 Conference Paper

Generating Visual Sensing Strategies in Assembly Tasks

• Jun Miura
• Katsushi Ikeuchi

It is generally very difficult, if not impossible, for a robot to perform fine manipulation tasks without the benefit of some form of sensory feedback during actual task execution. As a result, sensing planning is an important component in assembly task planning. This paper describes a method of generating visual sensing strategies based on knowledge of the task to be performed. The generation of the appropriate visual sensing strategy entails knowing what information to extract and where to get it. This is facilitated by the knowledge of the task, which describes how objects are assembled. This knowledge, coupled with known sensor modeling, results in an abstract template of sensing strategy called the sensing task model. By instantiating the appropriate sensing task model at planning time, the sensing strategy is efficiently generated. Our method has been implemented using a laser range finder as the sensor. Experimental results involving typical assembly tasks show the feasibility of the method.

IROS Conference 1995 Conference Paper

Modelling planar assembly tasks: representation and recognition

• George V. Paul
• Katsushi Ikeuchi

The assembly plan from observation (APO) system observes a human operator perform an assembly task, analyzes the observations, models the task, and generates the programs for the robot to perform the same task. The task model of the observed task is defined as a sequence of contact states of the part being assembled and the motion which causes the transition between the states. The freedom of the assembled part can be represented as a polyhedral convex cone (PCC) in screw space, which can belong to one of a finite number of distinct contact states. These contact states correspond to topologically distinct intersections of PCCs with a linear subspace T in screw space. Any observed assembly task can be represented as a finite sequence of critical contact states and the motion between them. The abstract task model is used to program the robot to execute the observed assembly task. We illustrate the application of the theory by implementing the APO system for assemblies in a plane.

ICRA Conference 1995 Conference Paper

Partitioning COntact-State Space Using The Theory of Polyhedral Convex Cones

• George V. Paul
• Katsushi Ikeuchi

The assembly plan from observation (APO) system observes a human operator perform an assembly task, analyzes the observations, models the task and generates the programs for the robot to perform the same task. A major component of the APO system is the task recognition module, which models the observed task. The task model in the APO context is defined as a sequence of assembly states of the part being assembled and the actions which cause the transition between the states. The state of the assembled part is based on its freedom, which can be computed from the geometry of the contacts between the part and its environment. This freedom can be represented as a polyhedral convex cone (PCC) in screw space. We show that any contact configuration can be classed into a finite number of contact states. These contact states correspond to typologically distinct intersections of the PCC with a linear subspace T in screw space. The models of any observed task can be represented as a path in a transition graph obtained from these contact states. We illustrate the theory by implementing the APO system for polygonal objects assembled in a plane using rotation and translation.

ICRA Conference 1994 Conference Paper

Building 3-D Models from Unregistered Range Images

• Ken Higuchi
• Martial Hebert
• Katsushi Ikeuchi

The authors describe an approach to building a three-dimensional model from a set of range images. The authors' goal is to build models of free-form surfaces obtained from arbitrary viewing directions, with no initial estimate of the relative viewing directions. The approach is based on building discrete meshes representing the surfaces observed in each of the range images, to map each of the meshes to a spherical image, and to compute the transformations between the views by matching the spherical images. The meshes are built using an iterative fitting algorithm previously developed; the spherical images are built by matching the nodes of the surface meshes to the nodes of a reference mesh on the unit sphere and by storing a measure of curvature at every node. The authors describe the algorithms used for building such models from range images and for matching them. The authors give results obtained using range images of complex objects. >

ICRA Conference 1994 Conference Paper

Determination of Motion Breakpoints in a Task Sequence from Human Hand Motion

• Sing Bing Kang
• Katsushi Ikeuchi

This paper describes the authors' work on the temporal segmentation of grasping task sequences based on human hand motion. The segmentation process results in the identification of motion breakpoints separating the different constituent phases of the grasping task. A grasping task is composed of three basic phases: pregrasp phase, static grasp phase, and manipulation phase. The authors show that by analyzing the fingertip polygon (preshape) area and the speed of hand movement, they can divide a task into meaningful action segments such as approach object, grasp object, manipulate object, place object, and depart. The authors introduce a measure called the volume sweep rate, which is the product of the fingertip polygon area and the hand speed. The profile of this measure is also used in the determination of the task breakpoints. The temporal task segmentation process is important as it serves as a preprocessing step to the characterization of the task phases. Once the breakpoints have been identified, further analyses such as grasp recognition and object motion extraction can then be carried out. >

ICRA Conference 1994 Conference Paper

Grasp Recoguition and Manipulative Motion Characterization from Human Hand Motion Sequences

• Sing Bing Kang
• Katsushi Ikeuchi

We are developing a system capable of observing a human performing a task and understanding the task well enough to replicate it. This approach is called Assembly Plan from Observation. In order to replicate the observed task, we have to analyze the entire sequence. This can be done by first segmenting the task sequence into its constituent pre-grasp, grasp, and manipulation phases. This paper describes the different analyses that can be done subsequent to the temporal segmentation. These include human grasp recognition, extraction of object motion, and the spatiofrequency (spectrogram) analysis of the manipulation phase. >

IROS Conference 1994 Conference Paper

Robot task programming by human demonstration: mapping human grasps to manipulator grasps

• Sing Bing Kang
• Katsushi Ikeuchi

To alleviate the problem of overwhelming complexity in grasp synthesis and path planning associated with robot task planning, we adopt the approach of teaching the robot by demonstrating in front of it. A system with this programming technique is able to temporally segment a task into separate and meaningful parts for further individual analysis and recognize the human grasp employed in the task. With such derived information, this system would then map the human grasp to that of the given manipulator plan its trajectory, and proceed to execute the task. This paper describes how grasp mapping can be accomplished in our system. The mapping process essentially comprises three steps. The first step is local functional mapping, in which grasps of functionally equivalent fingers are established. This is followed by gross physical mapping which produces a kinematically feasible manipulator grasp. Finally, by carrying out local grasp adjustment using some task-related criterion, we arrive at a locally optimal manipulator grasp. We describe these steps in detail in this paper and show results of example grasp mappings. >

IROS Conference 1994 Conference Paper

Virtual reality modeling from a sequence of range images

• Harry Shum
• Katsushi Ikeuchi
• Raj Reddy

Virtual reality object modeling from a sequence of range images has been formulated as a problem of principal component analysis with missing data (PCAMD), which can be generalized as a weighted least square (WLS) minimization problem. An efficient algorithm has been devised to solve the problem of PCAMD. After all visible P regions appeared over the whole sequence of F views are segmented and tracked, a 3F/spl times/P normal measurement matrix of surface normals and an F/spl times/P distance measurement matrix of normal distances to the origin are constructed respectively. These two measurement matrices, with possibly many missing elements due to occlusion and mismatching, enable us to formulate multiple view merging as a combination of two WLS problems. By combining information at both the signal level and the algebraic level, a modified Jarvis' march algorithm is proposed to recover the spatial connectivity among all the reconstructed surface patches. Experiments using synthetic data and real range images show that our approach is robust against noise and mismatch. A toy house model from a sequence of real range images is presented. >

IROS Conference 1993 Conference Paper

A grasp abstraction hierarchy for recognition of grasping tasks from observation

• Sing Bing Kang
• Katsushi Ikeuchi

This work focuses on the abstraction hierarchy for a grasp which has been recognized from low-level hand-object interaction data. Previous work done on grasp classification and recognition is discussed. The proposed abstraction hierarchy is presented with illustrations, implementation issues, as well as experimental results. Issues pertaining to the conceptual analysis of the other aspects of recognizing grasping tasks are also presented. The authors report on the current status of the project and future work.

IROS Conference 1993 Conference Paper

Toward assembly plan from observation - Task recognition with planar, curved and mechanical contacts

• Katsushi Ikeuchi
• Masato Kawade
• Takashi Suehiro

The authors have been developing a novel method for programming a robot, called the assembly-plan-from-observation (APO) method. The APO method aims to build a system that has threefold capabilities. It observes a human performing an assembly task, it understands the task based on this observation, and it generates a robot program to achieve the same task. This paper concentrates on the APO's main loop, which is task recognition. Using object recognition results, the task recognition module determines what kind of assembly task is performed. A previous system, recognizes assembly tasks which only handle polyhedral objects. The system reported here handles curved objects and other mechanical contacts as well. The authors define task models for these cases and show that task models are useful in recognizing assembly tasks, and that it is possible to generate robot motion commands for repeating the same assembly task.

IROS Conference 1992 Conference Paper

Grasp Recognition Using The Contact Web

• Sing Bing Kang
• Katsushi Ikeuchi

ICRA Conference 1992 Conference Paper

Inspecting specular lobe objects using four light sources

• Fredric Solomon
• Katsushi Ikeuchi

The authors propose a noncontact method of measuring surface shape and surface roughness. The method uses four lights which sequentially illuminate the object under inspection, and a video camera for taking images of the object. Conceptually, the problem has three parts: shape extraction, pixel segmentation, and roughness extraction. The shape information is produced directly by three-light and four-light photometric stereo methods. After shape information is obtained, statistical segmentation techniques can be applied to determine which pixels are specular and which are nonspecular. Then, the specular pixels and shape information, and a simplified Torrance-Sparrow reflectance model can be used to determine the surface roughness. The method has successfully been applied to a number of synthetic and real objects. >

IROS Conference 1992 Conference Paper

Task Oriented Vision

• Katsushi Ikeuchi
• Martial Hebert

ICRA Conference 1992 Conference Paper

Towards an assembly plan from observation. I. Assembly task recognition using face-contact relations (polyhedral objects)

• Katsushi Ikeuchi
• Takashi Suehiro

The authors propose a novel method to program a robot, the assembly-plan-from-observation (APO) method. The APO method aims to build a system that has the capability of observing a human performing an assembly task, understanding the task based on the observation, and generating the robot program to achieve the same task. Assembly relations which serve as the basic representation of each assembly task are defined. It was verified that such assembly relations can be recovered from the observation of human assembly tasks, and that from such assembly relations, it is possible to generate robot motion commands to repeat the same assembly task. An APO system based on the assembly relations was demonstrated. >

IROS Conference 1992 Conference Paper

Towards An Assembly Plan From Observation: Part II: Correction Of Motion parameters Based On Fact Contact Constraints

• Takashi Suehiro
• Katsushi Ikeuchi

ICRA Conference 1991 Conference Paper

A three-finger gripper for manipulation in unstructured environments

• C. Francois
• Katsushi Ikeuchi
• Martial Hebert

A gripper is described for manipulation in natural, unstructured environments. The specific manipulation task is to pick up surface material such as pebbles or small rocks in a natural terrain. The application is to give autonomous sampling capabilities to an autonomous vehicle for planetary exploration. The authors describe the task analysis process that led to the selection of a configuration with three soft fingers. They carry out a complete analysis of the stability of a grasp for this gripper including an analysis of the deformation of the fingers at the points of contact. The implementation of a grasp selection algorithm is described, and results on three-dimensional representations of objects computed from range data are presented. >

ICRA Conference 1991 Conference Paper

Recovering shape in the presence of interreflections

• Shree K. Nayar
• Katsushi Ikeuchi
• Takeo Kanade

An algorithm for recovering the shape and reflectance of Lambertian surfaces in the presence of interreflections is presented. The surfaces may be of arbitrary but continuous shape, and with possibly varying and unknown reflectance. The actual shape and reflectance are recovered from the pseudoshape and pseudoreflectance estimated by a local shape-from-intensity method (e. g. , photometric stereo). Thus, the algorithm enhances the performance and the utility of existing shape-from-intensity methods. From the results reported, two observations can be made that are pertinent to machine vision: interreflections can cause vision algorithms to produce unacceptably erroneous results and hence should not be ignored; and at least some interreflection problems are tractable and solvable. >

IROS Conference 1991 Conference Paper

Trajectory generation with curvature constraint based on energy minimization

• Herve Delingette
• Martial Hebert
• Katsushi Ikeuchi

The trajectory generation problem for mobile robots consists in providing a set of trajectories that are 'smooth' and meet certain boundary conditions. The authors present a method to generate curvature continuous trajectories for which the curvature profile is a polynomial function of arc length. An algorithm based on the deformation of a curve by energy minimization allows one to solve general geometric constraints which was not possible by previous methods. Furthermore, it is able to take into account the limitation of radius of curvature of the robot by controlling the extrema of curvature along the path. >

ICRA Conference 1989 Conference Paper

Shape and reflectance from an image sequence generated using extended sources

• Shree K. Nayar
• Katsushi Ikeuchi
• Takeo Kanade

The authors present a method for determining the shapes of surfaces whose reflectance properties may vary from Lambertian to specular, without prior knowledge of the relative strengths of the Lambertian and specular components of reflection. The object surface is illuminated using extended light sources and is viewed from a single direction. Surface illumination using extended sources makes it possible to ensure the detection of both Lambertian and specular reflections. Multiple source directions are used to obtain an image sequence of the object. An extraction algorithm uses the set of image intensity values measured at each surface point to compute orientation as well as relative strengths of the Lambertian and specular reflection components. The proposed method is called photometric sampling, as it uses samples of photometric function that relates image intensity to surface orientation, reflectance, and light source characteristics. Experiments were conducted on Lambertian surfaces, specular surfaces, and hybrid surfaces, whose reflectance models are composed of both Lambertian and specular components. The results show high accuracy in measured orientations and estimated reflectance parameters. >

AIJ Journal 1984 Journal Article

Shape from regular patterns

• Katsushi Ikeuchi

AAAI Conference 1982 Conference Paper

A Model Based Vision System for Recognition of Machine Parts

• Katsushi Ikeuchi

This paper describes a vision system based on the photometric stereo system and a model generator called GEOMAP. The photometric stereo system obtains a needle map from shading information of an observed image. An extended Gaussian image of the needle map reduces possible attitudes of an object relative to the viewer. A model-generator called GEOMAP generates a needle map which would be observed from the viewer direction determined from EGI. Comparing the needle map by the GEOMAP with the needle map by the photometric stereo system makes final recognition of the object.

AIJ Journal 1981 Journal Article

Numerical shape from shading and occluding boundaries

• Katsushi Ikeuchi
• Berthold K.P. Horn