Stanford CS324: Perception and Manipulation for Robotics (Spring 2011)

Announcements

To sign up to mailing list for class announcements click here.

04-26-2011

• CLASS CANCELED FOR TODAY - the PR2 lab will happen at Willow Garage on April 28 - feel free to come in any time after 4 pm. (Let the instructors know if you are coming in early). Dinner will also be provided.

04-25-2011

• See #Repositories, Mailing list below for instruction on how to get onto Piazza class questions and discussion.

04-21-2011

• Next Tuesday (04/26/2011)'s class will be a lab - Working with the PR2 - where you will learn how to use the PR2 robot. The lab will be held in Gates B38. We will be there for office hours starting at 4 pm so you can come in anytime after that.

04-05-2011

• All students in the class can get key access to the robot room (115 Gates) and to the computer lab (B38) by going to the key office in the Gates building.

03-31-2011

• Thursday's (March 31) lecture will be a lab held at Willow Garage. Here's the address:

  • 68 Willow Rd, Menlo Park, CA 94025

• We will go over the lab in class - read the lab before you come if you can, but you can actually do it in class.

• Before you come for the lab, please make sure you do the following:

  • Register and get an account at kforge.ros.org. Please choose a login name that is the same as your SunetID so we can identify you easily

  • Ask to be added to the Stanford CS324 course project
• Install Ubuntu (Maverick) on your laptop - make sure you install Ubuntu Maverick, but Lucid will work as well.

  • Here are instructions on how you can dual boot Ubuntu on your Windows laptop (Thanks to Ben Cohen from the Penn PR2 team for setting this page up)

• Install ROS on your laptop

  • Install the diamondback version of ROS for Maverick or Lucid by following the installation instructions here. When you come to the actual Installation step, install the pr2-desktop variant of ROS using the following command:

    •       sudo apt-get install ros-diamondback-pr2-desktop ros-diamondback-pr2-navigation-apps ros-diamondback-wg-robots-gazebo

03-28-2011

• Go through the first reading before the end of the week.

Old Announcements

• Old Announcements

Course Description

This course is for graduate and advanced undergraduate students interested in perception for robotics, particularly for the purposes of manipulation. Work will be done on real 2 armed robots (the "PR2") and it is hoped that many of the projects will become publishable research. The course is project centric, lectures will cover the background of many areas of robotics each centered around an advanced software tool in its area. These tools will allow students to begin projects at the state of the art in robotics and advance from there.

The course will be implementation heavy, with a lot of programming in C++ or Python. At the end of the course you can expect to walk away with valuable experience in implementing state-of-the-art algorithms on robots working in real unstructured environments, it is hoped that many of the projects will lead to publication.

Time and Location

• Robot Perception TTh 5:15-6:30 Gates B12

Prerequisites

• Introduction to Computer Vision CS223B or equivalent
• Introduction to Robotics CS 223A or equivalent
• Ability to program in C++ and/or Python. Matlab will not be used here, assignments will be checked in to a code repository and run on simulators or robots in real time. Expect a substantial programming load.

Useful:

• Machine Learning CS229
• Three-Dimensional Computer Graphics CS248
• Convex Optimization EE364a

Course Materials

• You will make use of the framework and tools inside of the Robot Operating System (ROS)

  • ROS Wiki

• You will need to leverage computer vision tools, especially the Open Source Computer Vision Library (OpenCV) and the Point Cloud Library (PCL)

  • Highly recommended reference to OpenCV: Learning OpenCV

  • OpenCV Wiki

  • PCL Wiki

• Notes

  • Advice for Applying Machine Learning

  • Sparse auto-encoder theory from Andrew Ng's CS294A class; Code and Data.

Repositories, Mailing list

Piazza class mailing list

• Piazza class list is at http://www.piazzza.com/class#cs324/1

  • Sign up for Piazza at http://www.piazzza.com using your Stanford mailing address

Setup your kforge.ros.org accounts

• Register online at kforge.ros.org for an account

• Log into kforge.ros.org
• Click on the "Projects" tab

• Click on the "Stanford CS324 Spring 2011" project link

• Near the bottom of the page, click on "Join this project".
• Sign up to mailing lists|

  • For class announcements https://kforge.ros.org/mailman/listinfo/cs324-spring2011

Install ROS

• Install ROS on your laptop

  • Install the diamondback version of ROS for Lucid by following the installation instructions here. When you come to the actual Installation step, install the pr2-desktop variant of ROS using the following command:

    • sudo apt-get install ros-diamondback-pr2-desktop ros-diamondback-pr2-navigation-apps ros-diamondback-wg-robots-gazebo

Checkout Class Repository from kforge.ros.org

• Check out a copy of the main class repository

  • svn co https://kforge.ros.org/cs324/spring2011-svn/trunk cs324
    

• Check out a copy your personal repository

  • svn co https://kforge.ros.org/cs324/spring2011-users/[your_sunetid] cs324_[your_sunetid]

• Setup ROS environment variables:

  • Add the following lines to your .bashrc
    source /opt/ros/diamondback/setup.bash
    export ROS_PACKAGE_PATH=path_to_your_user_name_directory:path_to_cs324_directory:$ROS_PACKAGE_PATH

  • source ~/.bashrc

  • Please keep all your work inside of your own directory

Course Projects (Tentative List)

Here's a list of candidate projects in no particular order. Individual students and student teams will be allowed to choose projects not in this list after discussing them with the instructors. All teams and students must have a project chosen by the end of the 2nd week of classes. The projects consist of a combination of perception and manipulation elements.

• Find and pickup silverware
  • perception - find all silverware on a table
  • manipulation - put all silverware into a bin
  • bonus - separate silverware (forks, spoons, …)
• Find and pickup all transparent objects on a table
  • perception - find all transparent objects on a table
  • manipulation - pick up the objects and put them in a bin
• Object handoff
  • perception - detect and recognize human hand
  • manipulation - get object from the person’s hand
• Lock opening
  • keyhole detection - detect and register a keyhole
  • manipulation - align the key and insert into the keyhole
  • bonus - open the lock
• Liquid recognition
  • perception - recognize liquid in a bottle, glass
  • manipulation - sort empty vs. full bottles
• Gesture recognition
  • perception - recognize a set of human gestures related to guiding a robot such as pointing, come, stop, grasp etc.
  • manipulation - gestures will command some action, e.g. grab glass
• Sorting clothing types
  • perception - recognize whether its a shirt, pant, etc.
  • sort into different piles
• Clean out whiteboards
  • perception - find all whiteboards in a room, document what’s on them
  • manipulation - erase whiteboards
  • bonus - erase whiteboards except writing within a red rectangle

New:

• Find and pick up textured and textureless items from a table
  • perception -- Recognize textured and textureless non-transparent, non-shiny rigid objects
    • From our 35 textured items but including bowls, mugs, plates, staplers.
  • manipulation -- clear the table
  • bonus: into sorted bins
  • bonus: with dirty dishes
  • bonus: set the table.
• Recognition of vegatables
  • perception -- Recognize 10 or more vegetable types
  • manipulation -- pick them up and sort
  • bonus -- be able to slice one or more of them
• Detection of wires and cables
  • percepiton -- be able to detect potentially tangleing wire and cables on the floor
    • Same for ropes on a table
  • manipulation -- untangle a rope

Grading

Grading will be done on 4 criteria below:

1. 15%: Homeworks. Students are allowed 1 free late pass as long as all homework is in by Project Kickstart date.

2. 25%: Project presentation PHASE 1

3. 20%: Project presentation FINAL

4. 40%: Project

The homework will tend to be "easy", they are all just individual tools that you will or may use in the projects. The set up time you might have to spend for the homework (say by loading and compiling libraries) is the same time you would have to spend anyway to get going on your projects.

Do not fear taking one of the more challenging projects, we know what areas are hard and will grade accordingly. Pick projects you really want to do and aim at substantial, publishable work (though we realize that publication will probably involve a post class work). Many of these projects are things, that if they work, will become part of the standard robot "skill set" with your names attached. Most of the work really requires that you have good research skills and analysis, things that are not formally taught but will matter much more to your futures than degrees etc.

Course Schedule

Tue,Thurs, 5:15-7:00pm.

Instructor Bios

Robotics is a vast field, to bring this together, the instruction will be shared among 4 lecturers who are expert in computer vision, 3D perception, motion planning and numerical simulation.

• Dr. Gary Rost Bradski:

  • Area: Computer Vision

  • Research Interests: Perception for manipulation, especially of objects (flexible/transparent/articulated) that have been hard to automate

  • Gary is a Senior Scientist at Willow Garage and Consulting Faculty at Stanford. Gary founded and still manages the Open Source Computer Vision Library (OpenCV) and was a lead on the computer vision team of Stanley, the robot that won the $2M DARPA Grand Challenge. He also helped found the Stanford Artificial Intelligence Robot (STAIR) project at Stanford. Gary's book on computer vision has been a best seller in computer vision and machine learning for 3 years now: Learning OpenCV: Computer Vision with the OpenCV Library.

• Dr. Sachin Chitta:

  • Area: Motion Planning and Manipulation

  • Research Interests: Mobile Manipulation, Real-time Motion Planning, Learning for Manipulation

  • Sachin is a Research Scientist at Willow Garage, Inc. His areas of interest include planning, sensing and control for mobile manipulation. He was a post-doctoral fellow at the Grasp Laboratory in the University of Pennsylvania working on modular robots and quadruped walking from 2005 to 2007. He graduated with a PhD in Mechanical Engineering from the University of Pennsylvania in 2005. Prior to that, he obtained a B.Tech in Mechanical Engineering from IIT Bombay in 1999. He joined Willow Garage in 2007 and has been working on motion planning, learning for manipulation and navigation with moveable objects. His ultimate goal is to create more capable, safer robots that are easier to use by non-expert users.

• Dr. John Hsu:

  • Area: Physics Simulation

  • Research Interests: Accurate simulation of perception, kinematics and dynamics. Transfer of simulated learning to real world robotics.

  • John received his Ph.D. in Aeronautics and Astronautics from Stanford University in 2004. His work included algorithm development for solving complex unsteady flow and finite element structural dynamics. Recently he's leading development for rigid body dynamics simulation at Willow Garage.

• Dr. Radu Bogdan Rusu:

  • Area: 3D Perception, 3D Processing

  • Research Interests: Real time perception and 3D semantic mapping in the context of mobile manipulation platforms.

  • Radu is a Research Scientist at Willow Garage and founder of the Point Cloud Processing Library (PCL). Dr. Rusu received his Ph.D. in Computer Science from the Technische Universität München (TUM), Germany in 2009 with summa cum laude, and his postgraduate diploma and engineering diploma (M.Sc. Eng. equivallent) in Automation and Computer Science in 2005 and 2004 respectively from Technical University of Cluj-Napoca (TUCN), Romania. During his Ph.D. studies, Dr. Rusu was affiliated with the AIC (Artificial Intelligence Center) at SRI (Stanford Research Institute) as an International Visiting Researcher working on the Leaving Flatland project, and more recently, Willow Garage, working on 3D semantic mapping techniques for mobile robots. You can find out more about his work by visiting his personal web page.

• Dr. Caroline Pantofaru:

  • Area: Person Perception

  • Research Interests: Using 2D+3D to find and track humans, especially for the purposes of interacting with robots.

  • Caroline completed her Ph.D. at The Robotics Institute, Carnegie Mellon University in May 2008 under the supervision of Martial Hebert. She also spent some time in Cordelia Schmid's group at INRIA in Grenoble, France. Her thesis, titled Studies in Using Image Segmentation to Improve Object Recognition, focused on issues related to using the image regions generated by unsupervised image segmentation to recognize objects and accurately identify their shape, especially in a weakly supervised training scenario where accurate object outlines are unknown.

Previous Years

• Stanford_CS324_PerceptionForManipulation/Spring2010