Exploring Tekkotsu Programming on Mobile Robots:

The Tekkotsu Simulator

Prev: Vision
Up: Contents
Next: Visual Routines

Contents: Purpose, Compiling the simulator, Running the simulator, Image and posture files, Image/posture sequences, Debugging with GDB, Injecting events, Setting breakpoints

Purpose of the Simulator

The Tekkotsu simulator allows you to run behaviors on your computer instead of on the robot, using pre-recorded sensor data. This provides several benefits: The simulator is not complete: it does not yet simulate the effects on the world caused by moving the robot's body. It doesn't simulate the world at all.

This chapter provides only a basic introduction to the simulator. More advanced features are discussed on the simulator page in the Developer Resources section of the Tekkotsu web site.


Compiling the Simulator

You can compile the simulator by typing "make sim" in your project directory. The result will be an executable file called tekkotsu-ERS7 (or tekkotsu-QBOTPLUS, depending on the robot target model selected in your project/Environment.conf file.) This sounds simple enough, but there are a few caveats:

The project/Environment.conf file defines both a target model (TGT_ERS7 for the AIBO, QBOTPLUS for the Qwerkbot, etc.) and a default target platform. The target platform is APERIOS for the AIBO, but the for the simulator it is LOCAL, since the simulator runs on your local host. Typing "make sim" forces the target platform to LOCAL for the current compilation, so that you don't have to edit the Environment.conf file. For robots like the Qwerkbot which are teleoperated, the target platform is always LOCAL.


Running the Simulator

Once "make sim" has completed successfully, you can run the simulator from your project directory and talk to it via its command line interface. You can talk to the robot it is simulating by using the ControllerGUI.

Talking to the Simulator

  1. In your project directory, type "./tekkotsu-ERS7" and hit return.

  2. The simulator will print out a bunch of start-up information. When this ends. hit return again and you'll see the simulator command prompt, "HAL:ERS7>". (HAL stands for Hardware Abstraction Layer.) You don't need to know any simulator commands to make basic use of the simulator, so we won't go into these commands now.

  3. Start up the ControllerGUI. For a hostname, give it the name or IP address of the computer on which you're running the simulator. On some systems you can use the name "localhost" to refer to your own computer without knowing its specific name.

  4. In the ControllerGUI, go to Root Control > Mode Switch, and start the Hello World behavior. You will see a bunch of lines print out in the simulator window, showing output that was sent to cout, serr, and several other streams. On the AIBO you would have to telnet to port 59000 to see these messages, but with the simulator this is not necessary.

  5. The full Tekkotsu functionality is available to you through the ControllerGUI, but not everything works the same as on the AIBO. For example, you can go to Root Control > Status Reports > Battery Check, and a battery status report will be printed in the simulator window, but all the values will be zero. The simulator doesn't bother simulating the AIBO's power limitations.

  6. The cleanest way to exit the simulator is to type ^D (control-D) or "quit" or "exit". There are other ways to exit, such as by typing ^C (control-C), but these force an abort, which causes a lot of error messages to be displayed.

Note: sometimes when the simulator exits abnormally, some of its processes are left lying around. If that happens, the next time you run the simulator you may experience interference from these old processes, e.g., ControllerGUI may be unable to connect. Use the command "ps -a" to check which processes are running, and do "killall -9 tekkotsu-ERS7" to kill any lingering ones.


Supplying Image and Posture Files

You can supply the simulator with a series of camera images to use as input to your behavior. Currently these must be JPEG files in RGB format or PNG files in YUV format. Vision applications often want to know the camera pose at the time each image was taken, so the simulator also provides a way to load posture files.

Recording Image and Posture Data

  1. You'll want to record your images at full resolution, so begin by starting up the ControllerGUI and double clicking on the "Take Snapshots" script. (See the section on Collecting images in the Color Image Segmentation chapter for instructions on how to set up this script.)

  2. Start up the Raw Cam viewer, or kill it and restart it if it was already running when you ran the Take Snapshots script. Point the AIBO's camera at something interesting.

  3. Make a temporary directory to hold the image and posture files.

  4. In the Raw Cam viewer, Click on "Freeze Frame" and then "Save Image". Check the box that says "Save joint positions as well". Then specify a filename in the temporary directory you created, such as pic01.jpg. Be sure to store your data in RBG format if using JPEG files, or YUV format if using PNG files.

  5. Click "Unfreeze", and repeat the steps above to take as many additional pictures as you like.

  6. When you've finished collecting data, look in the temporary directory. You should see a series of .JPG or .PNG files, and a parallel series of .POS files. These posture files are human readable, but they use a more compact format than the ones created by the Posture Editor.

By default, images are taken from the project/images directory, and postures are taken from the same place. Create a project/images directory now if you don't already have one.

Using a Single Stored Image with the Simulator

  1. Move a single JPEG or PNG file and its associated posture file into the project/images directory. Make sure there is only one file of each type in the directory.

  2. Start up the simulator, and connect to it with the ControllerGUI.

  3. Click on the Raw Cam or Seg Cam viewer and the image should pop up on your screen.

  4. In the ControllerGUI, go to Root Control > File Access > Posture Editor, and you will see the joint positions that were loaded from the posture file.

If you put multiple files in the project/images directory, the simulator will loop through them, in alphabetical order, at 30 frames per second. Try it and see. There are ways to slow or stop this process, discussed below.


Image/Posture Sequences

The AIBO records camera images every 33.3 msec, yielding 30 frames/second. New sensor frames arrive every 32 msec, which means the AIBO can generate posture files at 31.25 frames/sec. Thus, the simulator does not expect a one-to-one correspondence between image and posture files. It loads the files in the images and sensors streams independently. For each stream, the files are loaded in alphabetical order. If you want to test the AIBO's response to a moving stimulus, or to a stream of images taken as the head moves around, you can record such a sequence by not clicking on "Freeze Frame"; in that case the Save button is labeled "Save Image Sequence" instead of "Save Image".

But for some kinds of experimentation you will want to cycle through a handful of carefully selected images, and for each image, you will need to know the camera pose at the time the image was taken. In this case you want to force the image and sensor streams to stay in synch, and you want to be certain that the corresponding sensor data have been loaded just before each image becomes available for processing. To do this, you will need to adjust the frame rate of the sensor stream so that it matches the image stream, and you will need to set the start time of the sensor stream so that it just precedes the start time of the image stream. Then both streams will remain in synch.

Using Multiple Stored Images with the Simulator

  1. Move several of the image and posture files you recorded previously into the images/ subdirectory.

  2. Invoke the simulator with the following command line switches. freeze tells the simulator not to move to the next image until you give the advance command. Setting Vision.Verbose=true causes the simulator to announce each image file as it's loaded.
       ./tekkotsu-ERS7 freeze Vision.Verbose=true
    Note: you can also take these actions from the simulator's command line, using the commands freeze and set Vision.Verbose=true. However, if the simulator starts up in un-frozen mode, the image and posture files will already be out of sync.

  3. Start up the ControllerGUI and turn on the Raw Cam viewer. Type status to see which files the simulator will load next.

  4. Type advance in the simulator and you should see the first camera image. (Alternatively, go to Root Control > Vision Pipeline in the ControllerGUI and double click on "Advance Frame".) Each time you type advance or double click on "Advance Frame", the image and sensor information should advance again, and eventually loop back to the beginning of the sequence.

Explore more:

  1. Turn the "Advance Frame" command into a ControllerGUI script so you can invoke it from the Scripts menu.

  2. Type set to see the current settings of all simulator variables.

  3. Try set Speed=0.025 using the simulator's command line interface. What do you see in the Raw Cam window?

  4. Boot the AIBO and use the Walk Control to drive the AIBO around. While doing so, record a 10-15 second image sequence using the RawCam's "Save Image Sequence" function. Be sure to check the box to record posture files as well. (You must specify a filename prefix, like "myseq", in the Save Image Sequence dialog box, and the ControllerGUI will generate files myseq000.jpg, myseq033.jpg, and myseq000.pos, myseq031.pos, etc.) Move these files into the simulator's images/ directory and play them back at half realtime by setting Speed=0.5.


Debugging with GDB

GDB is the Gnu debugger. There are several good GDB tutorials on the web. You can also learn about gdb by typing "gdb" to run the program, and then typing "help" at the (gdb) command prompt.

In this section we're going to write a program that can crash the AIBO. Debugging such programs is difficult because there is no way to examine the program's state after the AIBO crashes. But with the simulator, we can use gdb to examine the execution stack and determine the cause of the crash.

#ifndef INCLUDED_DstBehavior_h_
#define INCLUDED_DstBehavior_h_
 
#include <iostream>
#include <sstream>

#include "Events/EventRouter.h"
#include "Events/TextMsgEvent.h"

using namespace std;

class DstBehavior : public BehaviorBase {

public:
  DstBehavior() : BehaviorBase("DstBehavior") {};

  virtual void DoStart() {
    BehaviorBase::DoStart();
    cout << getName() << " is starting up." << endl;
    erouter->addListener(this,EventBase::textmsgEGID);
  }
 
  virtual void processEvent(const EventBase &e) {
    string* menu[4];
    menu[1] = new string("appetizer");
    menu[2] = new string("main course");
    menu[3] = new string("dessert");

    const TextMsgEvent &txtev = *dynamic_cast<const TextMsgEvent*>(&e);
    const string &userinput = txtev.getText();
    istringstream ins(userinput);
    int item;
    ins >> item;
    cout << "Now serving item " << item << "."
	 << "  Enjoy your " << *menu[item] << "!" << endl;
  }

};

#endif

First we'll demonstrate the crash:

Crashing DstBehavior

  1. Compile the simulator with the version of DstBehavior given above.

  2. Run the simulator, and start up the ControllerGUI.

  3. Activate the DstBehavior.

  4. In the ControllerGUI "Send Input" box, type the following:
        !msg 1
    
    This should produce correct output.
  5. Now send the following input:
        !msg barf
    
  6. The simulator will crash and print out a bunch of information. Try running it again and supplying the following input:
        !msg 0
    

Now we will use gdb to determine the cause of the crash.

Diagnosing Crashes with GDB

  1. In order to use gdb with the simulator, you will need two separate shell windows: one for each program. Begin by running the simulator in the first window. When the simulator starts up, it will print out some lines that look like this:
    Spawning processes...
      Initializing runlevel CONSTRUCTING
                                   Main:  ProcessID::getID()=0   pid=7736
                                 Motion:  ProcessID::getID()=1   pid=7737
                              SoundPlay:  ProcessID::getID()=2   pid=7738
                              Simulator:  ProcessID::getID()=3   pid=7739
    
    Note the value 7736 shown in red above. This is the pid (process id) of the Main process, which is where your behavior will run. When you try this yourself, you will see a different pid value each time you run the simulator.

  2. In the second shell window, cd to the project directory containing the simulator. Then use gdb to attach to the Main process by typing the following:
        gdb sim-ERS7 7736
    
  3. When gdb starts up, it attaches to the Main process and suspends it. Then it prints its command prompt: "(gdb)", and waits for instructions. If you try to use the ControllerGUI at this point, you will get no response, because Main is suspended.

  4. In the gdb window, type "c" (for "continue") and hit return. This resumes the Main process.

  5. Use the ControllerGUI to activate DstBehavior.

  6. Send the input !msg 3 and verify in the simulator window that the correct output is produced.

  7. Send the input !msg barf. You should see no output in the simulator, and the ControllerGUI should no longer respond. The behavior has crashed.

  8. Switch to the gdb window and you will see a message saying "Program received signal SIGSEGV, Segmentation fault." This means the program tried to access a nonexistent memory location. gdb also shows its best guess as to the line in the program where the error occurred. (When statements are spread across multiple lines, this guess might be off by a line or two.)

  9. Now try out the following gdb commands:

    bt   backtrace: show the execution stack
    p item print the value of the variable named 'item'
    p userinput print the value of 'userinput'
    p menuprint the array 'menu'
    p menu[2]print element 2 of 'menu'
    p *menu[2]print the string pointed to by menu[2]
    i locinfo locals; show local variables in this stack frame
    llist: show source lines
    l -list backwards: go earlier in file

  10. Use the "help" command to view the documentation on each of the above commands, e.g, "help p" shows the documentation for the print command.

  11. Type "q" to quit gdb.


Injecting Events via ControllerGUI

Using the ControllerGUI's !post command, you can manually inject events into either the AIBO or the simulator. Not all event types are supported at present. Most importantly, timer events are not supported. But other simple event types such as button presses, statemachine events, and audio events are supported.

Here's a little demonstration program that responds to head button press events by speaking a digit:

#ifndef INCLUDED_DstBehavior_h_
#define INCLUDED_DstBehavior_h_
  
#include "Behaviors/BehaviorBase.h"
#include "Events/EventRouter.h"
#include "Sound/SoundManager.h"
#include "Shared/ERS7Info.h"

class DstBehavior : public BehaviorBase {
 public:
  int counter;
  DstBehavior() : BehaviorBase("DstBehavior"), counter(0) {}

  void DoStart() {
    BehaviorBase::DoStart();
    erouter->addListener(this,EventBase::buttonEGID,HeadButOffset,EventBase::activateETID);
  }

  void processEvent(const EventBase&) {
    advance_the_counter();
    char filename[30];
    sprintf(filename,"numbers/%d.wav", counter);
    sndman->PlayFile(filename);
  }

  void advance_the_counter() {
    if ( ++counter > 9 )
      counter = 0;
  }

};

#endif

We will use this sample program to illustrate event posting from ControllerGUI. The three arguments to the !post command are the event generator (buttonEGID), the source (HeadBut for the head button), and the event type (A for Activate, S for Status, or D for Deactivate).

Injecting Button Press Events

  1. Compile the demonstration program above, and start the simulator.

  2. Connect to the simulator with ControllerGUI, and activate DstBehavior.

  3. In the ControllerGUI's "Send Input" window, type the following:
       !post buttonEGID HeadBut A
    

  4. The simulator should display the message "Playing numbers/1.wav". (The simulator does not yet have sound support, so it won't actually play the audio files.)

  5. In the ControllerGUI's "Send Input" window, press the up-arrow key to recall the previous command, and hit Return.

  6. The simulator should display the message "Playing numbers/2.wav".

Explore more:

  1. The !post command can take a numeric source id instead of the symbolic value "HeadBut". Suppose you wanted to send an event indicating a right front paw button press. What numeric value should you use for the source id? (Hint: look in ERS7Info.h.)

  2. Notice that the definition of processEvent above takes an argument of type const Eventbase&, but doesn't name the argument. What does the compiler do if you supply an argument name?

  3. Modify the program to accept all types of button press events, and modify processEvent to print out the result of the event's getDescription() function. Then try sending different kinds of button press events to your program by varying both the source and the event type.


Setting a Breakpoint

Prev: Vision
Up: Contents
Next: Visual Routines