Using TOS as educational tool in a simulation course

I am a professor working in the program in Information Technology at Simon Fraser University. Improving instruction and facilitating active learning are my priorities. Computer Simulation is the favorite course that I teach. I was especially interested in offering a good project for my class in which students could be gaining hands-on experience in conducting simulation studies.

Once having started participating in the RoboCup in 2002, I got curious whether simulated soccer could be used for teaching simulation and modeling in the undergraduate curriculum. Although exercises with this application appeared to be emotionally engaging for the students, its complexity was a major concern, as students could get stuck in many small details. As the result, they would be just learning how to program in C++ under Linux and the soccer application itself. However, the objectives of my course are quite different.

So, I turned to the Tao of Soccer for three reasons:
1. TOS is way simpler than the RoboCup soccer; that would reduce the learning time for the students who must get well-familiar with the application before working on their projects.
2. TOS is a Java application; in my school, Java is the main programming language used in most courses.
3. Yu Zhang, the TOS creator, was within my reach; indeed, Yu had helped me a lot with making a qick start with TOS. He also has made some modifications to TOS on my request in order to adapt it for the eductaional purposes.

The project
The project in my simulation class lasts five weeks. The general idea is modifying the player behavior and measuring the resulting improvements to the simulated soccer team. By the end of the project students are expected to significanly improve the deafult soccer team. The main learning objective of the project is, getting hands-on experience in conducting simulation studies. Each student is expected to spend 40 hours working on the project. Students are organized in teams of three.

Individual assignments
Students are asked to post on the web board their bright ideas on how to better improve the player performance and how to conduct the simulation study. Discussing postings made by others is also encouraged. For each posting, student receives up to 4% towards the final mark, with the total set at 20%. This is creating incentives for the participation in the discussion; other students can learn from their peers. Questions are also welcome, yet not for the mark.

Team assignments
Week 1 assignment requires creating and submitting the project plan. Thus the instructor is making sure that students have got familiar with the requirements of the project and the reporting schedule, and that teams have started working together.

Week 2 assignment is very simple. Students are asked to gather statistics on the ball X-coordinate when one of the soccer teams has fewer players than the other. Then the confidence interval should be calculated and the number of observations determined for getting 95% and 99% confidence of that the mean X-coordinate is indeed biased. This exercise allows students getting familiar with TOS and estimating the duration of the simuation runs in the studies that follow.

Week 3 and 4 assignments require modifying player behavior and evaluating the gains, if any. In Week 3 only one modified feature should be investigated; however, in Week 4, two features and their combined effect should be studied. Students are free to choose any player features; they are also instructed to propose performance indicators for measuring the impact of the changes made to the player behavior. These issues are discussed in the online conference and in the workshops.

Deliverables
Of the three assignments due in weeks 2,3, and 4, only two were mandatory. Each team was thus expected to submit the following deliverables:
1. Project plan (Week 1)
2. One written report on the simulation study (Weeks 2-4)
3. One PPT presentation made in a workshop (Weeks 2-4)

Workshops
Each week, we had a workshop (2 hrs 50 min). The activities in the workshops were distributed as follows.

Week 1. Instructor introducted TOS to the students and explained for them what tasks they are expected to accomplish, how to report the results, and how students will be evaluated.

Weeks 2-4. Teams of students in turns were making presentations on their progress in the simuation study. The presentations and the instructor's comments were helping other teams to better work on their assignments.

The Tournament
In the end of Week 4, students participated in the simulated soccer tournament by playing in groups of 3-4 teams. In each game, a representative of a third team was serving as the referee; this individual reported the result by posting it on the web board.
In Week 5 workshop, the four best teams who had made it to the semi-finals presented their accomplishments and played the final games for places 1-4 in the tournamnet. The photographs posted by Yu Zhang in this blog show this event.

Evaluation
A small fraction of the final mark in this course depends on the results in the tournament. Students would have received 100% mark if their performance in all assignments was excellent and their team has made to the top 2/3 in the tournament, i.e. was playing not too bad. So, top 2/3 teams were actually receiving a bonus. The winner in the tournament thus got 9% bonus towards the final mark.
The bulk of the mark (50%), however, was attributed to the two mandatory reports on the simulation studies (1 verbal+PPT and 1 written). Contribution to the online conference (20%) was making a difference between A+ and B-. The rest marks students received for attending the workshops (8%), sumitting the project plan (10%), serving as referees (5%), and rolling out their soccer teams to the competiton (5%).

As an instuctor, I believe that this evaluation scheme creates the optimal balance between encoraging students to learn the simulation methodology and engaging them emotionally.

My expectations of the futurte developement of TOS

I have some concerns about possible improvements of the TOS soccer team whose code could be made available for free download by some enthusiasts. As an educator, I really do not want this to happen.

I would say that as it is now, TOS 1.4.2 meets all the requirements for using in a course like mine. The default team plays soccer very poorly; yet the code contains many placeholder methods; it is easy for the students to identify them as potential objects for the improvement. With the change of only one player feature in Week 3, students could easily have gained 3 to 7 in the score difference while playing against the default team. For obtainng 95% confidence of this difference in the score, it was sufficient to run 20-30 games, which takes up to 10-12 hours. In five weeks, students have to make up to three simulation runs like that if everything goes smoothly. I believe that this is the upper limit that students can afford while conducting simulation studies according the book. The duration of the simulation runs is thus the major limiting factor.

If significantly improved teams emerge and their code made available on the sourceforge Web site, my future students would be compelled to deal with these more sophisticated applications. Thus I foresee two problems: (1) students would need to spend more time on learning the code and (2) further improvement would not be so apparent and would require way longer simuation runs than nowadays.

So, if publishing improved TOS teams is inevitable, I would recommend to create a separate educational version of TOS that would not be compatible to the versions which TOS enthusiasts would be futher improving.

Vadim Kyrylov
http://www.sfu.ca/~vkyrylov