Ronald D. Owston & Herbert H. Wideman[2]
York University
Use of the Web for teacher professional development has now become relatively commonplace. Formal online courses, focussed collaborative projects, knowledge building groups, discussion forums, resource portals, and online mentoring are just a few of the applications that have been made of the Web to develop teacher competencies. Owston (1997), while optimistic about its potential, nevertheless urges caution in embracing Web-based learning unless evidence is obtained on its effectiveness. Otherwise, we may just be embarking on another educational bandwagon. Windschilt (1998) further points out that much of the literature in the field still tends to be anecdotal and falls short of asking critical questions about teaching and learning. Hence continual efforts must be made to evaluate Web-based teacher education projects and programs. What is needed is not to simply ask the question "Does it work?" We need to understand the conditions under which Web-based professional development is most effective in fostering teacher-centered, constructivist pedagogies and communities of practice (Bonk & King, 1998). Evaluation designs need to focus on the context in which the innovation occurs, including such factors as the technology used to deliver the innovation, the characteristics of the individuals involved, organizational factors, and teaching and learning issues (Heinecke et al., 1999; Sherry et al., 1997; Neale & Carroll, 1999).
This paper focuses on the authors’ recent efforts to evaluate three Web-based projects aimed at developing various aspects of pre- and in-service teachers’ professional competencies. The first project examined the creation of a knowledge building community between a pre-service candidate, a host teacher, and high school students using the Web-based groupware tool Knowledge Forum. The second project was an evaluation of two sites participating in Canada’s SchoolNet GrassRoots program. This program provides funding to schools to develop Web-based collaborative classroom projects. Although the program’s focus is not on teacher development per se, our evaluation centered on how teachers benefited professionally by developing and leading GrassRoots projects in their schools with their colleagues. A pilot video-on-demand service to high school teachers was the third project we evaluated. This service aimed at providing teachers with video clips from a public educational television station that they could use to develop classroom teaching materials. The approaches employed in these evaluations included classroom observation, the taking of field notes, interviewing, document analysis, and other procedures similar to those recommended by Owston (2000). We begin with an overview of the three evaluation projects. From an analysis of the three projects, we then describe and discuss (1) project outcomes that promoted teacher knowledge building, (2) contextual factors that influence success of Web-based professional development activities, and (3) methodological issues that emerged from the studies.
New teachers cannot be expected to be able to initiate and support collaborative knowledge building projects with their students if they are not given the opportunity to develop the requisite understandings and skills in their pre-service experience. Few of them would have had any exposure to this approach to learning in their own schooling. The transformation in perspective and practice required can only come about through sustained leadership and support from teacher educators at all levels. Teacher candidates have to be given the opportunity to move beyond theoretical understanding through immersion in classrooms in which knowledge building is taking place, working with host teachers who model desired practices and provide the ongoing coaching necessary for sustainable change.
This pilot study--the first stage of what we hope will be a larger investigation— examined the experiences of one teacher candidate whose practicum placement provided just such an opportunity. Her host teacher was engaged in knowledge building activities with her Grade 12 students in novel studies, a process facilitated by the use of the Knowledge Forum software developed by Scardamalia and her associates (Scardamalia, Bereiter, & Lamon, 1994). Knowledge Forum, the direct descendant of the Computer Supported Intentional Learning Environment (CSILE), provides an intra- or internet based multimedia communal database which makes use of shared data structures and cognitive supports to foster joint knowledge creation around a topic or problem under investigation. Students enter their developing ideas into knowledge base notes concerning their topic, notes upon which others can then reflect and respond or seek further clarification. Emergent lines of enquiry in any database are networked together and organized via links in a graphical representation that all students can access, and all entries are tagged with the author’s name, making it possible for teachers to track individual students’ contributions and respond to them. Prompts—chosen in advance by the teacher—are selectable during the creation and response process to scaffold higher levels of cognition and self-monitoring. A primary goal of the environment is to support students in the development of mature, autonomous learning skills that allow them to participate in collaborative knowledge building.
The school hosting the pilot study, Kent Secondary School, was in a large urban area, and served a working class community. Its students were predominantly the children of first generation immigrants, or had immigrated themselves, and for most, English was the second language in the home. The grade 12 English class in which the project was implemented was observed several times when students were working with Knowledge Forum under the guidance of both the teacher and the teacher candidate. Towards the end of the project, the teacher and the candidate were interviewed and the interviews transcribed for analysis.
The host teacher participating in the pilot, Sarah Woolcott, was head of the school’s English department; she also served as the on-site Adjunct Professor for a nearby faculty of education, and in that role coordinated the activities of the approximately 15 teacher candidates at the school. (Currently she is seconded full time to that faculty, where she is serving as a course director responsible for practicum courses in the pre-service program.) She had prior experience using CSILE in the 1998-1999 school year with Grade 12 students as a vehicle for discussing the Joy Luck Club. In the summer of 1999 she attended a workshop on the use of CSILE in the classroom conducted by Mary Lamon, a researcher/facilitator working with Marlene Scardamalia. Several of the school’s teacher candidates attended as well. In the fall of 1999, the simplified, Web page-based version of CSILE was used by teachers and a few teacher candidates—including Elizabeth, the candidate who participated in the pilot—as a forum for their program planning discussions.
Sarah had felt that much of the traditional Grade 12 English curriculum was hard for her students to relate to, given their multicultural backgrounds. She wanted to deeply engage her students in the study of a range of works written by authors with diverse racial and cultural perspectives. She thought that in-depth project work on student-chosen novels offered the best approach to reaching this goal, and she felt from her prior experience with CSILE that the use of Knowledge Forum could significantly facilitate meaningful discussion and reflection on the novels amongst her students. She was able to fit the project into the required curriculum by organizing it as an Independent Study Unit (ISU) for her class, an officially sanctioned vehicle for Grade 12 students which allows them to pursue independent study of self-chosen literary works for 20% of their final course grade.
Sarah introduced both the project itself and the concepts behind the Knowledge Forum by having students respond to the question “Why do you think we should study multicultural literature in high school?” on index cards which were posted on a board and read by other students. Sarah reported that as students read others’ thoughts they were sometimes led to reconsider and expand their own notions. About this time the facilitator of the earlier summer workshop on CSLIE came to Sarah’s class to introduce the Knowledge Forum software and how it could be used for knowledge building. In the class discussions that ensued, students self-selected into one of five novel discussion groups that had been mutually agreed on: Oprah’s Book Club novels, African American writers, Magic Realism, Monsters, and Potpourri of Novels. Sarah’s goal was to have students integrate the insights that they developed from reading individual genre works with the perspectives of other students through collaborative knowledge building in the Forum in order to understand the genre more fully. Each group had a Knowledge Forum database set up for their discourse. Their initial discussions were responses to general questions Sarah posed to each group asking students to discuss their thoughts about their novels, and how they connect with the novels others in their group are reading (based on those students’ initial comments about these novels). The ISU had three graded components that were set by the teacher, but the weighting for each was negotiated with the students. Sarah held discussions—some on Knowledge Forum—looking at grading rubrics, asking her students how they wanted to be evaluated. The facilitator put forward some procedures used in a graduate course, where these ideas were discussed on Knowledge Forum, and this triggered much discussion. It was decided that a second view [database] for each group would be developed once students started formulating essay theses about their selected novels. It was eventually decided, after a few weeks of discussion and reflection, that participation and Knowledge Forum discussion would be given the greatest weight in grading (50%); each group’s presentation to the class on their novel genre was weighted at 20%; and the individual essay on a student’s chosen novel was to be worth 30% of the ISU mark.
The teacher candidate hosted by Sarah during the 1999-2000 school year, Elizabeth, observed and taught in Sarah’s classes one day a week throughout the year; in addition, she had two extended blocks of teaching time – two weeks in February, and another four weeks in April-May. It was during the latter block that most of the student ISU work was undertaken. Elizabeth was enrolled in the second year of a three year Concurrent B.Ed. programme, and was also in her third year of her B.A. (hon.) programme, majoring in English. She had some experience using computers in her fist year practicum, and had made extensive use of a discussion forum “a lot like the Knowledge Forum” in a B.Ed. course. She had attended two one-day training sessions given by Mary Lamon in the fall and early winter on WebCSILE and Knowledge Forum. She had also done background reading on knowledge building, studied the Knowledge Forum manual, and discussed knowledge building and the software with Sarah prior to her involvement with Sarah’s project. In the fall, she and a teacher at the school had been delegated to lead a WebCSILE-based project development effort involving relevant school staff and teacher candidates, the goal of which was to create a three or four week media unit for Grade nine. Sarah supervised the teacher candidates in this work, and found she had to intervene to help them develop a process rather than a product orientation:
They
didn’t understand what the Knowledge Forum really was at first. So it was
good for them to do this planning, because they were making this unit about
making a magazine and it was becoming very, very--they wanted this product,
this product, this product, product, product. And I said I don’t really
care what the product is, it’s the ideas. So it’s what we talked
about, the ideas float around and you’ll find out what it is.
WebCSILE was also used in the fall by Sarah, Elizabeth, an a few others to formulate plans for the ISU project in grade 12. Sarah described how she guided this process:
I was
more interested in, and I think we talked about, the process of how you get to
what you’re going to write about, like a thesis or whatever. How do you
get there, what are you talking about? So we came up with an idea that the
independent study should be that they were going to focus them in groups;
they’re going to have some sort of common theme; they’re going to
start a discourse on whatever the issue, what the students’ thoughts
[were]… like give them a general idea, what are your thoughts, theories
and ideas and let it go from there.
Elizabeth was an active participant with Sarah in the planning and development of the ISU project. She worked with her to select the groupings for the novel studies, and to articulate the initial set of questions used to trigger the forum discussions. She also took on the responsibility of mentoring students who had come to the four computers at the back of the class to use the Knowledge Forum when she herself was not teaching the class. Elizabeth would also bring students to the room from another class. She read student postings as they were created and where she saw students working at superficial levels modeled deeper inquiry strategies by suggesting the use of appropriate prompts in Knowledge Forum, asking for further articulation, and posing questions designed to trigger reflection on the notes and responses being written. Over the six weeks of the project, Elizabeth was able to read most student postings, and she typed her own responses to some student entries, although time constraints prevented her from doing as much of this as she would have liked. Elizabeth described how she mentored her charges:
I would
bring a group down here and a lot of times they worked in partners, because
they were in the same theme group. So we would help them to figure out what to
write; at the very beginning they didn’t know what to write. So
we’d help them come up with what kind of questions to ask themselves, how
to use the scaffolds. Yea, so basically reading through it and seeing who
needed to …giving them some direction, because there is a lot on there
and in the end, if there is no direction it kind of starts to seem like they
are just doing it for the sake of just putting a few lines down…. We
would show them examples of when they can substitute things that they’re
saying with scaffolds, which might give them a sense of better direction.
Sarah also spent considerable time monitoring student activity on the Knowledge Forum, and sometimes both would be found at the back of the room watching or discussing student contributions. In our observations we found that they management issues rarely consumed their time. Students were very intent and involved in reading and responding to posts, and while a minority were short and superficial (“I agree. I think you are right”) most were much more considered, sometimes articulating lines of argument extending several paragraphs. There was some tendency to offer opinion without buttressing it with evidence, a practice both Elizabeth and Sarah worked to overcome in their notes and verbal discussions. (It was not our goal here to undertake an analysis of the students’ Knowledge Forum work in detail – see Lamon, Scardamalia, Shaw, & Fullan, 2001.)
Reflecting on her experience at the end of her practicum, Elizabeth stated that although the time required to help initiate, monitor, and sustain the students’ knowledge building was considerable, she found the experience very valuable. She felt she still had much to learn, both about the operation of the software – in particular, she wanted to know how to customize the scaffold prompts, and how to include multimedia objects in the knowledge bases—and about its incorporation into classroom practice. She felt that Knowledge Forum would have great utility as a medium for reflection on practice amongst teacher candidates themselves, as well as with host teachers and course directors at the faculty, provided the prompts used were customized (“Sometimes the things that I had to say, that I think were good points that might have been useful, may not fit [into the existing prompt structure]”). She saw it as having one major advantage over private journaling in its ability to foster collaborative reflection through the interaction of different perspectives. She found that her WebCSILE interactions with experienced teaching staff in the fall, when the Media Studies unit was being developed and plans for the ISU being brainstormed, provided critical perspectives she valued.
In the classroom, Elizabeth had observed that students who had been initially unable or unwilling to make significant contributions to their knowledge bases were guided by reading the postings of the “stronger student” who contribute a lot”; they provided models that gave the weaker students a better sense of how to participate. Others who rarely shared comments in class began to participate strongly in forum discussions—Knowledge Forum’s asynchronous, less public environment appeared to offer greater psychological safety for certain students. Asked what limitations or problematics she saw in the students’ activities, she indicated that a minority of students had a tendency to branch out of the group’s focus and had to be guided back to on-topic contributions. The forums also required regular monitoring to ensure that incidents of “inappropriate language” were quickly caught, and a few students had to be educated in collaborative etiquette so that their disagreements with did not descend into personal attacks. She had observed that at the start of the forum building, many students had not completed their novel reading, and so were not prepared to contribute at first and had to play “catch-up” later. Despite these limitations, she concluded that would definitely try to create a similar kind of knowledge building environment with the Knowledge Forum software when she starts to teach, because it fosters interaction around ideas and galvanizes participation by students who would otherwise be unresponsive.
For Elizabeth, the role of her host teacher in affording opportunities for her to learn about and engage in knowledge building with students was critical. She thought it untenable to expect a teacher candidate to be able to implement this pedagogy in a practicum if the host was not familiar and comfortable with the kind of teaching that Knowledge Forum facilitates, since hosts and teacher candidates must agree on the latter’s activities in the class. She recommended that teacher candidates become fluent in the use of Knowledge Forum early in their program, so that they can address the demands of fostering knowledge building in their practicum settings without at the same time being burdened with mastering a rather complex tool.
Sarah viewed Elizabeth’s participation as important to
the project’s success. Neither of them would have had the time to monitor
all threads of the knowledge building database, but between the two of them
they were able to do so, since the software had the ability to let each know
what branches had been read by the other. More importantly, one of the two
could be teaching the class (or segments of it) while the other worked with
students using Knowledge Forum. Sarah thought Elizabeth was an outstanding
teacher candidate, who had a good grasp of the purpose behind the use of
Knowledge Forum and was able to guide and scaffold students as needed.
Significant and sustained knowledge building occurred in Sarah’s class (Lamon et. al., 2001). It is highly doubtful, given the mentoring and supervisory demands the project imposed, that the same level of success could have been achieved if Sarah had not had assistance from Elizabeth or else someone equally competent. Elizabeth was able to be effective in guiding students to appropriate discourse in Knowledge Forum, fostering student reflection through both verbal and Knowledge Forum interactions. She was also able to model and scaffold metacognitive processes of self-monitoring and assessment, helping students choose and work appropriately with the cognitive prompting that is part of Knowledge Forum. She had a significant (if subsidiary) role in the project creation and planning process. Sarah appreciated her work as being effective and important to the project’s success.
For her part, Elizabeth found her experience on the project to be instrumental in shaping her teaching disposition. It drew her away from her initial product focus to one more concerned with monitoring and fostering the key processes in knowledge building. Despite the additional work it imposed, she was planning to pursue similar work in her own teaching career, using the Knowledge Forum software, the use of which she was eager to understand and extend more fully.
Both teacher and teacher candidate gained from this collaboration. Sarah was able to offer her students a rich educational experience that might not have otherwise been possible, and Elizabeth, under the tutelage of a master teacher, was afforded an opportunity to reorient her teaching to a more constructivist, student-centred perspective. While her conceptions of the process of knowledge building were not as sophisticated as Sarah’s, she seemed to have grasped the general principals involved and proved able to operationalize them in her practice with some success.
Our pilot, then, suggests that teacher candidates, given the right training and support, can though a process of tutoring, modeling, and reflective discussion be effectively educated in knowledge building pedagogy in a sustained practicum experience. There were, however, certain conditions present in the current case, which may not be present in a more “typical” teacher candidate practicum, and should these not hold elsewhere, the same level of success might be elusive. The host teacher was a pedagogical leader in her school, an innovative, motivated, and highly intelligent master teacher whose abilities were so appreciated that she was seconded to the faculty of education as a full time course director the year following this project. And the project itself had considerable support from Marlene Scardamalia’s knowledge building team; Mary Lamon gave a total of three daylong workshops that Sarah attended prior to the novel project, two at the school, and she also provided in-class training for Sarah’s students. Sarah herself had several extended experiences working with Knowledge Forum or its predecessors with students, teachers, and teacher candidates prior to the current project, and was enthusiastically committed to a knowledge building pedagogy. Elizabeth’s sustained presence in the school over the year afforded her the chance in the fall and winter to attend two of Mary’s workshops, and she had also worked with other teacher candidates and teachers in WebCSILE under Sarah’s guidance prior to the project. In sum, the level of preparation and support for this project on both the teacher’s and the teacher candidate’s part was much deeper than what might be expected in many teacher candidate placement contexts. This suggests that success in other contexts will be highly dependent upon placing teacher candidates with experienced hosts, and providing sustained training and support for participants.
GrassRoots is a national program run by Canada’s SchoolNet designed to promote “academic, employability and computer skills in Canadian youth by integrating information and communication technology (ICT) into learning” (SchoolNet, 2001). Through the program teachers are awarded small grants to develop, either by themselves or with colleagues, student projects that involve the creation of Web sites. We together with colleagues at Université Laval were contracted by SchoolNet to conduct case studies of three GrassRoots implementations in Ontario and Quebec (see Owston, 2001). The purpose of the research was to seek a deeper understanding of the implementation process at the schools, particularly in terms of how student teachers and regular classroom teachers acquired the necessary technology skills and how they integrated the program into their classroom practice.
The York team chose to do a micro level study of two individual GrassRoots school implementations in south central Ontario—Lake Elementary and Jones Elementary and Middle School (not their real names). The Laval team on the other hand decided to do more of a macro level study of implementation across two urban school districts in Quebec. Most of the schools studied by the two teams had pre-service student teachers and an existing partnership established with one of the two respective universities.
Despite the different foci of the case studies the methodology used by both teams was largely the same and both teams used a common interview schedule. Data were collected through a variety of qualitative techniques including individual interviews, focus group interviews, questionnaires, observation, and, only in the Laval case, email discussion groups. Interviews were tape recorded and transcribed. The data were analyzed by studying the raw field notes and interview transcripts to determine emergent patterns or themes. These themes were then summarized and used as the basis for writing individual case reports. After the individual case reports were written, the case reports themselves were analyzed to determine themes that were common to all three case studies.
The first school studied, Lake Elementary, is a K-8 school serving a rural, predominantly middle class population. The school has a history of engaging in innovative projects in the natural sciences. Thus the school’s first GrassRoots project grant received in 1998-99 for the development of a Web site about the school’s arboreal and butterfly gardens was a natural extension to their curriculum. The project, which subsequently won a national SchoolNet award, was largely considered a success in that it generated a great deal of student and teacher interest. Furthermore, the teachers involved developed skills in Web page construction and the integration of ICT into the curriculum. The success of their first project led the school to apply for and receive another GrassRoots grant for the following year. This project focused on creating a website for an ant farm that was cultivated at the school. Teachers reported that students became highly motivated to work on the GrassRoots projects. In addition to learning ICT skills such as scanning, use of digital cameras, basic Web page creation, and word processing, students developed teamwork skills and greater self-esteem as a result of the project.
Smith Elementary and Middle School, where the second case study was conducted, is a K-8 public school with about 650 students and 35 teachers set in a middle class Toronto neighbourhood. It is located in an area of the city that has a high percentage of families of Russian and Korean origin, and for 63% of the students English is a second language. York University had a cohort of nine pre-service candidates placed for their practicum at this school. As part of their practicum student teachers were assigned to work on a collaborative GrassRoots project under the aegis of the school’s technology committee. The goal of the project was to develop a school Web site. The plan was to involve school students in the latter stages of the project, however because the project got behind schedule and because teachers did not feel ready to involve students, there was little active involvement on their part. The teacher candidates divided responsibility for project development into two subcommittees, technical and content, which was a pragmatic way of dealing the wide discrepancies in their ICT skills. Although all student teachers reported improvement in ICT skills as a result of the project, some content committee members felt that they would have benefited from more exposure to the techniques used for Web site construction and to the operation of multimedia equipment. Learning to work effectively as a team in a truly collaborative project posed another challenge for some of the teacher candidates, however as time wore on their teamwork skills improved substantially. The teacher candidates all found they had a clearer sense of the possible uses of ICT in education after being exposed to numerous educational Web sites and reflecting on their pedagogical potential.
The Quebec case study involved nine elementary schools and one high school from the first school district and one other high school from a neighboring district. In these schools a total of 14 GrassRoots projects were carried out, five of which involved pre-service teachers with the balance being implemented by regular classroom teachers. A wide variety of project types were represented in this sample, ranging from a video conference project where students made presentations to peers at other schools to a project where students created snowboard designs that were submitted over the Internet to real companies. The case study was also informed by a discussion forum and questionnaire responses from 16 other pre-service students who conducted GrassRoots projects in schools and districts other than the ones formally studied. Overall, the case study demonstrated that, despite the lack of training of both regular teachers and student teachers on the project-based approach to teaching and the use of ICT, considerable success was achieved in implementing GrassRoots projects.
From the case descriptions eight overarching themes were identified. These themes are related to student outcomes, teacher outcomes, curriculum, technical support, preservice issues, project characteristics, sustainability/transferability, and GrassRoots funds use. Most relevant to this paper are the teacher outcomes, curriculum, pre-service, and technical support themes.
Even though the primary focus of GrassRoots is on student ICT skill development, we tentatively advanced the hypothesis that the greatest impact of GrassRoots may well be on teacher skills. Teachers almost universally reported increased confidence and skill in ICT use and in organizing project-based instruction in their classrooms. For example, teachers found that the GrassRoots experience motivated them to learn Web site construction skills and to organize students to work in teams that focus on one aspect of the larger project.
GrassRoots appears to encourage teachers to take a fresh look at how they teach to particular student outcomes and/or their expectations for their grade. The grade 3 ant farm project described in the Lake Elementary case provides an excellent example of this. The school’s first GrassRoots project was related to a butterfly garden. Following on the success and enthusiasm of this project, teachers searched for a new theme for a follow up project. One of the teachers picked up from a professional development workshop the idea of creating an ant farm. The grade 3 teachers then built a Web-based GrassRoots project around the ant farm that covered the curriculum expectations in a unique and engaging way for students. Clearly, the GrassRoots programme was a strong incentive for teachers to re-think their approaches to teaching the curriculum.
We found considerable variation in the amount of technical support available to teachers when they ran into difficulties with their projects. At Lake Elementary reliable, timely support was available to the GrassRoots teachers. The preservice teachers at Jones Elementary and Middle had to band together to help themselves and, when not finding the support they needed at the school, had to rely on help from technically-oriented friends outside of school. Likewise some teachers in the Quebec cases stated that timely support was not forthcoming, leading them to feeling discouraged and to begin thinking of abandoning their projects.
SchoolNet has the goal of encouraging teacher candidates from faculties of education to develop GrassRoots projects during their practicum experience. Indeed, two of the three cases studied had preservice candidates. We frequently found that teacher candidates were placed with host teachers who had little or no ICT skills. These teachers, while universally supportive of their junior colleagues, often times simply let the candidates flounder as they were not able to help them with either the technical aspects or how the technology could be integrated effectively into the curriculum. Teacher candidates reported that if they had other fellow preservice students at the same school, they were able to support each other when the host teachers could not. While we did not have time to study how these support networks functioned, clearly they deserve closer examination in the future.
Digital streamed video-on-demand offers promising potential to enhance and enrich classroom teaching and learning. In theory, the medium can deliver to the computer desktop high quality educational resources keyed to curricular outcomes. Students can view the content on its own or build multimedia presentations and reports incorporating segments of the streamed video. Teachers can construct Web-based learning activities and instructional units that contain video components which explain, illustrate, model, or simulate concepts or events in ways that would be extremely difficult or impractical using traditional teaching methods or media. The reality today is that this technology is still in its infancy and not sufficiently robust for widespread use in schools, nor do networks and servers available to the vast majority of schools have the capacity for large-scale simultaneous delivery of content.
Public television stations, with their vast archives of educational programming, are ideally situated to provide streamed video-on-demand service to schools when the technology and infrastructure become mature. In preparation for this eventuality, we need to begin to explore the potential educational models and classroom delivery systems for video-on-demand in closely controlled and robust environments. In order to test the concept of streamed video delivery to the classroom, TVOntario (TVO) entered into an agreement with a Toronto suburban school board to conduct a pilot project at one of its high schools. We were contracted to develop a research plan to evaluate the pilot, to implement the plan and conduct the research, and to share the findings with the television network (see Wideman & Owston, 2000).
Our research plan specifically called for an examination of:
· how teachers are able to integrate the system into their regular classes
· the utility of the currently available content and directions teachers suggest for future content development
· features that teachers and students find most useful and least useful
· teacher and student perceptions of the overall educational utility of the system
· professional development requirements for preparing teachers to use the system
· implementation issues that should be addressed before making the system available to a wider range of users.
Originally, the intention was to include a junior-intermediate level school in the study as well as the high school, however technical problems delayed implementation of the field trial, which prevented us from including the additional school. Three teachers at the high school expressed interest in using the video-on-demand system and, hence, were selected to be part of the pilot study. Again, because of delays and scheduling problems, only one teacher, “Tom,” actually used the system with one of his grade 9 classes. Tom described the grade 9 academic stream science class that we observed as a “decent class…having one or two students with chronic discipline problems.” The second teacher, “Bob,” provided Tom with extensive technical support and assistance in testing and stabilizing the delivery system and putting lesson content online. Both Tom and Bob had considerable experience in integrating computers and the Web into their teaching. Bob is an experienced HTML and Java programmer and he described himself as an “intermediate level” programmer in Visual Basic. Tom admitted that, beyond knowing the basic concepts behind HTML, he could not program in any of these languages. The third teacher dropped out of the study because the time demands of the project were too great.
The lab where the trial took place had seventeen 350 Mhz Pentium II Windows ’95 computers with 64 Mb of RAM and two gigabyte hard drives. They had full multimedia capability, including 64-bit sound and 32 speed CD ROMs. All machines were connected to the school’s 100 Mbit/sec Novell network, a high capacity Sun Solaris sever, and the Internet. The school’s network could receive external data at a rate of 24Mbits/sec via the school board’s Wide Area Network. TVO digitized a series of educational science programs that were selected by the teachers. These files were placed on a video server at the school board office. Client software at the school allowed teachers to create clips of streamed video. These clips were then integrated into Web pages that the teachers developed to illustrate complex physical processes related to the topic on which they chose to focus (electricity). When the teachers started using the client software they found many bugs and security flaws which they reported back to the software developers. They found that the streaming was too slow and unstable, and in an attempt to improve performance, the video server was eventually moved to the school before the system was used with students.
Our research team paid three visits to the school’s computer lab over during December 1999 to observe Tom’s use of the system with his students. Throughout the observations detailed notes were taken focusing on student and teacher behaviour, student reactions to the video content the speed of delivery of the streamed video, and technical and operational issues. At the end of each session, Tom was interviewed to obtain his assessment of the lesson and copies of the lesson worksheets were retrieved for study. Prior to the formal classroom observation, we made five visits to the school to discuss and document development and implementation issues, chiefly with Tom and Bob. A follow-up interview was also conducted with these two teachers in late February 2000. This interview, as well as the last meeting before the classroom observation, was tape recorded and transcribed. In addition, an analysis was undertaken of the digital video database structure and client user interface.
Our field notes and interview transcripts were analyzed using standard qualitative research procedures. This involved a thorough reading of the field notes and transcripts to look for emergent patterns or themes in the data. Once these themes were tentatively established, the data were re-examined to test the validity of the themes. Summaries of themes were then written, which, in turn, guided report writing.
As a proof of concept, the field trial was considered a partial success in spite of the technical difficulties encountered. When the video streaming system worked properly, it gave students dynamic illustrations of scientific processes and physical phenomena in a manner that more closely met the individual learning needs of students than would be obtainable by playing a video tape to the class as a whole. Students could view the video when they were ready to do so, and by pausing and replaying could match the rate of presentation of the concepts and processes to be learned with their own rate of assimilation.
With an improved interface for developing clips and better database indexing, the TVO video library could offer teachers a means for easily selecting for student viewing short segments of video which focus narrowly on those specific ideas or elements of procedural knowledge which can be more easily or completely mastered by means of interactive multimedia presentation. Evidence from the field trial suggests that when the delivery system works, and video viewing is tightly integrated with guiding questions and tasks designed to help students master salient knowledge, students can work in a focused manner with the presented video so that it becomes a true stepping stone to learning.
The higher degree of user control streamed video playback makes available to students makes it possible for teachers to develop more cognitively engaging assignments and tasks related to the material. They can present activities and challenges that encourage the development of critical problem-solving and other higher order reasoning skills, since students will have the freedom to move back and forth between cognitive processes such as problem articulation and reformulation and repeated engagement with new sources of knowledge in a manner not possible with whole-class video presentation. Supported by appropriate pedagogy, this tool could thus serve as a considerable resource in helping teachers attain one of the major aims of education. Unfortuantely, the teachers in the project were so focussed over most of the semester on getting everything to work properly that they had no real time to reflect on how the video could be used to faiclitate more contructivist, student directed learning. Very little time went into the planning of the tial’s curriculum, and the worksheet activities which students engaged in differed little from those found in traditional classrooms—video simply provided a more dynamic and multi-media souce of imformation than texts or reference works.
Proof of concept is one thing, and having a viable and scalable technology and implementation package another. Clearly, the TVO video-on-demand environment is far from ready for prime time. Major changes in many aspects of the software environment, delivery system, and underlying video database will be needed if the service is to be of value to the average teacher. With the modifications that we suggested for the TVO system, teachers with a modest computer background should be able to master the basics of embedding video links in Web pages after taking a well-designed one-day professional development workshop. We recommended that this be a mandatory requirement for getting access to the system to ensure a minimum level of user competency; it would help avoid a scenario in which unprepared users become frustrated with the system. Applicants for the workshop should be ‘pre-cleared’ in terms of the adequacies of the facilities and bandwidth available to them in their school for using the service before being accepted, for a similar reason—school networks relying on dialup modems and/or five year old computers will not be able to access video streaming adequately and such a check will avoid considerable frustration. Advanced workshops for teachers wanting to have their students use video streaming in their own projects should also be offered. We recommended that alternate modes of distribution also be considered, since many schools will not have the technology in place to access streamed delivery. CDs containing short video segments for different grades and subjects, together with some suggestions for classroom use, could appreciably increase the number of teachers able to make use of the rich video resources available from TVO.
The notion of teachers as knowledge builders derives from the work of Marlene Scardamalia and Carl Bereiter (e.g., Scardamalia & Bereiter, 1993; Bereiter, 2000). It is based on an attempt to create a culture of continuous professional improvement among teachers, whether they are physically located in the same school or connected through virtual communities. In contrast to traditional forms of professional development where teachers are exposed to “one-shot” workshops or short courses, knowledge building encourages teachers to partner with their colleagues and become co-learners to move their collective understanding forward through reflective discourse. Knowledge building is seen as more beneficial than traditional professional development because teachers take control over their professional development and learn what they need to know, when they need to know it, and how they will learn it. Nor was this discourse a "one-way street": for her part, the teacher found this process of reflection of value in formulating new pedagogical and curricular strategies for knowledge building in her class.
In all three evaluation projects we witnessed teachers collaborating in knowledge building activities, either with their peers or with student teachers. The Knowledge Forum project promoted intense reflective collaboration between a master teacher and her teacher candidate which led to the candidate’s reformulation of many of her core beliefs about the teaching process, and supported her in the process of acquiring the skills and strategies needed to actualize these beliefs in her practice. The more successful GrassRoots projects had many of the hallmarks of teacher knowledge building, including changed role of teachers, more powerful uses of technology, communities of practice, and seeking help from those with expertise outside of the school (see Doubler, Laferriere, Lamon, & Rose, 2000), although there was no formal focus on the process per se. At Lake Elementary, for example, one teacher was more knowledgeable about Web site construction than other teachers. He helped his colleagues these skills as needed and they discussed how they could integrate the website into their teaching units. As mentioned earlier the student teachers formed self-help networks to divide responsibilities, learn specific skills, accomplish specific tasks, and share their findings with each other.
Finally, in the TVO project, through almost daily collaboration over several months, the two teachers developed considerable techincal and operational expertise in using the technology and integrating it into their teaching. They jointly explored the potential of the video tools, sharing pedagogical and curricular ideas around its use to some degree, and selecting and integrating video clips for their own and others’ use. But the high demands imposed on them by trying to work with the limited and buggy video delivery software meant that they were left little time for reflective collaboration directed towards new forms of constructivist practice. As a consequence most of their classroom activities were conventional worksheet-based exercises which simply assimilated video use into traditional forms of pedagogy.
In all three evaluation projects the classroom and school contexts clearly played an extremely influential role in project success. With the exception of the knowledge building project, student teachers and classroom teachers in the cases we studied worked under sub-optimal conditions in terms of their preparedness to use the technology for pedagogical purposes, technical support to use the technology itself, and student access to the technology. Under these circumstances, it is remarkably difficult to achieve the intended student outcomes. Indeed, it is a testimony to these teachers’ tenacity that they were able to accomplish as much as they did. For example, in the GrassRoots cases supervising classroom teachers frequently had to leave student teachers to their own devices because they simply did not have the technical nor pedagogical expertise to help them develop Web sites to support their curriculums. This prompted the student teachers to form their own support groups and to seek help from friends outside of school. In the TVO project, the teachers were generally able to handle the technical problems and software glitches. However, they were so immersed in those problems that they had little time to focus on the pedagogical aspects of integrating the technology into their work. As a result, Tom, the teacher who used the streamed video in his class, resorted to developing little more than electronic worksheets. This was done more out of expediency rather than his belief about how the medium could be best used because his teaching approach generally favoured constructivist learning activities for his students. And while preparedness and support levels were much higher for the knowledge building project, a great deal of time had to be spent earning new software in the transition from WebCSILE to Knowledge Forum, time that might have been devoted to more collaboration with students or further project development.
Technical support left much to be desired in all cases except at Lake Elementary in the GrassRoots project and (to some extent) in the knowledge building project. At the other GrassRoots sites the lack of support either prompted student teachers to seek outside assistance as mentioned above or discouraged the classroom teachers and caused a few of them to abandon their projects. Teachers in the TVO project were clearly frustrated with the lack of support that they were getting from the software developer and, moreover, that they were testing with students what amounted to a “beta” or prototype software product. Had it not been for the great attention being paid to the TVO trial including senior government officials and foreign teams visiting the project, the teachers in all likelihood would have abandoned the project early on. In the knowledge building project, the teacher was able to deal readily with client-side software problems, but the off-site location of the Knowledge Forum server required her to leave the room to call for a re-boot of the server software when bugs emerged on the server side—a fairly regular occurrence which temporarily impeded student work.
Student access to technology was somewhat problematic in the Knowledge Forum project, as there were only four computers available in the teacher’s class. Sarah was able to work around this by having some students access the computers when they were scheduled in other classes. This was workable only because of the presence of the teacher candidate, who could supervise these students. In the other two cases, access was not as great a factor. In the GrassRoots project teachers were able to organize students into teams to develop different aspects of their Web sites, or student teachers themselves formed development teams; in the TVO project the teachers doing the development has no difficulty gaining access to computers, however for the trial students had to share computers in the lab and no more than four or five simultaneous logins to the video server were possible because of its poor robustness.
Our studies relied on qualitative methodologies as our goal was to understand the processes related to professional development through Web-based projects, and the factors affecting appropriation of the requisite knowledge and skills by teachers and teacher candidates. For us, illuminating the perspectives of practitioners is critical to any analysis, since actors’ beliefs, perceptions, attitudes and knowledge play a crucial role in mediating actions and outcomes. Sustained field observation is also essential, as it remains the only way to document the impact of professional development experiences on everyday practice in the classroom.
The comprehensive delineation of a project’s various contexts is a critical step in the evaluation of these programs, as context helps to shape both a program’s implementation and its impact. Technology-augmented innovations in education, because of their greater complexity arising from the use of technology, are typically subject to greater variability in their implementation than other new programs. We (and others) have found large differences both across projects and across sites within projects in teacher and teacher candidate comfort levels, experience, training, and support in using technology, and in applying constructivist pedagogy in their practice., all of which have a direct bearing on outcomes. The key factors that must be catalogued and analyzed include participants’ professional development activities relevant to the innovation, the resources and support available to them, relevant school and external policies, teacher and school characteristics, and the nature and extent of the implementation itself.
As pointed out earlier we used several different data collection sources in gathering this data, including classroom observations, field notes, interviews, and documents. The rationale behind using multiple overlapping data sources is to allow the strengths of one data collection method to compensate for the weaknesses of another. In general, this technique strengthens the credibility of a qualitative study. We tended to rely mostly on interviews in these studies; classroom observations were secondary, followed by the other data sources. We find that we frequently have to choose between allocating time and resources in an evaluation project to one or the other of these two sources, because rarely is there sufficient time to assign equal amounts to both. The result is often a trade off in terms of inferential and face validity. As Yvonna Lincoln (2000) points out, “observational methods have high face validity, but low inferential validity (e.g., we saw what they did, but we do not know the meaning or purpose of the activity); interviews, on the other hand, provide the reverse: high inferential validity, but low face validity (e.g., we know what the ritual "means" to participants, and how they make sense of it, but we have not yet seen the ritual performed).” In our three studies, the compromise that resulted in higher inferential validity was preferable as it better addressed the needs of the intended audiences for our reports.
A major area that will need attention from those of us studying teacher preparation over the next several years is outcome assessment. As increasing numbers of teachers at various points in their careers receive training for technology use in teaching, more sustained, longitudinal research will be necessary to answer several important questions: What preservice and inservice programs and experiences best promote constructivist practice with technology? How can “one-shot” R&D projects that foster teacher and teacher candidate knowledge building around teaching with technology be extended and replicated beyond the “hot house” context of support provided by high-powered R&D teams? Why do certain programs die off? What happens when teacher candidates educated and experienced in constructivist practice, and the use of technology to support it, find themselves in a more conventional school context after graduation?
Bereiter, C. (1999). Education and Mind for the Knowledge Age. Available at:
http://csile.o ise.utoron to.ca/edmind/edmind.html
Bonk, C. J., & King, K. S. (Eds.). (1998). Electronic collaborators: Learner-centered technologies for literacy, apprenticeship, and discourse. Mahwah, NJ: Erlbaum.
Doubler, S., Laferriere, T., Lamon, M. & Rose., R. (2000). The Next Generation of Online Teacher Learning: A White Paper for the Innovative Learning and Technology Center. http://www.cilt.org/seedgrant/ online_Learning.html
Heinecke, W.F., Blasi, L., Milman, N., & Washington, L. (1999, July). New directions in the evaluation of the effectiveness of educational technology. Paper presented at The Secretary's Conference on Educational Technology, Washington, DC.
Lamon, M., Scardamalia, M., Shaw, P., & Fullan, M. (2001, January). School Improvement through Learning Communities: Virtual and Real. Toronto, ON: Final Report for the Ontario Ministry of Education and Training.
Lincoln, Y.S. (2000, June). Data matrices vs. case studies: Strengths, weaknesses and benefits compared. Unpublished report prepared for the Second Information Technology in Education Study (SITES-M2) project.
Owston, R. D. (1997). The World Wide Web: A technology to enhance teaching and learning? Educational Researcher, 26(2), 27-33.
Owston, R. D. (2000). Evaluating Web-based learning environments: Strategies and insights. CyberPsychology and Behavior, 3(1), 79-87.
Owston, R. D. (2001, February). Final Report: Case Studies of GrassRoots Implementations (Centre for the Study of Computers in Education Technical Report No. 2001-1). Toronto, ON: York University, Faculty of Education. Available http://www.edu.yorku.ca/csce
Neale, D. C., & Carroll, J. C. (1999, December). Multi-faceted evaluation for complex, distributed activities. Paper presented at the Computer Supported Collaborative Learning conference, Stanford University, Palo Alto, CA.
Scardamalia, M., Bereiter, C. & Lamon, M. (1994). Bringing the classroom into World III. In K. McGilly (Ed.). Classroom Lessons: Integrating cognitive theory and classroom practice. Cambridge, MA: MIT Press. [See also http://www.knowledgeforum.com for details on the software itself.]
SchoolNet (2001). Website found at http://www.schoolnet.ca/grassroots/e/index.asp
Sherry, L., Lawyer-Brook, D., & Black, L. (1997). Evaluation of the Boulder Valley Internet Project: A Theory-Based Approach to Evaluation Design. Journal of Interactive Learning Research, 8 (2), 199-234.
Wideman, H. H., & Owston, R. D. (2000, March). An evaluation of TVOntario’s video-on-demand pilot project (Centre for the Study of Computers in Education Technical Report No. 2000-1). Toronto, ON: York University, Faculty of Education.
Windschitl, M. (1998). The WWW and classroom research: What path should it take? Educational Researcher, 27(1), 28-33.