The Smart Science Initiative is a partnership between universities and schools formed to deliver science content and associated mathematics via an intelligent personal tutoring system to Years 9 and 10 students. The UNSW-led partnership brought scientists and science education experts from UNSW, Flinders University and the University of Western Australia together with high school science teachers from across Australia, as well as experts in online participatory learning from the Massachusetts Institute of Technology and Arizona State University and technology partner, Smart Sparrow. Read about the vision for the project here.

The project created four interactive Smart Science modules that are focussed on Australian big science questions. These modules are:  Are We Alone, EΔRTH, Nano Design Lab, and Superbug Challenge. Each module was mapped to the Science Understanding, Science as a Human Endeavour, and Science Inquiry strands of the Australian Curriculum (Science). The technology platform is adaptive and has learning analytics capabilities that enable teachers to analyse their students’ learning and adjust parameters to cater to the students’ needs.

This report provides a summary of the evaluation of the project. Report prepared by Associate Professor Wan Ng, School of Education, University of Technology Sydney and Dr Jennifer Fergusson from the School of Education, University of New South Wales.

Research Methods

Quantitative data was gathered from students using online pre- and post-questionnaires. Teachers also completed an online post-survey. Qualitative data was also gathered from students in focus group interviews pre- and post-project. Classroom observations were carried out and focus group interviews were conducted with teachers at the time of the initial consultative workshops.

Research instruments

A research instrument was developed to investigate the intrinsic motivation of the students in learning science. The factors explored included self-efficacy, valuing science, interest in science, enjoying science and personal goals (i.e. future participation in science). The items for the questionnaire were obtained from other established surveys.

Data analysis

Focus group interviews were recorded, transcribed and analysed. Both parametric and non-parametric tests were conducted to analyse scale data. T-tests and Wilcoxon Signed Rank tests were used to detect any differences between pre and post-test data. T-tests and Mann Whitney U tests were used to detect gender differences and differences between selective and non-selective students. Effect sizes were also calculated. The Analytics tool embedded in the Smart Modules was also employed to analyse student engagement with the modules. 

Participants

A total of 15 schools across New South Wales, South Australia and Western Australia participated in the project.  Four of the schools were single sex schools – one boys’ and three girls’ schools. There were two non-government schools and three selective schools. A total of 61 Year 9 and 10 classes participated in the project. The number of students who filled in the pre-project survey was 1344 but only about half of them (673) completed the post-project survey, resulting in 504 responses that could be matched pre- to post-project, representing about 38% of the students who completed the pre-survey.  The numbers of matched responses were:

102 (95 girls and 7 boys) for the Are We Alone module,
198 (115 girls and 83 boys) for the EARTH module,
97 (47 girls and 50 boys) for the Superbug Challenge and
107 (64 girls and 43 boys) for the Nano Design Lab.

A total of 162 students participated in pre-project interviews and 118 post-project.  Observations were made in 21 classes. There were 51 different teachers who implemented the modules in their classrooms and 20 teacher responses to the post-project survey.

Findings

The majority of students indicated that the modules were easy to use (60%),that they learnt a lot of science from the modules (58%) and were challenged to learn by the modules (57%). Similarly, the majority found that their understanding of scientific processes, how scientists work and the life and work of scientists had increased as a result of doing the modules. However the majority (58%) of the students did not think that learning science using online modules only was better than the way science was normally learnt. A blend of both online and face-to-face learning is still important. Examples of student comments are:

I wish there were more like this on other subjects, not just science, and from what I did, I liked the stimulations very well.

I think that using digital technology as a means to learn science is very beneficial when balanced with practical learning and should be encouraged in classes.

A common item that showed significant decreases in mean values from pre- to post-project for all four modules was the statement ‘Scientists use different types of methods to conduct scientific investigations’. This may be because the modules did not explicitly demonstrate a range of scientific methods within it and the methods presented in the modules did not align with what teachers normally teach as the ‘scientific method’ in the classroom, i.e. suggesting there is only one scientific method.

While there were no significant changes in students’ intentions regarding future science choices between pre- and post-project surveys, a number of students indicated in the post-project interviews that the modules stimulated their interest in learning more about the topic, for example:

 I actually went home and looked at that, the topic.  I actually went in and looked that up because I thought that was really interesting.

Teachers’ post-project responses were largely positive. Teachers found the modules to be visually appealing and motivating and there was general agreement that the students learned a lot about science from the modules, e.g.

It was very visually appealing. They enjoyed the concept and were very positive when it was working effectively. They found it a fun way to learn.

As reflective practitioners, the general view of the teachers was that they would use the modules differently in the future, integrating them into other units of work or using parts of them rather than stand-alone modules (with the exception of gifted and advanced science students).

1. The research identified two major challenges in the project that impacted the motivation of the students and teachers. These challenges were:Technical issues. In the deployment of the modules, in particular the first two modules, there were major technical issues that were related to school technology and some small glitches in the modules. This made it difficult to evaluate the modules separately from the technology issues and could explain the relatively stagnant responses between pre- and post-project surveys.Integration versus stand-alone implementation.

2. The modules promoted independent learning especially for high ability and gifted students in the project but for most mainstream students there is still a need to scaffold learning. There was general agreement amongst teachers with respect to this in the post-project teacher surveys. As part of the sustainability of the project, teachers from several schools have committed themselves to undertake the task of producing worksheets for the modules. These worksheets will be uploaded on the project website and shared with future schools embracing the modules.

Overall, the modules represented a promising addition to the resources available to science teachers and students.