Essay: Effectiveness of physics teaching module (esp. uncertainty in measurements)

I. INTRODUCTION
Physics, being inherently an experimental science, involves direct observation and physical experimentation. The undergraduate laboratory is thus an essential part of any physics course, developing practical, computational and experimental skills, and certain habits in novice physicists, training them to ‘think like a physicist’ over the course of their degree. The University of Durham consequently offers a series of laboratory-based modules, starting with the introductory module Discovery skills in Physics (PHYS1101) for Physics and Natural Sciences undergraduates.
I.I COURSE DETAILS AND LOGISTICS
The module, PHYS1101, is structured as a series of eighteen 3-hour laboratory sessions offering a range of experiments designed to introduce the students to the fundamental ideas and practices of experimental physics(ref!!! taken from durham uni website), along with nine 50 minute lectures on error analysis to complement them (the latter being based on the textbook ‘Measurements and their Uncertainties’ by Ifan Hughes and Thomas Hase).
The laboratory aspect of the course is assessed by a written experimental report submitted towards the end of the academic year as well as a formal assessment of laboratory practice, together constituting 85% of the total module mark. The course also includes computing workshops (nine 1-hour sessions), the assessment of which constitutes 20% of the module mark.
I.II MOTIVATION FOR THE STUDY
“The analysis of uncertainties…is a vital part of any scientific experiment…but I have found that it is often the most abused and neglected part.” (Taylor, 1997, p. xv)
There is an increasing awareness of the importance of the laboratory experience. The ability to analyse and make meaningful conclusions from real data is widely considered to be the most important skill that could be taught in an introductory physics laboratory. (ref!!! Development of the Concise Data Processing Assessment James Day and Doug 10.1103/PhysRevSTPER.7.010114) According to the Durham University’s Physics department’s official web page, PHYS1101 aims to ‘provide a structured introduction to laboratory skills development, with particular emphasis on measurement uncertainty and written and oral communication skills’ (ref!!! taken from faculty handbook). Even though teaching uncertainty analysis is one of the main goals of many science laboratories, there remains a major concern about students’ lack of ability and understanding in the area.(ref!!! Rebecca Faith Lippmann)
This study focusses on the population of students enrolled in PHYS1101 at the University of Durham in the academic year 2018/19, evaluating the effectiveness of the module in general and focusing on students’ understanding of uncertainty in measurements in particular. The study also aims to inform of any shortcomings in the delivery of the module as well as probe any potential for improvement.
I.III THEORETICAL BACKGROUND
As stated in the previous section, the ability to interpret measurements and uncertainties is a major goal of undergraduate physics laboratory courses. Uncertainty analysis involves propagating uncertainty, comparing sets of data, choosing how to report data, and other related activities. The definition of measurement uncertainty is provided by the International Organization for Standardization (ISO) as “the parameter associated with the result of a measurement that characterizes the dispersion of the values that could be reasonably attributed to the measurand” (ISO, 1993). In Physics, the concept of uncertainty in measurements is regarded as what Meyer and Land 2003 term a threshold concept.
Threshold concepts are pivotal but challenging concepts in disciplinary understanding. A threshold concept can be considered as akin to a portal, a conceptual gateway, leading to a new and previously inaccessible and perhaps initially troublesome, way of thinking about something. Once through the gate, learners acquire a transformed way of understanding, interpreting or viewing something, without which it is difficult to progress within the curriculum (ref perkins in chap 3 of book). Examples include that of entropy or gravity in physics, the central limit theorem in statistics, opportunity cost in economics and so on. In more detail, for example, the idea of a limit in mathematics is a threshold concept; it constitutes a fundamental basis for understanding the foundations and applications of various other branches within mathematics and is a gateway to mathematical analysis. However, the concept proves to be problematic for many in its application in that it may lead to counterintuitive and puzzling results. For example, the limit as x tends to zero of the function f(x)=[sin(x)]/x is 1; how can something which is getting infinitesimally small as x tends to zero, divided by something else behaving similarly lead to a finite answer?
It is argued that the acquisition of a threshold concept brings with it new and empowering forms of expression that in many instances characterize distinctive ways of disciplinary thinking. As a consequence of comprehending a threshold concept there may be a transformed internal view of subject matter or subject landscape, which may represent how people perceive, apprehend or experience particular phenomena within a particular discipline. However, such transformation may prove troublesome to certain learners for a variety of reasons, not the least of which is that such transformation entails a letting go of earlier, comfortable positions and encountering less familiar and sometimes disconcerting new territory.
In an attempt to characterize threshold concepts, Meyer and land 2003 conducted extensive research and discussions with practitioners in a range of disciplinary areas. It was suggested that threshold concepts are likely to be i) transformative, in that once understood, its potential effect on student learning and behaviour is to occasion a significant shift in the perception of the subject, ii) irreversible, in that once acquired, it is unlikely to be forgotten or unlearned, iii) integrative, in that it exposes the previously hidden interrelatedness of certain ideas and procedures, iv) bounded, in that any conceptual space will have terminal frontiers, bordering with thresholds into new conceptual areas, v) troublesome – Meyer and Land argue that a threshold concept may on its own constitute, or in its application lead to what Perkins (1999) refers to as troublesome knowledge.
Perkins (1999) defines troublesome knowledge as that which appears counter intuitive or alien. Together, Perkins and Mayer and Land identify the following forms of troublesome knowledge; ritual, inert, conceptually difficult, alien knowledge, tacit knowledge and troublesome language. Ritual knowledge has a rather meaningless character, more of a plug and chug kind of thing, that students use in routine without understanding the underlying logic behind it. On the other hand, inert knowledge, suggests Perkins, sits in the mind’s attic, only unpacked when specifically called for, such as passive vocabulary – words that we understand the meaning of but rarely ever use. Conceptually difficult knowledge is a combination of the complexity of certain ideas (momentum as a vector, wave-particle duality etc.) and often reasonable but mistaken expectations of how something works (why don’t heavier objects fall faster) whereas alien knowledge refers to what appears counter-intuitive to students, or which comes from a perspective that conflicts with their own (p.9). Another type of troublesome knowledge, tacit knowledge, represents a difficulty more for the instructors than the learners. It is identified as something that remains mainly implicit (Polanyi 1958), taken for granted by the teachers but that are crucial for the students to understand in order to become ‘a part of’ the disciplinary community. The case of tacit knowledge is similar to that of good news – bad news; on one hand it is very efficient for knowledge to work tacitly in our minds, however on the other, it presents trouble to students as they struggle detecting and tracking their teachers’ tacit presumptions, leaving gaps in their understanding. Troublesome language arises when certain words which the learners are comfortable using in their everyday lives, are given a different meaning than what they are used to, or in some cases, when new words are introduced without defining them first.
‘When troubles come, they come not single spies’ (Hamlet, IV, v. 83-4).
To complicate matters further, in some instances students may grasp concepts but find difficulty in appreciating what is termed as ‘the underlying game’ – an epistemological barrier. Epistemes are ‘manners of justifying, explaining, solving problems, conducting enquiries, and designing and validating various kinds of products or outcomes’. Schwab 1978, Bruner 1973, Perkins and Grotzer, among others, have all emphasized on the importance of students understanding the structure of the disciplines they are studying. Grotzer and Perkins argue that students’ confusions about science concepts reflect not just the concepts per se but also the underlying causal models characteristic of them. The most pervasive problem with epistemes is perhaps the tacit knowledge of instructors. Often taken for granted and thus rarely made explicit (peter davies ch 5 of book), epistemological obstacles present resistant difficulties to students, blocking a transformed perspective and leaving them stuck in a suspended state.
The metaphor of a threshold concept represented by a ‘portal’ invites consideration of how the portal initially comes into view, how it is approached, negotiated and experienced as a transition. The transformation as a result of acquiring a threshold concept may be sudden or it may span a considerable period. Meyer and land 2003, drawing on the writings of Gennep 1960 and Turner 1969, term this transitional state as liminality (from Latin meaning ‘within the threshold’). While thresholds may be seen as leading the learner through a transformational landscape towards a pre-ordained end, liminality on the other hand can be viewed as a less predictable, more ‘fluid’ space, simultaneously transforming and being transformed by the learner as they move through it. (MEYER N LAND 2005!) The state of liminality can be considered as an in-between, suspended state, in which the students occasionally get ‘stuck’ (ellsworth 1997) and oscillate between earlier, less sophisticated understandings and the fuller appreciation of the concept. One outcome of this is that the students’ understanding lacks authenticity and approximates to a kind of ‘mimicry’This mimicry might be a coping strategy on the part of students in a desperate attempt for understanding and clarity. Another outcome is that students can become frustrated, lose confidence and give up that particular course.
It is worth pondering over why some students productively negotiate the liminal space while others find difficulty in doing so. The notion of ‘pre-liminal variation’ is a useful means of understanding how different students approach the liminal stage. Certain ontological factors have been shown to cause pre-liminal variation in students. If threshold concepts have a transformative effect on students, ‘stripping away old identities’ (ref maggi savin-baden in book ch11) then it is very easy for students who do not fit into what Ellsworth 1989 calls the ‘mythical norm’ of ‘young, white, heterosexual, able-bodied, Christian, English-speaking and male’, to find anxiety in approaching a threshold concept which may result in an active refusal of learning. ‘Real learning requires stepping into the unknown, which initiates a rupture in knowing…’ (Schwatzman 2010, p.38); it is easy to imagine the apprehension of students in letting go of what they/the knowledge and ideas hold dear and inviting a shift in their subjectivity and perhaps even identity. Although one might expect that physical sciences in general should not invite such anxiety, however, considerable research has shown that ontological factors affect students’ achievements even in disciplines considered to be objective by nature. Differences in social and cultural backgrounds of students are likely to cause pre-liminal variation in students, affecting how they ‘think’.
Building on the notion of troublesome language, Scwartzenburger and Tall 1978 point out, certain words have a particular significance for students and students continue to rely on the meanings they associate with those words even after formal definitions have been provided to them. What a word might mean to one student may mean something completely different to another, based on the social set-up they have grown up in. For example, in terms of pre-liminal variation, the word ‘limit’ may be thought about as a boundary, a barrier, the end of something etc., something that is real, visible or reachable, depending on what the student is most commonly exposed to in everyday language. However, in mathematics, limits are not reached, they are ‘tended to’. Such instances naturally cause problems often leading to frustration on the part of students in the liminal stage. Similarly, for courses taught in English, it has been identified that students whose first language is not English experience difficulties, often hindering their academic achievement at University (MEYER AND SHAHANAN 1999,2000). Students’ educational background is also another factor that has been shown to cause pre-liminal variation in students. Naturally, not all students take A-levels and even those who do, not in the same subjects. Moreover, varying teaching as well as learning styles, that the students are exposed to before starting university, also contribute to the variation in how students approach a threshold concept and tackle the liminal space. As an example and reflecting on personal experience, students from asia? are not ?? teaching/assessment is focused on theoretical questions rather than numerical problems, whereas at university most of it is numerical problem solving. So a student from that region first has to overcome the obstacle of getting used to the new style of learning and then tackle the actual problem at in question…
The question arises, how might we better help students through difficult conceptual and affective transitions while taking into account the variation in students’ prior experiences and performances? Perkins advocates a constructivist approach. Constructivist techniques have a diagnostic nature. Seasoned teachers know what troubles the students are likely to encounter and draw on active, social and creative learning to address them (ref!!!book). Constructivism can also help tackle forms of troublesome knowledge mentioned earlier. For example, it often helps engage students in qualitative problems rather than purely quantitative ones, encouraging them to focus on the character of the phenomena rather than just mastering computational techniques. This can prove particularly useful for conceptually difficult or ritual knowledge. Similarly, asking students to present and elaborate alternative perspectives may be considered as a constructivist response to alien knowledge and so on. Studies shows that active engagement in learning, as is practiced in constructivist model of learning, leads to better retention, understanding and active use of knowledge in students (eg. Bruer 1993, Gardner 1991, Duffy et al. 1992, Riegeluth 1999, Dori and Belcher 2005, Beichner and Saul 2003).
An assessment of conceptual understanding in students is usually accomplished by means of a diagnostic test. …characteristics of a diagnostic test…
Uncertainties in Measurements was identified as a Threshold Concept in Physics by Wilson et al., 2010 (Wilson, Akerlind, Francis, Kirkup, McKenzie, Pearce & Sharma, 2010). The process of identification assessed all the characteristics of a threshold concept: transformative, integrative, irreversible, bounded and troublesome and concluded that the measurement uncertainty meets all of them. The concept of uncertainties in measurements is central to the understanding of what it means to ‘think’ like a physicist, yet studies have shown that students worldwide have difficulties understanding the concepts of measurement and their associated uncertainty [2–17] (ref!! graphical paper) and leave the introductory courses without having fully grasped the essence of it. Several diagnostic tests have been developed to study students’ understanding of measurement uncertainty in physics lab courses, including Concise Data Processing Assessment (CDPI), Laboratory Data Analysis Instrument (LDAI) and the Physics Measurement Questionnaire. Both CDPA and LDAI are multiple choice assessments. CDPA measures students’ understanding of measurement uncertainty as well as mathematical models of data while LDAI assesses students’ data analysis skills within the context of a single experimental report with emphasis on measurement uncertainties. PMQ was developed by researchers at the University of Cape Town, South Africa and the University of York, United Kingdom. It is a mixed response survey, consisting of multiple choice questions along with open-ended responses justifying students’ choices and highlights the difference between the conceptual understanding of students and their procedural abilities. The PMQ focusses on questions concerning three major areas: data collection, data analysis and data comparison.
This study makes use of PMQ to identify issues in students’ understanding of the topic and evaluates the effectiveness of the Discovery skills module. + about new diagnostic test
II. METHODOLOGY
II.I PRELIMINARY RESEARCH
Similar pedagogic research in this area was conducted in 2017/18 at the department of physics in durham, by James Morgan/ref. PMQ was administered to … students both before and after instruction and it was found that a large majority of students displayed surface understanding of the key concepts. Various key areas where students’ understanding lacked particularly were also identified. This study caries on the work to verify the findings and further the investigation.
II.II ACTION PLAN
In order to successfully achieve the aims of this study, the research was divided into three main components; a) administration of PMQ, to analyse student responses and verify previous findings, b) student and staff discussions, carried out as a combination of interviews and focus groups to gain a better insight into the troublesome concepts and common misconceptions that the student population in question is likely to struggle with and c) development of a new diagnostic test, better suited to the student population arriving at Durham university and motivated by the insights gained by discussions with the members of staff and students. The ultimate aim is to correctly assess whether the current pedagogy is working in terms of the conceptual understanding of students.
II.III PMQ ADMINISTRATION
In order to quantify the effectiveness of existing instructional techniques, PMQ was administered to students in a pre-course/post-course protocol. Students completed the questionnaire on a voluntary basis and as a home exercise. An incentive of £25 amazon vouchers to two randomly selected students was offered in order to encourage student participation. Student responses to the open-ended questions from the PMQ probes were analysed and initially sorted into codes based on the paper by Buffler et al (who code the responses in ascending order of sophistication), independently by the two people conducting this study. The agreement was poor (~72%) and hence the codes were tweaked in order to better represent our student population’s responses. Once again, the responses were coded independently with better but still poor agreement (~80%). The codes were then adjusted one final time, resulting in an agreement of 92%. Each response was coded only once, therefore, for responses that fell into multiple codes, the most sophisticated code was recorded. This presented an issue, as it was noted that a majority of students were coded as giving more sophisticated answers whereas their responses in general represented a highly flawed and confused understanding. The responses were then coded again, this time as having either set-like or point-like reasoning. This approach for coding student responses was used by Allie and Buffler et al. later on in their studies. Set-like reasoning represents student thinking that there is one true or ‘correct’ value for a measurement and any deviation is due to the mistakes of the experimenter, while point-like reasoning is used by students who believe that each measurement is an approximation of the measurand and that multiple readings should be used to build the best approximate.
II.IV FOCUS GROUPS
To gather general feedback about the course, two focus group sessions were conducted, with .. and 6 students attending each, respectively. The students attended the focus groups voluntarily and an incentive of £15 amazon voucher was promised to each attendee. The sessions were semi-structured in order to allow delving deeper into specific student responses. The students were asked about their thoughts on the overall delivery of the module, with particular emphasis on error lectures, as well as to identify areas where they were struggling.
II.V STAFF INTERVIEWS
Interviews with staff members were also conducted in order to gain their views regarding students’ understanding of uncertainty in measurements and the common misconceptions faced. Dr. Aidan Hindmarch, who is currently the course leader for the level 2 laboratory module, was asked whether he …
Prof. I. G. Hughes, co-author of the textbook for Error lectures in PHYS1101 and the person responsible for the re-design of the module in 2004, was also interviewed. When asked about the need for re-design and whether he thinks the module is now working effectively, he said that
II.VI STUDENT INTERVIEWS
Student interviews should always be used when developing educational tests, and the value of the kind of information extracted from such interviews is stressed in the 2001 NRC report [21],
II.VII DESIGNING A NEW DIAGNOSTIC TEST
Having identified the key areas where students struggled and the skills that they should possess by the end of the year, … questions were drafted (see the appendix for the final versions) that directly related to the module’s learning goals. The questions include a combination of MCQ style questions, open-ended questions and graphical questions.