Part 1. Critique of a research study
The current paper presents and critiques a study conducted by Gwo-Jen, Ting-Chia et al., which was published in 2016 in the Computers & Education section of ELSEVIER journal with the title ‘Interaction of problem-based gaming and learning anxiety in language students’ English listening performance and progressive behavioural patterns’. The aim of the study was to investigate students’ learning behaviours with varying levels of English anxiety in a game-based environment via a quasi-experiment. Furthermore, the credibility of the study will be critiqued after the presentation of the research design of the study and the epistemological assumptions that underpin it.
Based on the given general description of the research goals, though it is hard to convincingly demonstrate, the researchers aimed to establish a causal relationship between students’ anxiety and learning performance in a problem-based English listening game via a quasi-experiment. They tested ‘theoretical postulates’ using empirical data, indicating the positivist assumptions that underpin their research design and following a deductive logic (Bhattacherjee, 2012). In particular, a quasi-experiment was conducted with the goal to address the following research questions: 1. Does the problem-based English listening game benefit the students’ learning achievement more than the conventional technology-enhanced instruction? 2. Can the problem-based English listening game enhance the students’ learning motivation in comparison with the conventional technology-enhanced instruction? 3. Can the problem-based English listening game reduce the students’ English anxiety in comparison with the conventional technology-enhanced instruction? 4. Are there differences between the learning behaviours of the students with lower and higher levels of English anxiety learning when playing the problem-based English listening game? 5. Are there differences between the learning achievements of the students with lower and higher levels of English anxiety learning when playing the problem-based English listening game?
Following a quasi-experiment design where the assignment is by self-selection or administrator judgment (Cook, 2015), the researchers, in this research design, chose one control group (38 students) and one experimental group (39 students) by inviting students from 2 classes in the same English training program taught by the same teacher. The problem-based English listening game was introduced to the experimental group; the control group learnt the same content by their usual practice using current facilities like slides, music player, etc. To answer each research question different research instrument were used, such as a pre and post-test were to measure listening achievement after the intervention, questionnaires were delivered before and after the intervention to measure motivation/ anxiety of students in different English levels and a progressive analysis of the quasi-experiment result was to understand the relation of learning behaviours with anxiety.
Drawing from the research methodology and its positivist underpinnings, the quality of the study is examined in relation to ‘the credibility indicators’ suggested by O’Leary (2014), which are objectivity, validity, reliability, generalizability and reproducibility. To begin with, generalizability is a significant characteristic of the positivist experimental design, which aims to draw objective results (O’ Leary, 2014). Considering the fact that the sample of the discussed study was neither randomly selected nor randomly assigned, its findings cannot be generalized to the population (O’ Leary, 2014). As a result, its external validity is questioned.
In terms of validity, though the non-random sampling strategy of the quasi-experiment caused concerns to the internal validity of the paper (1), the statistical conclusion validity of these 5 research questions was highly maintained when several statistical tests were applied and all showed acceptable indicators of covariance (Lipsey, 2000). The researchers also managed to control the independent variables by choosing samples from the most similar background (taught by the same teacher, on the same content, in the same English course). This non-random selection is to create a comparison where the treatment group matching with the comparison group on observable characteristics such as age, education, levels (White& Sabarwal, 2014, p.2). O’Leary (2014) explains that when one has validity, they can ensure that they have eliminated any possible causal relationships other than the ones being researched, thus measuring what they intended.
Furthermore, the discussed study is characterized by applying standardized and tested instruments to guarantee the reliability and validity of their quantitative research (Morse, 2002). For example, General English Proficiency Test (GEPT), the standard Tawain English test was used to measure students’ performance in pre-test and post-test and the Pearson’s correlation between the two tests was 0.65 (p < 0.001) showing that the difficulty of the two tests was the same and valid in helping researchers assess and compare the score between the two groups after the intervention to determine their impact on the students’ performance (Lewin, 2005). Correspondingly, the researchers used questionnaire to determine the impact of the game in enhancing students’ learning motivation and reducing their anxiety. The questionnaires developed by Pintrich and De Groot was modified to investigate students’ learning motivation and that by Horwitz for anxiety. The Cronbach’s a value of the questionnaire in this study was 0.94, showing acceptable reliability in internal consistency (Parsian & Dunning, 2009). Additionally, their coding scheme in the quasi-experiment was created by 3 experts in game-based learning reviewing related literature comprehensively to propose trigger points of students’ behaviours, then pilot testing the scheme and concluding after testing. These three stages are to ensure the secondary data can appropriately be interpreted to answer the research questions 4 and 5 (Kimberlin & Winterstein, 2008)
Correspondingly, the objectivity of the research was secured since the researchers remained distance from the study (Geoff & Judy, 2004). The methods and data were clearly described that other researchers can later assess the paper’s validity and reliability and the paradigmatic ways to achieve objectivity by using various measurement and quantification were traceable. What were measured and quantified were also verified relative to a standard (2). Hence, the reproducibility of the research was not ensured by the fact that the research result cannot be generalized though the methodology can be repeated in another study (O’Leary, 2014).
Apart from the weaknesses that have been previously stated, the study has some further limitations that were recognized by the researchers. The most significant limitation is the scale of the sample size which affects the generalizability of the research. In addition, the design for ensuring background and length of the research will also need to be reconsidered in a further research. For example, the design that allowed both group, before the experiment, had four weeks of classes about basic learning knowledge of English listening skills may actually ‘triggered some other factors which influenced their learning’ as said by the researchers. Though not being mentioned in the papers, the possible threats to the validity by this practice do present. Such as the preparation time (4 weeks) was much longer than the intervention period (one week) as the result, the learning habits formed in these 4 weeks may affect the learning behaviours in the later time regardless of the intervention or not. Another risk is that students did not learn at their own pace in the preparation time and the researchers did not describe if students had had any experience with such game-based learning so students in experimental group may suffer from different stress than the other students.
Another significant limitation was that the anxiety of students should be explored further with other research methods such as interviews or think-aloud method. Despite the strength of quantitative research such as the study can be replicated in different areas or over time with the production of comparable findings (3) and providing data that is descriptive allowing us to capture a snapshot of a user population (4) only quantitative measurement may have not reflected all the individuality of participants’ anxiety.
A final factor that should be regarded in the critique of the study is ethics. The researchers claimed to follow the ethics criteria suggested by an authorized ethics committee in Taiwan. That is, participation of the children was approved by their parents, and the participants were protected by hiding their personal information during the research process; moreover, they knew that their participation was voluntary and that they could withdraw from the study at any time. This claim to follow an ethical standard will allow the research to be accountable to the public, obliged to moral and social values and to promote the values that are essential to collaborative work, such as trust, accountability, mutual respect, and fairness (5). However, there are still some ethical issues that the researchers should pay further attention such as what would happen to the participants once the trial intervention ended (6). The students in the experimental group who was concluded to show stronger motivation when study with the game (research question number 2) may have to face with stress when they have to go back to the less favourable way of learning if game-based learning is not available anymore after the research. The familiarity and the effectiveness of the game to the teacher may also affect students’ performance. If after the intervention, the teacher finds game base learning more time consuming and put more pressure on his shoulder, he will be likely not to choose to use it in classroom later.
In conclusion, the current paper attempted to critique the study conducted by Gwo-Jen, Ting-Chia et al. which followed a quasi- experimental quantitative approach. Having described the research design and the methods used to address the research questions; its credibility was examined based on positivist credibility indicators. Consequently, it is argued that the study can be considered credible. Although there are some problematic issues, the researchers’ acknowledgments of these limitations raise the credibility of the study. Furthermore, their findings are worth to be considered for the reference of a larger scale research of the same field.
Part 2. Critical analysis of relationship between epistemology, knowledge and methodology used in educational research.
According to Guba (1990), research paradigms can be characterized through their ontology (What is reality?), epistemology (How do you know something?) and methodology (How do you go about finding it out?). To find knowledge, researchers make implicit epistemic statements. Some familiar names of epistemic stances are: pragmatism, positivism, interpretive, etc. Each of these makes claims as to what kind of knowledge can be created through research, and how it is gathered and presented (Tennis, 2008). Therefore, it is critical for researchers to have the ability to identify the relationship between the epistemological foundation of research and the methods employed in conducting it (Dawn, 2007). In the below discussion, the writer will discuss the relation among epistemology, knowledge and methodology of two epistemic stances which are positivism underpinning the research paper “Interaction of problem-based gaming and learning anxiety in language students’ English listening performance and progressive behavioural patterns” by Hwang et al (2017) and social constructionism underpinning “The ‘digital native’ in context: tensions associated with importing Web 2.0 practices into the school setting” by Crook, C.K., (2012).
Looking at the first paper by Hwang et al (2017), the relationship between the epistemic stance and methodologies can be traced when researchers attempted to identify causes which influence outcomes (Creswell, 2009). The researchers stated to test their hypothesis of the causal relationship between learning anxiety and students’ performance by a quasi- experiment following deductive logic. Furthermore, knowledge in this paper was drawn by analysing quantitative data, following the positivists’ view of reality which is universal, objective, and quantifiable (Dawn, 2007) . The underpinning epistemic stance of positivist also stressed that reality can be measured hence the focus is on reliable and valid tools to obtain it (7). Therefore, the researchers tried to enhance the reliability and validity of the research by using different measurement such as pre/ post-test, questionnaires, quasi-experiment to find answers for their five research questions. Then in the analysis, they applied different rulers such as ANCOVA, t-test, Crobach alpha, Peason’s correlation to calculate the correlation of variables, then interpreted the data to answer research questions once the calculation showed acceptable standard.
In term of knowledge produced and left out under the positivist’s stance, in the systematically controlled investigation, a quasi-experiment, the researchers succeeded in describe knowledge from data, hence, failed to recognise the ability of the human person to interpret and make sense of his or her world (Dawn, 2007). Applying only quantitative measurements, the paper’s validity is, therefore, challenged. The in-depth knowledge for students’ behaviours was not considered. The relationship between students behaviours with anxiety was tested but the fact that same behaviours may derive from different reasons. In this case, students’ low performance should not only blamed on levels of anxiety. Other variables which cannot be controlled and counted in the quasi-experiment can be students’ experiences, social, emotional and economic status. Correspondingly, Wilson (2010) described the role of the researchers in positivist approach as being independent from their research and purely objective, which means they maintain minimal interaction with the research participants when carrying out research. In this discussed paper, the researchers stay distance from the participants with no direct interaction with participants. The three researchers just analysed quantitative data from instruments and behaviours recorded between the interaction of students with the game.
In contrast to the quantitative methods of positivism, in the study by Crook (2012), the researchers used qualitative data to generate knowledge and the epistemic stance underpinning the paper was social constructionism. Social constructionism see reality as “being subjective and differs from person to person” (Guba & Lincoln, 1994, p. 110). In his study, Crook conducted structured conversations with 53 mixed gender focus groups to make it possible to mix a quantitative and qualitative approach to the data. In his research design, individual viewpoints were valid and revealed in interviews and not affected by others as in the manner of constructionism. Each individual reality is true for the person because of their unique experiences but the reality is independent of that person because of his inability to change it (Gergen, 1999). “This interpretive methodology design is directed at understanding phenomenon from an individual’s perspective, investigating interaction among individuals as well as the historical and cultural contexts which people inhabit”(Creswell, 2009, p. 8).
His constructivist stance also left a remark on the way he produced knowledge following inductive logics and his subjectivism on the real world problem – the usage of web 2.0 in school practices (Grix, 2004). Though there was no hypothesis tested, his subjectivism imposed on the research via structured questions used in his conversations with the focus groups. His knowledge goal was to categorize knowledge of the users of web 2.0 in school practice to suggest a sophisticated awareness among young people of institutional, social and moral tensions associated with modern web-based services. This goal falls under the constructionism epistemology “Knowledge and meaningful reality are constructed in and out of interaction between humans and their world and are developed and transmitted in a social context” (Crotty, 1998, p. 42). His findings were being generated from the data, not preceding it (Cohen et al., 2007, p. 22). Although the knowledge produced was “sensitive to individual meanings that can become buried within broader generalizations” (Samdahl, 1999, p. 119) due to the fact that “it is usually fragmented and not unified into a coherent body. Generalizations which are deemed useful to policy makers are often absent because its research usually produces highly contextualized qualitative data, and interpretations of this data involve subjective individual constructions” (James, 2012). Regard to the role of a researcher, he did not only act as an observer, he involved in the research by making structured conversations with participants, then interpret knowledge through his or her understanding of what he thinks the world is like (Dawn, 2007)
From above comparison of the relationship of epistemology, methodology and knowledge between the two paradigms of positivism and constructionism, it is obvious that researchers with different epistemic stance will play different role in his research. His research design and findings, therefore, will also be driven accordingly. However, there are both advantages and shortcomings each theoretical framework, choosing a social construction, researchers will need to depend on his beliefs and values and their strict attachment to methods and measures (8)
Part 3: Reflection, drawing on your experience across the module how your underlying epistemological beliefs, research approaches, research designs and ethical considerations have been shaped and developed, with application to your own research question.
Initially, I have interest in two area of interest with questions: 1. How to apply the world recognized English teaching apps ‘abcmouse.com’ to teach English to students of Hoa Thuy Tien kindergarten? and 2. Social barriers that online learning programs may encounter in Vietnam market
These questions were raised because online learning environment is becoming more favorited in my country but there are cases reported failing in expanding the market in Vietnam though succeeded in other markets. Relating to the knowledge that I learnt from week 3 of the course Researching digital learning, I intended to conduct an action research for the question number 1 and a digital ethnography research for question number 2. Rationales for choosing these two research methods will be discussed in the following parts.
For the first research question, action research method is chosen due to the need of my institute in enhancing the quality of teaching and learning. As Mills (2011) defined that ‘Action research is any systematic enquiry, by a teacher, administrator or practitioner with a vested interest in the pedagogical process or the learning environment for the purpose of gathering information regarding how their particular school operates, how they teach and how their students’ learn.’ The intended action research is to find out the ways that can ensure the efficiency of ABCmouse.com app in our school.
Though more comprehensive literature review will need to be done in the research to consider factors effecting learning efficiency, some can be named as culture, motivation, learning perception, etc. The intended research is to test found theories by observation within the context of applying ABCmouse.com in Hoa Thuy Tien kindergarten. The plan, therefore, follows the principles of Social constructivism which emphasizes the importance of culture and context in understanding what occurs in society and constructing knowledge based on this understanding (Derry, 1999; McMahon, 1997). Assumptions of Social constructivism is based on specific assumptions about reality, knowledge, and learning in which, reality is constructed through human activity (Kukla, 2000); knowledge is also a human product, and is socially and culturally constructed (Ernest, 1999; Gredler, 1997; Prat & Floden, 1994); learning is social process (McMahon, 1997). On the way of the research, our school will be willing to adjust and change our teaching and managing practices to make the most of the app, the found knowledge won’t necessarily be seen as the end product (O’ Leary, 2014).
Regarding to the second research question, digital ethnography method was chosen. The research is intended to be done because the recent report on statistics of internet users in Vietnam (2013) revealed that as of November 2012, the number of internet users in Vietnam was over 31 million users, nearly half of them between the ages of 15 to 24 hence only 21% of them using educational apps for their learning purposes. English learning apps, therefore, haven’t received enthusiasm yet. The discussion of app recommendation for Vietnamese can be found from online websites, forums and reviews on Apple store, Amazon or Android store. These reviews can be analyzed to make conclusion about preferences of Vietnamese in choosing educational apps. Digital ethnography was chosen because it is a research methodology studies the rich and complex nature of online social interactions, analysing and capturing data from dialogue and of the transfer of knowledge that occurs within them (Sarah, Heather, et al, 2015)
Due to its design, the research environment is considered naturalistic environment as the researcher will observe the nature of social (online) interactions. She may re-direct dialogue (i.e. introduce a topic) but will not influence the consequent evolution of that conversation. Furthermore, interpretivism is the epistemological underpinning of digital ethnography because it is very much based on qualitative observation (O’ Leary, 2014). It could be said to fall under the interpretivist paradigm because the planned methodology will be interviewing and observation and analysis of existing texts (Hine, 2009)
On its way of the course, I also learnt about ethical issues that a researcher needs to consider when doing the research. The research questions that I intend to work on involve children and communities, therefore, the autonomy of the participants need to be ensured(O’ Leary, 2014).
Knowledge generated from week 4 to week 8 also enabled me to foresee the strengths as well as weakness of qualitative and quantitative methods. Understanding more of different paradigms and epistemologies underpinning them, I tend to move from constructivism and interpretivism paradigm to pragmatism paradigm. The change in the paradigm also makes the change in epistemology, from measuring or interpreting knowledge to ‘the best method is the one that solves problem’ (Johnson, 2004). Afterwards, I chose to do mixed methods as the mean to collect, analyze data and make conclusion on finding as to guarantee the credibility of the proposed research.