ay in herHistorically, there was a tension between “pure research” and “applied practice”. Sixty
years ago, folklorist Richard Dorson (1950) famously dubbed public folklore as “fakelore”
participating in heated debates over public folklore and its place. It divided theory and practice
and the connection between the two (Baron, 1992, p.309). Dorson directed his criticism towards
Benjamin Botkin, whom he saw as a folklore “popularizer” and who, according to Dorson, stood
in the way of research by simplifying tradition for mass consumption. Botkin answered by
pointing out the weaknesses of theorizing folklore for the sake of theorizing folklore alone,
stating that while “the study of folklore belongs to the folklorist, the lore itself belongs to the
people who make it or enjoy it” (Botkin, 1953, p.199). By the mid-1980s, the division between
“academic” and “public” folklorists prompted Barbara Kirshenblatt-Gimblett to question
whether all of this arguing comes down to “a mistaken dichotomy.” In perpetuating this
dichotomy, academic programs in folklore have “consistently refused to examine their own
essentially and inescapably applied character” (1988, p.141). Public folklorists, for their part,
have contributed to the chasm because while “folklore in the public sector . . . has its own
intellectual tradition,” much of it “remains to be written” (p.149).
Some academics still believe that work of public folklorists does not have much of
theoretical thinking, thus undermining public folklore. It is not the same division as it used to,
and the attitude towards public has changed over the years; and it is important to illustrate how
public folklore is concerned with the same theoretical discussions as academics.
Not enough attention and focus goes to some parts of public folklore. And in this paper I
want to look at them promoting the importance to such aspects of public folklore as evaluation
and research methods. Because public folklore can and should have a strong theoretical
foundation behind it. That the methods we use are crucial and there is a need to discuss
problematic aspects that come with public folklore.
No matter what project you create, whether it is a public folklore project or a business
project, the evaluation is a crucial element often overlooked or underestimated by many.
“Evaluation is a process that takes place before, during and after an activity. It includes looking
at the quality of the content, the delivery process and the impact of the activity or programme on
the audience(s) or participants” (Research Councils UK, 2011, p.2).
Evaluation is crucial in understanding and determining whether a project achieved what it
was set out to, how well it was implemented, what impact the project has had. It s an opportunity
to reflect critically on both the activities and processes, their impact and value. This knowledge
can be used internally by the team to drive improvement and externally to demonstrate
achievements.
Evaluation is more than just information gathering. It is a reflection of what the
activity/project means, the ways of interpretation, the contributors of your data and stakeholders.
And analyzing that information allows your and your team to improve the project, achieve
success and goals, be efficient. Data collection in itself is pointless without thorough analysis and
reflection (Research Councils UK, 2011, p. 3).
Often evaluation is seen as a part that is needed only for submission to the funders, as a
proof of success. Most funding organizations and agencies require evaluation reports to see the
impact of the project. However, evaluation should not be treated as an unwanted nuisance that
you are forced into, but rather it should be used as an ongoing management and learning tool to
improve an organization's effectiveness. Not as a check mark at the end of the project to fill the
requirements, but as a useful method of tracking progress and success in order to gain new
perspectives on the activities that can potentially improve the project. It is imperative to conduct
internal evaluations to get information about the programs to make better decisions about the
implementation of those programs. The crucial part to make internal evaluation an ongoing
process at every level of an organization in all program areas. Another important aspect is to
remember to involve all of the program's participants. Staff and sponsors are going to have
different perspectives on the same issue. While often it is not possible to make everyone happy
due to limitations, it at least opens a conversation that can lead to a compromise.
Evaluations should be done not only internally, but on a larger scale as well. External
evaluations conducted by someone from the outside of the project or organization can provide a
different point of view, findings overlooked by the staff, emphasize aspects the team did not pay
enough attention to. Usually those are done for funding purposes; however, it might be a sound
idea to invite an external evaluation, especially if it is a big project.
There are two types of evaluation: formative and summative. Formative evaluations are
ongoing evaluations that are conducted during program development and implementation. They
serve as a helping hand in directing your project in the best way to achieve the goals and improve
program. Summative evaluations are conducted once the programs are well established,
providing with the information of how well the program is achieving its goals. Within the
categories of formative and summative, there are different types of evaluation and methods to
evaluate (Rossi, et al, 2004).
Common types of formative evaluation (Spaulding, 2008):
1) Implementation evaluation, which monitors the precision of the program delivery.
2) Needs assessment, which identifies who needs the program, how much they need it,
and what are possible ways can be implemented to achieve the need.
3) Structured conceptualization, which assists stakeholders with defining the program, the
target audience, and the possible outcomes.
4) Process evaluation, which investigates the process of delivering and implementing the
program, including the possible alternatives.
Common types of summative evaluation:
1) Goal-based evaluation, which determines if the intended goals of a program were
achieved.
2) Outcome evaluation, which scrutinizes whether the program caused verifiable effects
on specifically defined target outcomes.
3) Cost-effectiveness and cost-benefit analysis, which determine the efficiency of a
project by standardizing outcomes in terms of their dollar costs and values.
4) Impact evaluation, which encompasses the overall effects of a program, whether they
were intended or unintended, by asking what impact the program had on a larger scale (for
example, on a community).
An evaluation can use quantitative or qualitative data, and often will include both types.
Both methods provide important information for evaluation, and both can improve the project as
each has its own limitation. Combined they generally provide the best overview of the project
and allow more flexibility.
Quantitative data answers such questions as “How many?”, “Who was involved?”, “What
were the outcomes?”, and “How much did it cost?” There are numerous ways to collect
quantitative data. Some of them are: questionnaires, surveys, observation, review of databases.
And each method also has different ways of implementation. For example, surveys can be
distributed online or by a person; involve writing your responses in the form or answering
questions on the phone (Holland et al., 2005; Garbarino et al., 2009). While gathering the data, it
is crucial to choose the right method. It will depend on resources, circumstances and the goal of
your evaluation. What type of information you want to get out of evaluation? – that question will
drive the choice of method.
“Quantitative data measure the depth and breadth of an implementation (e.g., the number
of people who participated, the number of people who completed the program). The strengths of
quantitative data for evaluation purposes include their generalizability (if the sample represents
the population), the ease of analysis, and their consistency and precision (if collected reliably)”
(Clinical and Translational Science Awards Consortium & Community Engagement Key
Function Committee Task & Force on the Principles of Community Engagement, 2011, p. 175).
There are obvious limitations to evaluating quantitative data like a lack of context, poor response
rates, and difficulties in measurement. They provide an overview of what happened, but usually
not why.
Qualitative data answer such questions as “What is the value added?”, “Who was
responsible?”, and “When did something happen?’’ Qualitative data can be also collected using
various methods, the most common being interviews and participant observation (Patton, 2002).
One of the strengths of observation is the context that allows for a better results and
understanding of a situation. It helps to explain behaviors and motivations; asks for reasons, not
just for facts. The researcher sees what is actually happening, which can change the results
tremendously as often participants lie or omit the truth in simple questionnaires and surveys
(Ericsson et al , 1993). While it is a useful method, there are also limitations to it. It requires
resources, time, and willingness from people you want to observe.
The context is in general a strength of qualitative data, explaining complex issues and
answering such questions as “why” and “how” behind the “what.” However, it also complicates
analyzing and interpreting data, requiring more substantial human and financial resources
(Patton, 2002).
Choosing an appropriate evaluation might become tricky as there is a big variety of them;
and each has its own advantages and disadvantages. For example, an email survey can gather
data quickly and present it in eligible format. However, it will not include those without
technological access and people are less motivated to respond as opposed to a survey that was
physically given to them. A pen and paper questionnaire needs less technical support and is more
accessible, but can be harder to distribute and collect. Analyzing responses can also take more
time and will require a person to do, when with online surveys there is a software that does
analysis for the researcher (betterevaluation.org). There is also a question of motivation. Often
the team behind a project does not want to spend their time and energy on thorough evaluation,
providing only a superficial evaluation. However, of you really want your project to succeed or
to know if it was successful, evaluation becomes an integral part. It improves not only the project,
but people working on it as well. The lessons learned from well-done evaluation can help in the
future projects and endeavors.
There are some situations where evaluation may not be a good idea: when the program is
unstable, unpredictable, and has no consistent routine; when those involved cannot agree about
what the program is trying to achieve; when a funder or manager refuses to include important
and central issues in the evaluation (Thomson, G. & Hoffman, J., 2003). However, in general
evaluation is important and should not be overlooked.
It is not so simple to just enforce the evaluation. There are many challenges that come
with it and that is possibly the reason why not enough attention paid to it. From the outset it
should be acknowledged that seeking to place a value on the work public folklorist undertake is a
difficult and frequently controversial task. There are challenges that come with defining and
understanding certain terms; cultural and political issues involved. When we talk about “cultural”
or “social” impact, how do we measure them?
There are many different definitions and understandings of value and these can have very
different meanings to different people. Some of them include: educational value; cultural value;
intrinsic value; option value; heritage value; economic value; public value; social value; financial
value; blended value; instrumental value, etc. (Kelly & McNicoll, 2011, p.6). One of the big
discussions is whether ‘value’ can be assessed objectively or it is subjective in its nature?
Whether something can have an ‘intrinsic value’ that is beyond quantification?
The lack of a common terminology across different disciplines creates a problem,
especially since folklore is very much interdisciplinary in itself and often works with other fields.
While working on a project, it often involves people from various disciplines with different
backgrounds. Two things should happen to avoid the potential problems that arise in such
situation: develop a common terminology across disciplines or establish terminology and
concepts before a particular project. The last one is not ideal as it will work out internally, but
will encounter the same problems externally (Kelly & McNicoll, 2011, p.7).
There is no single recognized approach in assessing value. Each method comes with its
own advantages and limitations. Moreover, there is no consensus on defining ‘value’. What is
there is a variety of tools and methods, sometimes very similar, but they are not necessarily
consistent in approach nor posses a solid theoretical foundations. This has been noted by
members of the SROI network in their blog commentary on the subject of ‘Lack of consistency’:
“…The biggest problem that is faced by all of us interested in social value, impact,
returns – whatever language you prefer – is the lack of consistency. And yet I still keep hearing
‘we can’t support one approach’ or ‘organisations should be able to choose methods that are most
appropriate to them’ or ‘small and start up organisations should be able to do something simple’.
A similar message is found in a report about value measurement in the cultural sector:
“The cultural sector faces the conundrum of proving its value in a way that can be
understood by decision-makers …it will not be enough for arts and culture to resort to claiming
to be a unique or special case compared with other government sectors, the cultural sector will
need to use the tools and concepts of economics to fully state their benefits in the prevailing
language of policy appraisal and evaluation….” (O’Brien, 2010, p. 37)
So, what is the verdict? What is the best approach? As a folklorist my first impulse is to
go with qualitative methods and data. However, public folklore exist in a reality of funding
organizations, governments and businesses. Entities that prefer quantitative data. In all my
research I did not find one method or approach that would work for all types of projects. The
best advice is to combine qualitative and quantitative methods. Sally Fort in her book “Public
Engagement: Evaluation Guide” (2011, p.3) provides such guidelines:
“• Events and projects are all unique.
• Be sensitive and use common sense.
• Use the most appropriate methods for your work and audiences, participants or visitors.
• Check which information you need to collect for your event or project.
• Only collect information you will be able to analyse.
• Explain to participants why you are collecting the information.
• And that contributing information is optional for them.
• Use a combination of visual, auditory and kinaesthetic methods.
• Keep your aims and objectives in mind when using or creating evaluation tools.
• Plan to collect both qualitative and quantitative information; use a combination of open
and closed questions to help this process.
• Put yourself in the shoes of your participants, make it easy for them to complete your
evaluation requirements.
• Consider how the learning from your evaluation will be shared with those involved and
more widely”.
Those are broad advises; and when it comes a particular project, you need to assess what
methods are the best in this specific situation. It requires a pubic folklorist to be familiar with a
wide range of approaches and methods, be up to date with current discussions. It also means that
this topic is something that public folklorists need to get into. There should be more articles,
more discussions, more concerns. However, not much done in that regard. I myself did not think
of evaluation or research methods when contemplating about my future as a public folklorist or
potential ideas.
As I started to think over those topics and research them I came to a few realizations I
want to point out. And one of them is the lack of using Social Network Analysis in folklore. I
was introduced to it back in my undergraduate studies, in Social Sciences Research Methods
class. However, I did not encounter it while studying in Folklore program. I did not think of it
before I remembered of the method during my work term where I suggested to use it to illustrate
the impact of the Office of Public Engagement. After remembering it I realized we never talked
about it in Folklore classes; and that, in my opinion, should change.
Social network analysis (SNA) “is the process of investigating social structures through
the use of networks and graph theory” (Otte & Rousseau, 2002, p.441). Social network analysis
is a method that analyzes the connections between individuals or groups or institutions, who are
represented as nodes (actors) and their connections are represented as edges (links). Social
network analysis emphasizes interaction, the relationships between actors, rather than individual
behavior. By examining the network researchers are able to determine how they [networks]
influence the way actors/nodes function.
One social network can include actors of different types that have certain attributes, and
are associated with diverse types of interactions, which in turn have different intensities. Social
actors can be individuals, social groups, organizations, events, cities, countries. Links are
understood not only as communication links between actors, but also as links for the exchange of
various resources and activities, including conflict relations. Thus, network models consider
different social, political, economic structures as stable patterns of interaction between actors. A
special place is occupied by cognitive social networks, which reflect the opinion of each actor
about the relationships of other actors in the network (D'Andrea, 2009).
Dr. Daning Hu (2012, slide 24) explains which differences exist between a social network
analysis and a non-network explanation in their presentation. According to them, Social Network
Analysis:
– refers to the set of actors and the ties among them
– views on characteristics of the social units arising out of structural or relational
processes or focuses on properties of the relational system themselves
– inclusion of concepts and information on relationships among units in a study
– the task is to understand properties of the social (economic or political) structural
environment, and
– how these structural properties influence observed characteristics and associations
among characteristics
– relational ties among actors are primary and attributes of actors are secondary
– each individual has ties to other individuals, each of whom in turn is tied to a few, some,
or many others, and so on.
Social Network Analysis uses several concepts and terms such as: density, centrality,
indegree, outdegree, and sociogram (Laat et al, 2007, pp.87-103).
Density refers to the “connections” between actors/nodes. Density means the number of
connections an actor/node has divided by the total possible connections an actor/node could have
(Laat et al, 2007, pp.87-103).
Centrality refers to the behavior of individual actors/nodes within a network. It measures
the extent to which an actor/node interacts with other actors/nodes in the network. The more an
actor/node connects to others in a network, the greater their centrality in the network. Centrality
has such variables as in-degree and out-degree (Laat et al, 2007, pp.87-103).
In-degree centrality puts a specific actor/node as focus point. Centrality of all other
actors/nodes is based on their relation to the focal point of the "in-degree" actor/node(Laat et al,
2007, pp.87-103).
Out-degree centrality also puts a specific actor/node as focus point, but analyzes the outgoing
interactions of the actor/node. Out-degree centrality measures how many times the focus
point actor/node interacts with others (Laat et al, 2007, pp.87-103).
The researchers are presented with a huge number of computer software developed for
Social Network Analysis and visualization of networks. Some of them are: NetForm, IKNOW,
KrackPlot, Gephi, UCINET, FATCAT, MultiNet, GLAD, SNAPS, NEGOPY, GRADAP, InFlow,
gem3Ddraw, Moviemol, STRUCTURE, daVinci, GraphEd, GraphViz, MatMan, PermNet, etc.
As folklore, whether we talking about public folklore or academia, works a lot with
people and their interactions, Social Network Analysis can be a great tool to use. Aside from
providing analysis, it can also illustrate the impact of the projects and activities by showing what
and whom the projects impacted and to what degree. SNA is particularly useful with dealing with
huge numbers of people. It helps to find the key people, relationships, the ways information
spreads and interactions are made.
There is also a new way to interact with people, create projects, research and evaluate.
Something that will change all the sciences and should be talked about already as in the last
years it became a reality. I want to talk about VR or Virtual Reality.
The term “virtual reality” (VR) refers to an immersive simulation. “In general … the term
virtual reality refers to an immersive, interactive experience based on real-time 3-D graphic
images generated by a computer” (Pimental & Teixeira, 1995, p.15). “Our preferred definition is
an immersive experience in which participants … view stereoscopic or biocular images, listen to
3-D sounds, and are free to explore and interact within a 3-D world” (Pimental & Teixeira, 1995,
p. 91).
It would be a lie to say that virtual reality technologies have not been used in social
sciences. However, it was mostly used in large expensive labs that had enough resources to
develop or buy it. Nowadays VR technology went on another level, becoming more and more
popular among researchers. Thanks to the improvements to the technology with enhanced
realism, affordable costs, and increased possibilities for application, VR may be well on its way
to become a staple research method for social scientists (van Gelder, 2017).
A person using virtual reality equipment is is given an ability to look around or/and move
in the artificial world, interact with features and items in it. Virtual reality is accessed by wearing
a virtual reality headset and using controllers that usually accompany it. VR headsets are goggles
with a screen in front of the eyes. Programs may include audio through speakers in the headset or
external headphones. Some controllers provide an ability to feel the vibrations or rather
sensations.
Virtual Reality is not only used for gaming or scientific experiments, but a way to
communicate creating a virtual presence; you are able to see people in 3D in real time and
interact with them.
According to Kelly (2016), around 230 companies were developing VR-related products
(headsets, software, games, etc.) by the year 2016 with Facebook employing 400 people for VR
development. Other companies (Google, Apple, Amazon, Microsoft, Sony and Samsung)
financed and established departments and groups dedicated to VR technologies as well. On April
5, 2016, HTC shipped its first units of the HTC VIVE SteamVR headset. This was the first major
commercial release of a Virtual Reality technology allowing ordinary people to access a
technology that had been utilized only by few (Prasuethsut, 2016).
Now, VR is not just a technology available to the most high-tech labs, but to ordinary
people around the world. This changes a lot. It opens a way to communicate and interact with
audiences through VR technology. The cheapest VR headsets you can buy are sold at a price of
$20 dollars (Google Cardboard), which makes it very accessible.
In my opinion, public folklorists should be more open to new technologies and
approaches. There are already virtual museums that use VR technology, but the possibilities are
endless. This is a way to preserve culture, a way to reach out to people. It allows people with
disabilities to participate in the conversation and activities; allows to have a creative space with
no limitations.
Often folklorists are stuck in the past. But folklore is present and future. We should not be
afraid of those new technologies, but be the first to embrace them. Instead of using the same
methods and approaches, we should look for new ideas and dare to experiment with them.
Folklorists are looking at so many different topics, but forget that how you look at them matters
as much, if not more. The search for the best ways and practices should be on the front line of
our discipline. I do not have an answer to which evaluation method is the best or what new
technologies we will have an access to soon, but what I do know is that I am going to look for
them. And that is one of the biggest lessons of my work term.e…