Implementation and Evaluation of a Media Literacy Skills Curriculum: An Action Research Study

Received : 18.12.2021 Revision : 22.01.2022 Accepted : 13.02.2022 The literature highlights the significance of media literacy instruction in pre-service teacher education period, but there are few attempts to develop and implement curricula to this end. This action research study aimed to report the steps to adapt ‘the media literacy skills curriculum design’ for pre-service teachers, which was developed for face-to-face education environments, to an online learning management system in line with emergency remote teaching amid Covid-19 pandemic, and report the results of curriculum evaluation. The study adopted ‘The Curriculum Development through Action Research Model’. The participants were pre-service teachers at an education faculty of a state university in Turkey. The quantitative data were collected through Media Literacy Skills Scale and qualitative data were collected using several forms, rubrics, and reflection tools. The study revealed that the implementation of the media literacy skills curriculum had a strong positive effect on pre-service teachers’ levels of media literacy skills. The study also revealed changes in pre-service teachers’ perceptions of media and media literacy, interactions with media, as well as certain areas for curriculum development in terms of contents, teaching-learning experiences, and assessment components of the curriculum. The study discusses the results and offers implications for media literacy education in preservice teacher education.


INTRODUCTION
The rapid advancements in information and communication technologies (ICT) have not only affected individuals' job-related or daily life practices but also led to dramatic changes in construction, sharing, spread and consumption of information (Kellner & Share, 2007). Media tools and platforms have an intense role in this change. Media is now a global power shaping people's values, beliefs, behaviors, and decisions (Baran, 2014). As well as the advantages offered by new technologies and media, there are serious disadvantages of this fast-going process. Though it is now easier to attain information, the quality and credibility of the information that is bombarding people is a serious problem people confront today. Particularly new media offers a platform for the ones whose voices are unheard in the mainstream media, and it has the potential to enable social participation, equal society, and equal representation for all. However, there are also issues of privacy, security, bullying, addiction, or phishing (Burnett & Merchant, 2011).
The pervasive spread of media among individuals from all ages necessitates that individual should possess new sets of skills and knowledge. This sets of skills and knowledge gather under the umbrella of media literacy which is regarded as a 21st century skill (Finegold & Notabartolo, 2008; The Partnership for 21st Century Learning, 2009). This is particularly pivotal for the younger generation (Thoman & Jolls, 2004). Hence, media literacy education is now an obligation as opposed to a preference. The most important issue regarding equipping new generations with media literacy skills is related to teachers' skills and competencies with regard to media literacy. For effective media literacy education, it is imperative that not only teachers who teach media literacy but also all other teachers need to be media literate (Domine, 2011;Donohue & Kelly, 2016;Goetze et al., 2005). As well as teaching media literacy directly, teachers need media literacy skills to integrate new media in their courses (Tiede & Grafe, 2016). Despite the need for equipping teachers with media literacy skills, the literature highlights the lack of structured media literacy education in preservice teacher education period (Baker, 2010;Cramer, 2015;Jolls & Wilson, 2014;Redmond, 2016).
There have been few attempts to develop a media literacy curriculum for pre-service teachers, particularly in Turkey. Although there are practices of media literacy education in pre-service teacher education in some developed countries and several institutions developed on media literacy, Turkey is quite limited in this regard. To this end, the researchers developed a media literacy skills curriculum (MLSC) design for pre-service teachers (Erdem, 2018a); however, the implementation practice coincided with the global Covid-19 pandemic. The curriculum design was developed for face-to-face education, and the online platform used in the implementation of the MLSC had certain limitations, requiring adaptation to the new context. Therefore, this action research study was aimed to implement the media literacy skills curriculum with pre-service teachers through adapting it to an online learning management system (LMS) amid the Covid-19 pandemic and to reveal its effectiveness and areas for curriculum development.

THEORETICAL FRAMEWORK Media Literacy and Teacher Education
The widespread interaction with media, particularly the new media lately, makes media literacy a current issue and a skill to be acquired. Although there are various differences between mass media and new media, Buckingham (2009) argued that new media was a type of media, and traditional media and new media were already integrated.
Traditional or new, media is now an indispensible part of people's lives, making media literacy education pivotal. How to teach media literacy is a matter of discussion in the literature. Basically, there are two approaches in media literacy education. The first is the independent course approach which argues that media literacy should be taught as an independent lesson. The second is the integration into a curricula approach arguing that media literacy should be taught in various lessons through integrating media literacy contents to curricula of various lessons. Both have advantages and disadvantages, and both are needed for an integrative media literacy education (Baker, 2010;Tüzel, 2013). Media literacy education can be practiced with a media tools-based approach, thematic approach, media studies approach, or an integration approach (Wilson & Duncan, 2008).
The integration approach in media literacy education has gained acceptance lately. In Turkey, for example, there is an independent optional media literacy course at the lower secondary level; however, it was suggested to integrate these contents with the curricula of all related courses (Radio and Television Supreme Council, 2012). Undoubtedly, teachers need to be equipped with media literacy skills in pre-service or in-service periods to be able to teach media literacy to children while being a model for them. However, teachers' level of media literacy skills is a matter of question (Baker, 2010). For effective media education, media literacy education should be integrated into pre-service teacher education (European Conmission, 2006). For teachers to be able to teach media literacy skills to children, they should be first equipped with these skills, and they should understand and implement these skills (Jolls & Grande, 2005;Tiede & Grafe, 2016).
The literature stresses the contribution of media literacy skills to teachers. A media literate teacher encompasses media literacy pedagogy into one's instructional processes (Hobbs, 2010), better understands students' educational and social contexts (Goetze et al., 2005), can be a model for students, and finds new ways to involve students to media literacy education (Schwarz, 2001). Also, this teacher has an ability to teach critical thinking as well as media literacy (Flores-Koulish, 2006), knows how to teach in multi-model environments and to coordinate students' formal and informal learning (Resta & Carroll, 2010), and develops one's general teaching specialties (Redmond, 2016).

Education
Media literacy education started in Turkey with a delay when compared to Western countries. It started with the introduction of a media literacy course which was offered as an optional course for lower secondary level students. The Radio and Television Supreme Council in Turkey led the development of media literacy education. In coordination with the Ministry of National Education, this council organized various workshops and meetings and suggested such a course in schools. However, media literacy education was limited to this optional course at lower secondary schools. Some communication faculties at universities had media literacy courses, yet media literacy education was not offered to preservice teachers at education faculties (Erdem, 2018b). Although there was a media literacy course in lower secondary schools, the education faculties did not have any related departments. A very limited number of teachers had in-service education for media literacy education.
The introduction of a media literacy course at lower secondary school was also a signal rocket for researchers in Turkey. As of this year, academic interest has focused on media literacy and a great deal of research has accumulated in the Turkish literature. With regard to research on pre-service teachers, the studies focused on identifying pre-service teachers' levels of media literacy, revealing their opinions and perceptions on media literacy and the media literacy course, developing media literacy scales, examining media literacy education in Turkey and in the world, and identifying the relationship between media literacy and some other related variables (Erdem, 2018a). Not many studies addressed media literacy education practices in pre-service teacher education (Barut et al., 2016). Recently, there have been some attempts to develop media literacy curriculum for pre-service teachers (Erdem, 2018a;Karataş & Sözer, 2018). After these attempts, the Higher Education Council added a media literacy course to the curricula of pre-service teacher education programs as an optional world knowledge course. To the researcher's best knowledge, there has not been any published research to report these experiences with the new course.

Purpose & the Current Study
Despite the emphasis on the need for media literacy instruction in pre-service teacher education, teachers try to teach media literacy without receiving a media literacy course in the faculties of education (Scull & Kupersmidt, 2012) or in primary or secondary education (Robertson & Hughes, 2011). The studies refer to a lack of well-grounded media literacy education in teacher education programs (Donohue & Kelly, 2016;Manzoor, 2016;Stein & Prewett, 2009), leading to an incoherent, disorderly, and incomprehensible media literacy perspective (Jolls & Wilson, 2014). Similar to this problem across the world, teacher education programs in Turkey did not offer media literacy education to pre-service teachers until 2018. In this year, an optional media literacy course was introduced.
The researchers aimed to implement the media literacy skills curriculum developed for pre-service teachers (Erdem, 2018a); however, the Covid-19 pandemic broke out and the universities closed in Turkey and continued education through emergency remote teaching.
The researchers had to implement the MLSC using an online LMS which had certain limitations as the country was not ready for distance education. Besides, the curriculum design to be implemented was developed for face-to-face education. Therefore, the researchers had to adapt the MLSC to the new learning environment, which led to this action research. Aiming to adapt the MLSC to the new learning environment, this study aimed to report the steps in this adaptation as well as reveal the effectiveness of the curriculum and highlighting areas for curriculum development. The research questions are listed below.

Research Questions
1. What is the effectiveness level of the MLSC in equipping the pre-service teachers with media literacy skills? 2. How do the pre-service teachers' perceptions regarding media and media literacy differ before and after the curriculum implementation?
3. What are the pre-service teachers' opinions with regard to the effectiveness of the

MLSC?
4. What is the quality of the learning products the pre-service teachers developed during the course? 5. What are the practitioner's perceptions regarding the MLSC, implementation process, and students' learning products?

METHOD Research Design
This is an action research study. Action research is a process whereby participants explore their own practices in classroom systematically and carefully, making use of research techniques (Ferrance, 2000). It is a form of research that teachers use to solve problems and enhance their professional practice in their own classrooms (Parsons & Brown, 2002). The reason for using action research in the current study was that the study sought to implement a media literacy skills curriculum for pre-service teachers in a pandemic environment. The MLSC was originally developed to be implemented in a faceto-face classroom setting. However, due to the emergence of Covid-19 pandemic, the researchers needed to facilitate the curriculum through distance education. This unexpected case posed some problems and led the practitioner to make new decisions as to how to implement the curriculum within the limitations of learning platform. Action research aims to solve a particular educational problem, improve educational practice, and hence help make a decision at a single local site (McMillan, 2004). In parallel with this, it also aims to enhance the teacher's professional judgment, and provide insights into better means of achieving intended educational outcomes (Mertler, 2017). The researchers needed to adapt the curriculum to the new learning environment. This process led the researchers to pursuit a curriculum development practice using action research. The study particularly adopted the Curriculum Development through Action Research (CD-AR) Model (Saban, 2021) the stages of which are explained below.

Action Research and Curriculum Development
The term action research was first used by Kurt Lewin in 1946 and Stephen M. Corey was the first to adapt action research idea to education. As stated by Saban (2021), Corey (1949 thought that traditional research had little impact on school practices because they were conducted by out-of-school researchers while action research was conducted by the school staff to improve practices in the school. Since 1950s, action research has been used as a method to enhance curricula. Action research and curriculum development processes have a similar purpose/function (Saban, 2021). The current research study adopted the   Saban (2021) argued that he developed this model for practitioners to understand and improve the curriculum in practice. As in all action research models, this model also posits a cyclical problem-solving process. The model involves six interrelated stages, as visualized in Figure 1. What the researchers did in each stage of the CD-AR model is briefly explained below in line with the model.

Stage 1: Identify the Focus Area
In this stage, a problem situation needs to be identified. The curriculum design to be implemented had been developed by Erdem (2018a). The media literacy skills curriculum was developed for pre-service teachers to enhance their media literacy skills, so that they could use these skills when they become teachers. The background aim was to be able to teach media literacy skills to K-12 students in an integrated way. There are two approaches in media literacy education: an independent course approach and integration to curricula approach. Both have advantages and disadvantages, and both should be used for an effective media literacy education (Baker, 2010). There was already an optional media literacy course at lower secondary level in Turkey. For ensuring an integrated approach, teachers and pre-service teachers needed to be media literate. To this end, the MLSC was developed. After a few years, the researchers had the chance to implement the curriculum; however, the Covid-19 Pandemic emerged and updates to the curriculum to fit in the new context required addressing. The distance education learning platform offered by the university had several limitations, which made things more complicated. The live lessons were limited to 50 minutes. The practitioner could not see the students' cameras all together, and the students could not attend a simultaneous discussion. The researchers needed to adapt the MLSC in the best way to implement it in this new learning platform. Therefore, action research was used, the CD-AR model in particular, to both implement and evaluate the curriculum.

Stage 2: Perform a Needs Analysis
In this stage, students' needs should be analyzed to develop some strategic ideas to enhance the problem situation (Saban, 2021). First, the MLSC design had been developed based on an extensive needs analysis in the same education faculty. For the current practice, the researchers did not have the chance to meet students in person due to the pandemic.
The researchers conducted the needs analysis online in two formats: quantitative and qualitative data collection. First, the pre-service teachers took the Media Literacy Skills Scale (MLSS) (Erişti & Erdem, 2017). This allowed the researchers to evaluate the students' overall media literacy skills levels, as well as the specific skills of access, analyze, evaluate and communicate to focus on. The adopted media literacy definition in the media literacy curriculum implemented in the current study belongs to National Leadership Conference on Media Literacy, which suggests that media literacy is "the ability to access, analyze, evaluate and communicate media in a variety of forms" (Aufderheide, 1993). Both the curriculum and the scale used for needs analysis were in line with these abilities of access, analyze, evaluate and communicate. In the qualitative part, the participants were asked to answer six questions before the lessons started. The questions were aimed to reveal what they already knew about media literacy, how they defined the media literacy, the reasons for selecting media literacy course, their expectations, their interaction with media, their strategies in their interaction with media, and the metaphors they ascribed to media. The analysis of the qualitative data also led the way for adapting the media literacy curriculum.

Stage 3: Develop an Action Plan
This is the stage where an action plan based on the needs analysis is developed, and a timetable is formed in line this with plan. Based on the needs analysis and limitations of the LMS, the weekly schedule was changed and adapted the contents. Therefore, the weekly schedule was changed into this plan presented in Table 1 below. The needs analysis led to an action plan in which only one week was allocated to 'access' skill, and the main focus was on the 'analysis' skill. While implementing the curriculum, the skill of evaluate was merged to the skill of analysis. While working on analysis, we had to also make reference to the evaluate skill. Finally, the last three weeks were allocated to 'communicate' skill.

Stage 4: Implement the Action Plan
In this stage, the action plan was implemented with a flexible approach. The lesson started with live lessons, and the contents were introduced in these lessons. The practitioner (the first author) tried to be as interactive as possible, and urged the students to express their ideas using their voice or the chat box in the written form. The system did not allow for opening each student's voices. The students could ask for permission to talk and only one student could speak at a time. Therefore, the chat box was used mostly. Before the lesson, the procedure for getting permission from the Ethical Board of the related university for ethical approval for the study was already started. All the students were informed about the implementation of this new curriculum, and data would be collected from them as part of publishing this experience in a peer-reviewed journal. They could participate in any part of the data collection. They were informed that they had the opportunity to not take part in the research. They had the right to not provide data for certain questions. All the students gave their permission for data collection.

Stage 5: Evaluate the Process
Every two weeks, the practitioner used some time in the last five minutes of the lessons to talk about how the lesson was going on according to students, and their suggestions for further contents were collected. In week 9, the pre-service teachers reported that although the course was effective, they needed more discussion on new media.

Stage 6: Reflect on the Process
In each stage of the model, the researchers should reflect on the process. They reflected in each stage and designed the remaining weeks based on these reflections. Both to obtain the participants' reflections and reveal the effectiveness of the curriculum implementations, the participants answered some questions as to their expectations from the lesson, their interaction with media and whether their expectations were met or not, and whether there were some changes in their interaction with media.

Participants
The participants were pre-service teachers studying in the faculty of education at a state university in Turkey. The participants included 78 sophomore and junior pre-service teachers who chose media literacy course as an elective course in the spring semester of 2020-2021 academic year. The participants were accrued through the convenience sampling method. The students who opted for this course and who volunteered to provide data were the participants of this study. The number of the participants differed per data collection process based on their voluntariness. The quantitative part of the research involved 61 preservice teachers. These participants completed the MLSS as a pre-test and post-test. In the qualitative part, the number of the participants changed as per data collection steps.

Instruments
In the quantitative part of the study, the Media Literacy Skills Scale (MLSS) was used (Erişti & Erdem, 2017) as a pre-test and posttest. The MLSS was developed for measuring pre-service teachers' levels of media literacy skills in a similar context. The MLSS is a fivepoint Likert type scale. The scale was reported to be valid and reliable in its development report (Erişti & Erdem, 2017 Saban, 2000) and teacher reflection tool (Share & Thoman, 2007) were also used during data analysis that are explained in the findings section.

Data Analysis
Descriptive statistics were used to examine students' levels of media literacy skills.
For further analysis, distribution of the data was checked first. To this end, skewness and kurtosis values were calculated. These values were -.451 and 1.366 for the pre-test implementation and .456 and -.599 for the post-test implementation, respectively. Since these values range between -1.5 and 1.5, it is accepted that the data has a normal distribution (Tabachnick & Fidell, 2013). Therefore, parametric tests were preferred for further analysis.
To be able to check whether the difference between pre-test and post-test was significant, paired samples t-test was used. Additionally, Cohen's d value was calculated to examine the effect size of the difference. The qualitative data were analyzed using content analysis, as offered by Green et al. (2007). For reliability and validity of the qualitative data, the researchers used constant comparison method for coding. The consistency between the coders was ensured. Besides, direct quotations were used in the study. The participants were coded as P1, P2, and so on in order not to reveal their identities.

Ethical considerations
The students were informed about the research, and their consent was sought. They could participate in any part of the data collection. They were informed that they had the opportunity to not take part in the research. They had the right to not provide data for certain questions. The quantitative data were collected online, and the students used a nickname they chose for matching pre-test and post-test data as well as ensuring anonymity.
The participants were coded as P1, P2 and so on in the qualitative data for anonymity. All the data are hosted on the first researcher's personel computer and safeguarded by a password.
In this study, all rules stated to be followed within the scope of "Higher Education Institutions Scientific Research and Publication Ethics Directive" were followed. None of the actions stated under the title "Actions Against Scientific Research and Publication Ethics", which is the second part of the directive, were not taken.

Results regarding the Effectiveness of the Curriculum in Equipping the Pre-service Teachers with Media Literacy Skills
The effectiveness of the curriculum in terms of providing pre-service teachers with media literacy skills was measured through quantitative instruments. The students took the MLSS before and after the course implementation. The descriptive statistics regarding the pre-test and post-test are provided in Table 2 below.  Table 3.  (Cohen, 1988;Sawilowsky, 2009). This result proves that the implementation of the MLSC had a strong positive effect on preservice teachers' levels of media literacy skills.

Participants' Perception of Media Literacy and Media
The participants' perceptions of media literacy and media, and the change before and after the course implementation were analyzed through their definitions of media literacy and metaphors about media.

Participants' Definitions of Media Literacy
At the onset of the semester, some questions to the participants were posed to both for carrying out a needs analysis and enabling a comparison between students' ideas at the beginning and end of the course. Within this scope, the first question was on defining media literacy. The participants were asked to define media literacy and state what they know about it. 36 participants answered this question. Five categories emerged in the content analysis for these data. These categories and the participants' distribution are presented in Table 4. As is evident in Table 4, 27 participants referred to tasks related to the access skill.
This indicates that the participants thought that media literacy is about being able to access media content through effective use of media tools. Some definitions of the participants related to access skills are listed below.
"Being able to read visual, textual and audial media contents" P4 "Having information about the use of the internet, mobile phones, tablets, and computers".

P32
These definitions indicate that some of the participants' perceptions of media literacy were limited to access related tasks. Some of the participants combined both access and analyze skills in their definitions. For instance, P11 defined media literacy as "the skill needed to access information, images and videos on the internet and to understand them effectively and correctly". These participants emphasized accessing and understanding the contents. One of the participants said media literacy was "understanding the media terms in using media" (P16). In this group, eight participants focused on questioning the media contents, confirming the information in media and using media tools and platforms consciously. Therefore, their replies were associated with access and evaluate skills. P13 wrote: The internet allows us to access every opinion on any subject, today. In the world of internet, there is useful information but there is also much information that aims to lead us to wrong things by manipulating people. Thanks to media literacy, people can measure whether the information on social media is correct or not, and people can have the consciousness to change the wrong information.
There were no single definitions or explanation that focused solely on the skill of 'communicate'. After referring to 'access' or 'analyze' skills, four students also gave reference to tasks related to 'communicate'. For example, P30 wrote: "It is following events in our country and in the world through social media, examining the opinions, and expressing our opinion. Sharing contents on our areas of interest and making comments to others' sharing".
These results demonstrate that the participants had lack of awareness regarding the effects of media, and the functions of media that are related to the skills of analyze, evaluate, and communicate. The same question was asked after the course finished. A total of fortytwo participants answered this question in this stage. Four categories emerged in the content analysis for these data. These categories and the participants' distribution are presented in Table 5. As presented in Table 5, only two students defined media literacy as using media tools efficiently. Before the course, this number was eight, and 27 students' definitions were related to 'access' skill. This time, 29 students associated media literacy with gaining critical autonomy in interaction with media. The students' definitions included emphasis on understanding the background of media messages; conscious use of media; knowing how media is constructed and analyzing media; understanding and interpreting media messages; being a conscious media consumer and producer; critical analysis of media messages; recognizing manipulations; making distinctions between correct and incorrect information; being aware of media effects; and identifying hidden meanings in media.
Additionally, six participants provided the definition adopted in the course. Combining these two categories, 34 students associated media literacy with critical autonomy. Finally, five students emphasized the skill of communicate. They defined media literacy as a means for claiming rights and expressing oneself through creating and sharing one's own media messages. It should also be noted that communication is also a part of the adopted definition in the course. P43 stated: "Media literacy allows us to claim our rights and support our ideas.
I realized that doing this on media will not end up with bad results. In contrast, it can help us develop ourselves".
When the definitions before and after the course are compared, we can see that the participants' perception of media literacy changed to an extent. Previously, the participants considered media literacy as a means for accessing the media messages and using media tools efficiently. However, after the course, they perceived media literacy as a means for establishing critical autonomy in their interactions with media as well as using media for expressing oneself and claiming rights. One-to-one comparison also supports this shift in the participants' perceptions.

Pre-service Teachers' Metaphors of Media
The participants were asked to generate metaphors about media to be able to understand their perceptions towards media before and after the curriculum implementation. The aim was to see the difference in their perceptions. The participants' initial metaphors about media are listed in Table 6.  "Media is like a supermarket because I can find the things I want". P 9 "Media is like a toy because we played with toys when we got bored in the past. We now cling to media when we are bored. We spend good time". P36 On the other hand, the explanations of negative metaphors emphasized spending too much time on media (addiction) and negative effects of media. Sample quotations include: "Media is like poisoned honey because it tastes good and makes me happy momentarily but it damages me in the long run. It has a potential to turn into addiction". P25 "Media is like a mine field because people's lives fall apart due to a tiny mistake on media"

P13
Three participants generated neutral metaphors. These metaphors involve both positive and negative aspects of media. For instance: "Media is like life because there are both good and bad things in it" P 28.
The same data were collected after the curriculum implementation. The distribution of the latter metaphors is presented in Table 7. After the course, the participants provided 20 positive, five negative, and seven neutral metaphors about media. Regarding the distribution, the number of negative metaphors reduced while the number of neutral metaphors increased. The same themes emerged in the post-implementation data with the prior data in terms of positive metaphors.
They had positive perception about media because they had fun with media, and they perceived media as a means for accessing new information. The main difference was in the negative and neutral metaphors. Some of the negative metaphors transformed into positive or neutral metaphors. The neutral metaphors emphasized that media is beneficiary for people, yet they need to be cautious for not spending too much time and thus protecting from negative effects. For example, P19 stated: "Media is like a snowball because a perception about something is introduced in media and then it gets really big. We should be able to keep track". Similar to this statement, the participants in this category emphasized the need for conscious use of media. P18 stated: "Media is like a meal because it gives you enjoyment and the body needs it, but if you eat too much it harms you. We should know what to eat and how much to eat".

Participants' Views before the Course Implementation
The participants were asked some questions before the course was started to be able to understand their views after the course. Their views about the course curriculum after the implementation could be affected by some factors. The analysis of pre-implementation data revealed four categories and some codes which are presented in Figure 2.

Figure 2. The Participants' Views before the Course Implementation
The participants' views before the course implementation were examined and interpreted under four categories. The first category was the reasons for selecting media literacy course. Before the course started, the pre-service teachers were asked why they chose this optional course. Their responses were composed of four common codes: Quota problem, skills development (developing oneself in media and contributing to future students), enjoying media and having interest in media, and 'I like the teacher'. The most frequent code was enjoying media and having interest in media. Nineteen students' responses included this code. The participants stated they were constantly using media, and therefore, the name of this new course attracted their attention. Ten students stated that they had intended to select another course, but its quota was full. Therefore, they had to select this course. There were five different optional courses in total in that semester. Some of these students also stated that this was the most interesting one among the other options. Nine students stated that they wanted to improve themselves in terms of media use and hence contribute to their future students because their students would be using the media too much. The last code was that they selected the course because they took other courses of the instructor and liked him.
The second category was their interaction with media. To this end, they were asked how much time they spent with media tools. None of the participants said s/he never used media. While eight students used media occasionally on a daily basis (1-3 hours), 23 students used media more than three hours a day, mostly 6-7 hours a day. Despite this intense interaction with media, the third category revealed that 22 students thought that they were not addicted to the media. Only five students reported addiction to media. Four students also thought that they demonstrated addiction behaviors from time to time. As for the final category, their expectations from the lesson included being an effective user of media tools and using media consciously.

Participants' Views after the Course Implementation
The analysis of the pre-service teachers' views after the course implementation revealed two categories. The categories are presented in Figure 3.

Figure 3. The Participants' Views after the Course Implementation
The first category is their views on the course in general with the sub-categories of meeting the expectations, contentment status, and change in interaction with media. Thirtythree participants participated to this data collection stage and all of them except for two participants stated that the course met their expectations. For instance, P29 stated: The course was beyond my expectations, actually. It provided surprising information about media and helped me recognize the hidden messages in the advertisements and understand the propaganda techniques. I started to approach more carefully to media messages, and I now question the media contents.
The two participants who stated that the course did not meet their expectations stated the reasons that the course needed to be face-to-face due to its contents, and it did not focus on the nonsense contents of the television series. These data suggest that the participants' expectations from the course were met in the general sense.
Given that nearly all the students were contended with the course, they were asked about the issues they were satisfied and dissatisfied with. The analysis of this second subcategory demonstrated that the issues they were not contended were again related to distance education. Two students said that they were not fully satisfied with the course due to the limitations of the online education. Another student mentioned that some of the examples provided by the instructor during the course could be more up-to-date. When it comes to the issues they were satisfied with, the participants provided a number of issues.
They said that they were contended with the course because the teaching learning process was fun and enjoyable (8 participants), they acquired the needed skills in this course (7 participants), the assessment in the course helped them learn the contents (4 participants), the instructor exemplified detailed analyses (4 participants), and it was not a course for memorization (4 participants). The other issues of contentment included: the materials were adequate and accessible; the instructor took the students' opinions in each stage; the course involved critical thinking; this was the first time they initiated a social media campaign; everything about the media was included; they were not worried about grades; the instructor taught all the lessons; and the use of assignments instead of exams.
The third sub-category was the change in the participants' interaction with media after they took the course. The students were questioned if taking this course led to any changes in their interaction with media. Additionally, they were specifically asked what they did differently in their interaction with media when compared with their earlier experiences.
Most of the participants (25) emphasized that they gained awareness particularly about the analysis skill and they started to analyze the media contents they encountered. P21 stated: This course changed my interaction with media. I can now analyze deeply the media contents, particularly the ones I see in the advertisements. I can now understand why they use some symbols in the series or advertisements. I recognized that I had not thought this way before I took the course.
Additionally, five students highlighted that they started to question the background of the media texts, and eight students emphasized they started to compare the media contents in different sources. There are also statements such as "I care about media security", or "I can recognize the effects of media". These changes are related to the media literacy skills of analyze and evaluate. Only one participant referred to access skill regarding the change after the course. He communicated that he now used efficient ways while accessing information on the internet. As for the skill of communicate, two participants told that they gained awareness about social campaigns and a few students highlighted that they started using media to solve their problems and initiating campaigns on media. Some students started to create their own media contents. A participant underlined that she now shares media contents by considering the principles of communication in media.
Besides, six students said that they limited their interaction with media and started using media more consciously. A participant realized that staying completely away from media was not the solution. P15 stated: "Yes, this course changed my interaction with media because now I do not think that I should be completely avoid media to protect myself from its negative effects. Instead, I realized that I should use media properly. I try to protect myself from its negative effects and make use of its benefits". Finally, four students said that their interaction with media did not change at all.
The second category regarding the participants' views after the course was their views related to components of the course curriculum. They provided feedback for contents, learning-teaching experiences, and assessment and evaluation elements of the curriculum in practice. Regarding the contents of the course, most of the participants thought the contents were adequate for this course, but they also provided some issues after being asked what else could be covered in this course. The participants' suggestions for contents include more emphasis on printed media, on social media, news analysis, historical development of media tools, less known social media websites, series and movie analysis, more emphasis on social problems, the relationship between media and teacher, more emphasis on social media or media addiction, secure media applications, pre-school children's media experiences, shows on television, virtual money, online education websites, games, fraud on the internet, and cyber-bullying. While some of these contents are not related to the purpose of the course, some of them were not covered in detail due to time constraints. For instance, we mentioned about printed media, social media, news analysis, development of media tools and so on but our live lesson was limited to fifty minutes. Therefore, we did not have the chance to go deeper into all contents.
Most of the participants were contended with the learning-teaching experiences of the course. Quite a few students, however, emphasized the limitations of the online LMS we used, and they wished the course was face-to-face. Two participants maintained the need for more discussion during the live lessons. Indeed, the learning platform hindered this to some extent. It was hard for the students to speak during the lesson, because we could not open all participants' voice at the same time. A student had to click the button to speak, and after the instructor allowed the student to speak, s/he could speak but there were mostly connection problems in students' voices. Therefore, I mostly urged the students to use the chat box for instant comments and discussion. This hindered a classroom atmosphere. In addition, when asked about the positive aspects about the learning-teaching process, the students told that the instructor provided numerous examples, they easily understood the contents, the learning process was enjoyable, and the modeling of media analyses was effective. P18 stated: "The learning process was good in the general sense. We learned the contents without getting bored. The instructor did everything that is possible in a distance education setting".
Regarding assessment and evaluation, most of the students thought that use of projects instead of examinations was good for this course and they said that the assessment technique helped them learn the contents better. When asked for further suggestions, they had some recommendations. The most frequent was giving more project homework throughout the semester. Projects such as creating a video or introducing a product like an influencer could be assigned to students in addition to the existing projects. Some students recommended that the students could make presentations regarding the course contents, or they could present their projects during the live lessons. However, the time constrains made this impossible. Besides, a few students also recommended examination measuring their content knowledge.

The Quality of the Participants' Learning Products
The assessment and evaluation of the course included a midterm and a final exam.
Instead of carrying out open ended or multiple-choice tests, participants were assigned two projects, one for the midterm and the other for the final grade. The midterm project was related to analysis skill. The students were expected to select a media content (advertisements, news, TV shows, and so on) and analyze this content based on the five core concepts (constructedness, content, format, audience, and purpose) and key questions provided by Thoman and Jolls (2005). Besides, they needed to identify and explain the propaganda techniques used in the media contents. Finally, the students were asked to write a reflection paragraph regarding the contributions of the course contents and the current project to their media literacy skills and provide recommendations for the following lessons.
The project products were assessed based on a rubric. The rubric was adapted from "key questions to ask when analyzing media messages" grid provided by National Assocation for Media Literacy Education (NAMLE, 2007). This grid was used as a rubric and each criterion in the rubric was graded from 1 to 4, unsuccessful, basic level, successful, and very successful.
The second project was related to communicate skill. The students were expected to observe their social circle, identify a problem, and start a campaign to solve the problem using media tools and platforms. The students were supposed to report this process based on an adaptation of the problem-based learning steps proposed by Saban (2000). The steps included 'identify the problem, determine what is known and what is needed and data collection, define the problem, generate solutions, choose the best solution and apply, and reflection and reporting'. The students needed to explain what they did in each stage and provide proofs. These stages were also used as a rubric. Similar to the midterm project, each criterion in the rubric was graded from 1 to 4, unsuccessful, basic level, successful, and very successful.
All the participants submitted their projects. The projects were assessed and graded based on the rubrics explained above. The quality of the projects was satisfactory. The students used the five core concepts and key questions offered by Thoman and Jolls (2005) and identified and explained propaganda techniques such as symbols, exaggeration, humor, iteration or stigmatization. The mean of the students' grades in the midterm project was 3.65 which was within the range of "very successful". In the reflection part, students noted that they could apply the analysis techniques to media contents, and they were happy about this. They could identify the propaganda techniques in the media contents that were not very salient without a deep analysis. Similarly, the final project was also satisfactory.
The students identified a social problem and initiated a social campaign on various sites such as change.org and then promoted this on social media. The mean of the students' grades in the final project was 3.70 which was within the range of "very successful".

Curriculum, and Learning Products
The media literacy skills curriculum design for pre-service teachers had been developed by the first author in his PhD thesis under the supervision of the second author, based on an extensive needs analysis and principles of curriculum development (Erdem, 2018a). However, this curriculum design was not implemented, and it was designed for face-to-face instruction. When the researchers had the chance to deliver media literacy course to pre-service teachers, the Covid-19 pandemic began, and we had to teach our courses online. The online LMS we were using had certain limitations. Within this context, modifications to the curriculum to fit in this new learning setting were facilitated. To this end, an action research approach was chosen. In addition to the data presented above, the practitioner (the first author) took some notes during the semester, asked the students for their opinions in every stage, implemented the course himself, and graded the students' projects (learning products). Therefore, the researchers had the chance to witness the entire research process. Besides, Teacher Reflection Tool (Share & Thoman, 2007) was also used as a self-evaluation rubric designed specifically for instructors teaching media literacy.
First, the curriculum covered all media literacy skills; however, the focus was on analyze and communicate skills because these are the least developed skills in the general sense. Particularly for the analyze skill, the students needed to gain critical autonomy in their interaction with media. The students' participation to online discussion was limited because participating to discussions in voice synchronously was a problem. Some of the students used the chat box for discussion, yet this was limited. This was one of the most prominent problems during the semester. With regard to the components of the curriculum, some of the contents in the access skill were skipped due to the fact that students already knew some of the contents. The contents of evaluate skill were merged with the contents of analyze skill because evaluate was complementary to analyze. Therefore, the researchers had to revise the contents of the curriculum throughout the semester.
With regard to assessment and evaluation, two projects were announced for midterm and final assessments at the beginning of the semester and administrative documents were arranged in line with this. It came out that although assigning projects for assessment was a good idea, the assessment component in the curriculum was not adequate.
The projects focused on analyze, evaluate and communicate skills of media literacy. And the students had the chance to apply what they learned in the project. The theoretical knowledge on media and media literacy and contents related to access skill were not assessed.
The progress of the course was continuously checked using the Teacher Reflection Tool (Share & Thoman, 2007). After the course was completed, the practitioner filled in the form once again considering the whole semester. The mean of the total score was 2.5, out of 3. The items ticked as often included "Do my students read and analyze both print-and non-print-based texts?", "Do my students work collaboratively?", "Do my students analyze texts from different perspectives?", "Do my students attempt to solve real problems that affect them and their community?", "Does my curriculum emerge from student interest?" and "Is there an understood norm where everyone participates and is listened to?". During the course, the students were informed about both traditional and new media. We analyzed both news and columns from newspapers and contents from new media such as advertisements on Youtube, entries on Twitter or Instagram. The students worked collaboratively. The students were given the chance to form groups and create the projects together. Some of the students did the projects in groups. The students analyzed media texts from different perspectives. They analyzed the texts in terms of five core concepts and propaganda techniques. They considered themselves as the audience in some examples and analyzed the texts in that context. Particularly for the final project, the students identified a social problem and started a media campaign to solve that problem. Some of the students continued this behavior after the course finished. The curriculum design was originally developed in the same education faculty a few years before the implementation and the researchers had an extensive needs and interest analysis in this stage. In the implementation stage, the students' opinions were always asked, and the contents and processes were modified accordingly. We set an online classroom setting where students understood that there was not a correct answer in media analysis, and everyone felt free to speak their opinions.
The items ticked as often included "Do my students write and create texts using both print-and non-print-based media?", "Does student work have a real-world audience beyond the teacher?", and "Are issues of social justice discussed openly and critically?". The students created media contents in the final project. The midterm project was based on analysis. The course was online, and we did not have the chance to meet face to face due to the pandemic. Therefore, we did not have the chance to create print media. The students had a real-world audience beyond the teacher in the final project. They actually started their campaigns and promoted them on social media. We talked about issues of social justice, yet the students were a bit reluctant to talk on these issues openly since the lessons were being recorded. The only item ticked as rarely was "Are my students talking more than I am?".
This was a general problem due to some technical and student related issues experienced in all courses in the faculty. Some of the students were not very eager to talk and the platform did not allow for a synchronous discussion. No items were ticked as 'never'.

DISCUSSION
This study reports the steps in the action research to adapt the media literacy skills curriculum design for pre-service teachers, which was developed for face-to-face educational environments, to an online LMS in line with emergency remote teaching amid Covid-19 pandemic. In addition, it reports the results of a curriculum evaluation process.
The curriculum design had been developed, but not implemented. With this practice, the curriculum enactment process is also evaluated and results regarding the evaluation of curriculum components are also discussed based on the students' views, their scores in MLSS, learning products, and the instructor's reflections. The results are discussed in line with the research questions.
The pre-service teachers' mean score from the MLSS was at medium level before the curriculum enactment. Also considering the possibility of overrating oneself in selfassessment scales due to effects such as social desirability (Dunning et al., 2004), it was concluded that the participants were in need of media literacy education. The studies in the literature regarding Turkish pre-service teachers' media literacy levels also reported similar findings (Erdem & Erişti, 2018;Uslu et al., 2016;Yılmaz & Aladağ, 2015). This result also supported the theoretical arguments in the literature suggesting that media literacy education is needed in pre-service teacher education (Considine, 2002;Fleming, 2013;Jolls & Grande, 2005;Redmond, 2016). Their mean score in the post-test after the curriculum enactment was at high level. There was a significant positive difference between the pre-test and post-test, suggesting that the implemented curriculum affected students' media literacy skills levels positively. Besides, effect size analysis revealed a very strong effect size, proving that the implementation of the media literacy curriculum had a strong positive effect on preservice teachers' levels of media literacy skills. This result demonstrates that the implemented curriculum was effective in equipping pre-service teachers with media literacy skills. As teachers need to first be media literates to be able to teach media literacy to their students and be a role model for them (Jolls & Grande, 2005), these pre-service teachers are expected to contribute to their future students in terms of media literacy.
To further analyze the effectiveness of MLSC, the participant pre-service teachers' perceptions of media literacy and media before and after the enactment of MLSC were analyzed. Before the course started, the participants' definition of media literacy referred to tasks related to access skill in media literacy. The participants thought that media literacy was about being able to access media content through effective use of media tools.
References to other skills of media literacy such as analyze or communicate were very limited or none at all, suggesting a lack of awareness regarding the effects of media, and the functions of media that are related to the skills of analyze, evaluate and communicate. After the curriculum enactment, most of the participants associated media literacy with gaining critical autonomy in interaction with media. And references to other skills of media literacy were more dominant this time. The content analysis revealed that that the participants' perception of media literacy changed to a great extent. As opposed to accessing media, after the course, they perceived media literacy as a means for establishing critical autonomy in their interactions with media as well as using media for expressing oneself and claiming rights. Media literacy is about gaining critical autonomy in interaction with media. It is the application of critical thinking to media, indeed (Jolls, 2008). Media literacy also involves creating media contents, sharing them with people, and getting into action using media (Hobbs, 2010;Schmidt, 2013). This result indicates a change in the participants' perception of media literacy in terms of skills such as analyze or communicate thanks to enactment of MLSC. The metaphor analysis also revealed a positive change in the participants' perceptions of media after the MLSC enactment. The positive metaphors in the pre-and post-implementation had similar themes; however, some of the negative metaphors transformed into positive or neutral metaphors. The participants emphasized making use of media but also being cautious for not spending too much time and protecting from negative effects. These findings also support the effectiveness of MLSC.
The participants were asked some questions before and after the MLSC enactment to evaluate the effectiveness of MLSC and also define areas for curriculum development.
The participants' views before the course implementation resulted in four main categories.
These data helped in understanding their answers after the course implementation. The first category, reasons for selecting the course, demonstrated that they had not made a conscious selection. The most common answer was that they liked media and used media in their daily lives and since there were also quota problems in other courses, they just chose this course.
Some students also referred to sympathy for the instructor. Few students told that they wanted to improve themselves in terms of media literacy and contribute to their future students. Teachers are not very conscious about 21st century skills and how to teach them (Donohue & Kelly, 2016). Regarding the finding in the question as well as the participants' responses of media literacy definitions and media metaphors in the beginning of the course, we can argue that pre-service teachers lack consciousness about the significance of media literacy. This lack of consciousness is even more evident in their responses regarding interaction with media and addiction to media. Although the participants reported 6-7 hours of media interaction, few students accepted addiction to media. People now live in a digital balloon (Pérez Tornero & Varis, 2010) due to the extensive interaction with media on first, second, and third levels (Masterman, 2005). Yet, just like fish in the sea, people are not aware of media since they are absorbed in it. The participants' responses regarding their media interaction supports this argument.
The participants provided their views on the course in general and on MLSC after the course. The analysis demonstrated that the participants were contended with the course and their expectations were met. The reason for discontentment in few students was related to distance education. Yet, they were satisfied with the course in terms of several issues including enjoyable learning process, equipping with media literacy skills, assessment procedures and being a conscious media consumer and producer. The participants reported that the course led them to change their interaction with media. They gained awareness about analyzing media contents, media background and effects, media use, fact-checking, and using media to participate to social life and solve problems. Besides, some students changed their interaction with media in terms of usage/screen time. They reported less media use.
The students also stated their opinions as to the components of MLSC. Most of the participants were contended with the course but when asked what else could be addressed, their responses included contents related to more emphasis on printed media and social media, news analysis, historical development of media tools, less known social media websites, series and movie analysis, more emphasis on social problems, the relationship between media and teacher, more emphasis on social media or media addiction, secure media applications, pre-school children's media experiences, shows on television, virtual money, online education websites, games, fraud on the internet, and cyber-bullying. While some of these contents are not related to the purpose of the course, some of them were not covered in detailed due to time constraints. These contents items note for curriculum development in terms of content development.
As for the teaching-learning process, most of the students reported contentment, except for complaints regarding online learning. The time for discussions on the contents was limited due to constraints in both time and LMS. This also refers to an area of curriculum development. In the assessment and evaluation component of MLSC, as opposed to written knowledge tests, the students were assigned projects for the midterm and final examinations. The students also reported positive feedback regarding the assessment and evaluation system of the course. This is also supported by the assessment scores of the learning products (project homework). The students' projects were within the range of "very successful" based on the related rubrics. The participants reported that the projects gave them the opportunity to apply what they learned in the course to real media contents, and hence improved their learning. However, the analysis of their responses revealed that two projects were not adequate for assessing the whole semester. The projects were related to analyze and communicate skills. Although all skills are related, skills of access and evaluate were not directly assessed, as well as theoretical basis of media literacy, indicating some problems in content validity of the assessment component. Accordingly, some students recommended more project homework throughout the semester as well as other suggestions such as written exams or presentations.
The first researcher was also the instructor of the course. This researcher's reflections also referred to lack of adequate and elaborate discussions, which was mainly due to the constraints of the LMS. The researchers had to make some arrangements in the curriculum such as skipping some of the contents in access skill or merging analyze and evaluate skills.
These actions also indicated areas for curriculum development. He also referred to inadequacy of projects in the assessment component of the course. The analysis of the Teacher Reflection Tool (Share & Thoman, 2007) also emphasized the problems due to the LMS.

LIMITATIONS
This study had certain limitations that should be considered in interpreting the research results. First, this study reports the results of the evaluation of the media literacy skills curriculum based on an implication in an online learning system. Implication of the curriculum may yield different results in face-to-face learning environments since some of the problems were related to the constraints in the LMS. A more comprehensive LMS could cultivate in better results. Second, the data were collected online. The participants were not in the campus due to the Covid-19 pandemic, so the researchers did not have the chance to conduct interviews in person. The students were reluctant to meet in online meetings, so their views were obtained using written forms. Elaborate in-person interviews could yield deeper data.

CONCLUSION AND IMPLICATIONS
This study revealed that the media literacy skills curriculum (Erdem, 2018a), which was developed for face-to-face education, had a positive strong effect in equipping preservice teachers with media literacy through enactment in an online LMS amid Covid-19 pandemic. Both the quantitative and qualitative findings support this result. Therefore, this study demonstrated that the MLSC can be used in pre-service teacher education period for teaching media literacy skills to prospective teachers. Given the peculiarities of the new century, media literacy is a must for all students. Teaching media literacy requires media literate teachers, and pre-service teacher education is an ideal starting point in equipping teachers with media literacy skills. This curriculum designed specifically for pre-service teachers effectuated good results despite an emergency remote teaching environment due to the unexpected pandemic.
In addition, the study reported the steps taken in the action research to adapt the MLSC to a new online learning environment. The adaptation and enactment of the curriculum culminated in good student performance, yet it also revealed certain areas for curriculum development. The study put forth the need for some arrangements in the components of the curriculum. First, the skills-based structure of the curriculum should be elasticized since the students may be knowledgeable in access skill and the skills of analyze and evaluate need to be merged in practice. Besides, the participants offered some ideas for curriculum. An important problem was related to lack of elaborate discussions on the issues which were due to the constraints of the LMS. If taught in similar environments, plans for improving a discussion atmosphere are needed. Another significant drawback of the curriculum enactment was about assessment. Although the projects were good in both assessing and improving learning, they were not adequate. More assessment techniques should be used for assessing theoretical base of media literacy, and skills of access and evaluate. There are valuable assessment options in the curriculum, which were not used at this time due to time constraints. They should be utilized in further implementations. The curriculum needs to be revised using the data that emerged in the current study. Further studies are needed which enact the curriculum in also face-to-face education. Besides, operability of the curriculum should be tested by enacting the curriculum in different context with different participants.