Is Competency-Based Accreditation the Death of the Credit?

I can only hope so. But don’t take it from me.

At the end of 2012, Carnegie Foundation announced it was looking into a new paradigm for the marking of student progress, away from the Carnegie Unit (more commonly known as the credit hour) towards a system based on competency (source). The Carnegie Foundation describes the Carnegie Unit as something that “was not intended to measure, inform or improve the quality of teaching or learning.

TechCrunch writer Gregory Ferenstein writes in mid-April of this year that this shift may already be happening [ahead of whatever decisions the Carnegie Foundation reaches] opining the days of the credit-hour degree “are numbered” (source). Mozilla is helping push this forward with its Backpack and OpenBadges projects. These allow for institutions or individuals to award digital versions of the merit badge to users based on specified competencies and performance criteria stored in the metadata of the badge itself. The Backpack associates a user with the badge which can then be displayed on other sites, creating a virtual badge sash on which to display accomplishments and skills an employer or client may find of value.

The idea excites me, and I don’t fear that this is the death knell of the university or the public school. What I hope it does accomplish, however, is a revamping of education that is so desperately needed. Thinking of my own experience, I was not permitted to take time off after high school before going into university. As a result, I feel I could have made better choices about my program and where I invested my money. That isn’t to say I didn’t get value for my investment-I did-but I may have chosen a different career path with a little more time to reflect and consider where I wanted to go in life.

As a teacher, I’m excited for the possibilities this presents. I’m a proponent of standards-based grading and ongoing formative assessment for learners. I think too much stake is put into a test, and the tests don’t often give valuable feedback about what the learner is doing well and where they need more help. From a teacher’s perspective, I am also excited by the possibilities this provides for breaking down the walls of the school (quite literally), creating space for more cross-curricular and interdisciplinary learning. It’s not bad to become an expert in a subject, but no part of our life exists completely apart from the others. No subject is an island.

In my own classes, I’ve been desperately trying to pull my students away from grade obsession and the dreaded summative assessment. In a culture that stresses standardized testing and rote memorization I’m certainly swimming against the tide. That said, the tide is changing and the Ministry of Education, Science and Technology is calling for a move away from “rote-based learning and teacher-centered instruction towards practice-based learning and student-centered instruction” (source). Well, it certainly sounds nice. And while I’m reserving opinion on its success until I see it, I also know Korean opinion can shift tectonically overnight. This is a country of proud tradition and rapid change. Still, there seems to be a focus on compartmentalized instruction rather than integrated learning and competencies. I would love to start a school. Anyone want to fund it?

With the push for standards-based education and the Common Core in the United States, many teachers are pushing back, arguing that they’re no longer able to teach as they are being stifled by the tests NCLB requires them to meet (example) and going so far as to advise against becoming a teacher completely (example). My only hope is that this combination of standards-based grading, competency-based credentials and frustration with the status quo will lead to the revolution so many have been waiting for.

Standards, projects, and communicating the "what" with students.

This semester I dove headlong into Project-Based Learning with my two conversation classes. These efforts sputtered along in my lower-level class, and met with some success with my intermediate class. I stuck with direct-instruction (with an eye on flipping and peer-instruction at a later date) for my writing class.

Last week I put out an anonymous survey to my higher-level class to get some feedback on how things were going. While the comments were almost entirely positive, the few negative comments made me wish I had run the survey earlier to make constructive changes. My interpretation of the negative feedback was that the students were having trouble seeing the “why” and the “what” of the projects we were doing, although I thought I had presented them well. Obviously not well enough for everyone. This feedback tells me I should probably develop a more structured presentation of the projects so that the students better understand exactly what – and why – we do what we do.

Next time I will:

  • Clearly link the project to the text
  • Indicate which chapters the project addresses and how
  • Clearly lay out grammar and language goals for the project

I will try these changes and run another survey after the first project next time to see how the students are responding to it.

CN: Unit 1 – Learning Theory

Learning Objectives

After completing this unit you should be able to:

  1. explain the difference between an archetype, a paradigm and a model; 
  2. discuss the key characteristics of Davies’ three educational technologies; 
  3. discuss the key principles of behaviorism, cognitivism and constructivism; 
  4. explain the key differences between behaviorism, cognitivism and constructivism; 
  5. discuss the difference between the objectivist and the subjectivist epistemologies; 
  6. discuss the implications of behaviorism, cognitivism and constructivism on instructional design and development; 
  7. explain the five perspectives on teaching; and 
  8. appreciate the value of different approaches to teaching and learning.

1. Archetypes and Paradigms

Reading:

Davies, I.K. (1978). Eucational Technology: Archetypes, Paradigms and Models. In J.H. Hartley & I.K. Davies (Eds.), Contributions to an Educational Technology, Volume 2. (pp. 9-24). New York: Kogan Page.

Questions for Reflection:

What is the difference between an archetype a paradigm and a model?

Archetype:

  • “the viewpoint or perpective used by someone engaged in an act of inquiry” (16)
  • appears to be consistent with what some would call a “worldview”
  • in education, an archetype serves as a common set of beliefs and assumptions the community operates under.

Paradigm:

  • a theory with legs.
  • includes definitions, statements and interrelationships between them
  • often qualitative
  • mind-map type models may be used to help explain the relationships
  • limited in scope
  • new and appealing enough to attract adherents from other paradigms
  • open-ended enough to be applied to a variety of problems, which thereby allow for refinement of the paradigm
  • is created to be disproven with the purpose of finding a more accurate paradigm

Model:

  • technically paradigms
  • more quantitative in dimension
  • used to determine effect on larger population in simulation with real individuals is considered largely unethical.
  • seek to represent reality accurately
  • is created to be used to solve a problem
  • usually specific to a particular phenomenon
  • different models will approach the same phenomenon from different paradigms.

 

What are key characteristics of Davies’ three educational technologies?

Technology 1:

  • Takes an engineering/hard sciences approach to application, focusing on better hardware
  • the latest machines make the best tools, ergo best education
  • focus on efficiency of education: bigger, broader audience outside of traditional boundaries
  • focus on transmission and reception problems

Technology 2:

  • behavioral science approach, focused on better software
  • technology = learning aids, facilitating new models of education
  • technology enhances teaching, providing guidance to new teachers and propelling experienced teachers to breakthroughs
  • focus on purposeful shaping of behaviour

Technology 3:

  • combination of hardware and software approaches
  • organic rather than mechanistic development
  • deep focus on processes and products of teaching and learning
  • more group than individual
  • concerned with quality and relevance of the experience
  • warmly human, total and integrated approach
  • emphasis on a range of contrasting skills
  • fundamentally a problem-solving approach*
  • problem boundaries may be hard to determine; incremental rather than substantial change for piecemeal progress
  • effectiveness > efficiency

*It strikes me that all of the technologies seek to answer problems, but they each see the problem differently.

 

What are the key characteristics of the audiovisual, the engineering and the problem-solving archetypes of educational technology?

Audio-visual:

  • oldest of the three
  • hardware is used to help present information
  • gives students (some) access to otherwise inaccessible content
  • can expand the learning environment to a larger geographical sphere
  • new instrument for assessment by speeding up and automating assessments

Engineering

  • influenced by Behaviourism
  • linear approach to learning, with defined edges and process
  • teacher vs. machine
  • Davies disapproves, argues that modifying the paradigm misses the need to assess the foundations of the theory

Problem-solving

  • started around 1973-74
  • as of 1978 not widely adopted although gaining ground
  • changes arise from dissatisfaction, moving to satisfaction as quickly as possible
  • solutions are situational, based on specific needs

What archetype, paradigm and model of educational technology do you think most of your teaching would fall into?

Archetype: Problem-solving

Paradigm: Subjective/qualitative

Model: ? I didn’t see any models in the article. As far as other pedagogical models are concerned, I subscribe to a constructivist model, and am presently reworking lessons to make use of problem-based learning and standards-based grading. This is in an almost completely (I say almost because although I haven’t seen otherwise I am cautious to say it doesn’t exist) direct-instruction environment where for my discipline there is essentially no curriculum. It is overwhelming to try to eek out a curriculum where there is so much disjointedness but I take it as a challenge and an opportunity to really teach what I think my students; a situation I know many of my North American colleagues would give teeth for.

Davies wrote his article in 1978, over 30+ years ago. Is it still relevant to the thinking of using technology in teaching and learning, today? Why or why not?

It is absolutely relevant. Although the technology has changed, the basic approaches (I, II and III) to educational technology have endured. Of note were some comments from page 12 that struck me as particularly insightful:

“Instead of viewing educational technology as an opportunity for renewing educational practice, it has, too often, been conceived as a means of doing what has always been done- only more efficiently” (pg. 12)

“technology is probably considered less self-limiting than our own perception or view of it” (pg. 12)

In my experience in speaking with other teachers and administrators, I can see how this has often been the case. Buus (2012) found that even today teachers need scaffolding to use new technologies for new pedagogy rather than pasting the new technology over the old pedagogy. This recent study supports Davies’ (1978) argument that it may only be a matter of increasing efficiency rather than determining how to readdress education entirely, which is the crux of his argument on the morality of education. If we fail to address why we teach something when new technologies present new possibilities, we are ignoring the very value of what we teach. It is a very common-sense and yet profound argument on the nature of education and its purpose to society.

This lack of insight, consideration of value, and attention to possible transformative nature of education technology is fortunately not global. There are teachers who are actively working on the what, why and how of doing education as new technologies offer new possibilities. In some ways, teachers are latching on to technology to move their pedagogy forward, with constructivist, standards-based, project-based and other de rigeur methodologies and curricula taking the front seat, but in other ways their use of these technologies paves the way for other teachers to jump over to more student-centered teaching styles away from direct instruction and other teacher-centered styles. As this jump is made, and technology is being used in new ways, they are employing what Davies (1978) refers to as “[t]he skills of effectiveness” (15).

2. Epistemological Traditions

Reading

  1. Pratt, D.D. (1997). Indicators of Commitment (pp. 22-25). Malabar, Florida: Krieger Publishing.

Questions for Reflection

As you read this excerpt, consider the following questions:

  1. What are the defining characteristics of the two epistemological traditions?

Objectivitism: holds that knowledge exists empirically, and it can be known in the same way by everyone. This knowledge exists whether or not the learner is interested in it, or is aware of its existence. Discovery. Modernist/Rationalist/Enlightenment ideas- through rational thought one can come to know the world. Amoral, neutral, detached. “Truth is a matter of the accuracy of reproduction (in language or action) of reality as judged by some authority.

Subjectivism: holds that knowledge is constructed by the knower. Interpretation. Post-modernist- reality is pluralistic (reality can be expressed in a variety of cultural paradigms) and plastic (people make or mold their own realities). There is no absolute Truth, only what people experience in the moment.

Objectivism seems to break down on the “some authority’ level, where some person through their own interpretation determines truthfulness. It seems, despite ones best efforts, that there is subjectivity in this- or that there very easily can be. While I agree with the idea in theory, especially for science and anything amoral, there are likewise personal aspects of truth that are subjective. The Truth of these subjective interpretations are also bound to some rationality. I quite like the conclusion of the objectivity section: “There is within objectivism another, slightly softer, belief about the separation of facts and values. It acknowledges that they are not actually separate, but interdependent, yet claims objectivity may still be achievable as long as values are allowed to dictate problems (what we examine) but not prejudge solutions (what we find). Therefore, we may allow that values will influence what we decide to teach, but we must guard against values distorting the content or influencing our decisions as to whether the content has been correctly learned.”

In the stump metaphor to describe subjectivism, I would say that while all the observers thought they were seeing something, it did not morph the stump into a policeman, a friend, or a foe. It was yet a stump. While the brain interprets the world for us, and we interpret it through our values, it doesn’t make something empirically true. It might true for us, it might be what the individual believes is true, but that doesn’t make it Truth, but rather, it is an interpretation. Interpretations have value, but they cannot supersede empirical truth. Where empirical truth is not knowable (for example, in determining someone’s motivations), interpretations (including the person whose motivations are being determined) may provide insight and valuable information leading to Truth.

  1. What do you think some of the implications of each tradition would be for the design of instruction?

Objectivists would have a list of facts or knowable things for someone to learn. Answers would be correct or incorrect. How students come to that knowledge may be through direct instruction, but may also be through project-based learning or other constructivist pedagogies, with the teacher serving as a guide, leading students to a particular outcome.

Subjectivists would likely allow for more free-form, self-directed learning and interpretation of results. I have no idea how this would be “fairly” assessed by the instructor, or what the assessment would look like.

Summary

With this very brief introduction to two major epistemological traditions let’s move on to examine three different theoretical perspectives on learning, beginning with behaviorism.

3. Behaviourism

Reading

Choose one of the following readings and consider the questions below.

  1. Elias, J.L. & Merriam, S.(1980). Philosophical Foundations of Adult Education. (pp. 82-90; 102-107). This article provides details on the background of behaviorism and some of its educational applications.
  2. Skinner, B.F. (1958). Teaching Machines. Science, 128, pp. 969-977. This article by B.F. Skinner provides a concrete example of how behaviorism has been applied to teaching.

Questions for Reflection

To focus your reading consider the following questions:

  • What are some of the ways in which the behaviorist principle of reinforcement has been applied to education?

Competency-based and criterion-referenced education, programmed instruction, computer-assisted instruction, mastery learning, teaching machines, contract learning, Personalized System of Instruction (PSI), Individually Guided Education (IGE), Individually Prescribed Instruction (IPI); the responsibility to learn is on the learner.

  • What are the roles of the teacher and the learner in the behaviorist framework?

The teacher should design environments that will evoke the behaviour that will ensure the survival of the species, societies, and the individual. The student is to respond to the environment in the desired way, acting for reinforcement and correction. They will demonstrate learning through specific behaviors.

 

  • What are some of the criticisms of behavioral objectives?

Learning is more complex than simple stimulus, response and reinforcement, that the behaviours pre-determined to indicate learning may be inadequate, learning doesn’t need to be structured, and may stem from the whole rather than through segmentation. Furthermore, behaviorism may not be the most appropriate form of learning for all subjects and has no guarantee of transferability.

  • What epistemological tradition (objectivism or subjectivism) do you think underlies the behaviorist view of learning?

Objectivist- if learning happens through breaking into chunks and memorization through repetition, then knowledge must exist outside of the learner and their interests.

  • Can you think of anything you do as an instructor that is influenced by the behaviorist view of learning?

Observation of change to determine outcomes. If my student doesn’t show improvement over the course of the semester, it is very difficult for me as an instructor to gauge how much they have or haven’t “learned.”

While this is more an issue for K-12 classes, discipline is also largely influenced by behaviorist approaches, on a consistent reinforcement schedule for infractions, and intermittent for rewards.

In designing lesson plans, the language is in behaviorist terms – students will be able to/will demonstrate/etc, and also the criteria for assessment, whether rubrics or otherwise.

  • In what contexts do you think it would be appropriate to use a behaviorist approach to teaching?

For EFL, it can be useful for learning new vocabulary and expressions. The common wisdom and research in the field show that repeated exposure is necessary for acquisition, at higher rates for active vocabulary than passive vocabulary (CITATIONS REQUIRED HERE). To achieve this in a situation where learners have little exposure to the target language outside of the classroom environment, repetition has been used to assist learners in their quest for greater vocabulary (and other language “chunks”) to aid in self-expression and fluency.

  • In what contexts do you think it would be inappropriate to use a behaviorist approach to teaching?

I believe that there are many instances where behaviorist approaches are inappropriate. It may be easier to approach answer the previous question than this one. In particular, where students need to engage in critical thought and need to be able to process information, to learn by “rote memorization” does not allow for internalization in such a way as is conducive to allowing space for failure and growth. That space for failure- and indeed, the expectation of failure- is foundational to scientific inquiry. In any area where students need to explore possibilities and reject incorrect ideas or outcomes, Behaviorism does not assist in this style of learning.

As a final note, I found the necessity of student activity (for reinforcement) to be interesting. If the student does not act, what is the teacher to do, from a behaviorist perspective? Is this a failure of the teacher to create a successful environment (one that elicits the desired behavior) or failure of the student (for not acting)?

References

Buus, L. (2012). Scaffolding Teachers Integrate Social Media Into a Problem-Based Learning Approach? The Electronic Journal of e-Learning, 10(1), 13–22.

Implementing Peer Instruction to the EFL Classroom

I’m working out some ideas on how to use Peer Instruction with Socrative for grammar or other direct-instruction aspects of the course. At this point it’s mostly notes to myself, if they’re useful to you or you wish to comment on them, have away at it.

One option:

  • Flipped Learning: Put the grammar lesson in a video for students to watch at home.
  • Have students complete a couple of questions for homework/reinforcement.
  • In class, start with a “quiz” (formative assessment) where students use Socrative to choose the best answer (or submit short answers) to a given question key to understanding the grammar lesson. As in PI, students discuss discrepancies for one or two minutes and then vote again.

 

To learn or to just get through

I’m having a hard time with motivation. I want to learn something, I don’t feel like I’m learning much, and as a result, I don’t care much. I’m trying to find a way to learn in my assignments and get something out of this process, but it’s really just slowing me down. I’ve gotten to the point where I just have to get through the work, and unfortunately, I’ll just have to do the learning later.

CN: Software Review Comparison

Assignment:

Find sites on the Web that reviewed one of the software programs you assessed; try to find more than one review of the software, but no more than three. You only need to do this for one of your two software programs. If you cannot find a formal review, you may use a testimonial or possibly contact a proponent or vendor of the program.
Compare the reviews you found with your own assessment of the program. In particular, look at these areas in your comparison:
1. Was the scope of your assessment similar to the online review? Does the online review cover criteria that you also considered important?
2. Does the reviewer agree with your findings? In what ways are your conclusions different?
3. In a more general way, comment on whether you think software reviews are useful for educators, and if most reviews were unbiased?

Submission:

Introduction

As Audacity is a popular piece of free software (“freeware”), reviews abound. However, these reviews were seldom from the perspective of or helpful for teachers trying to assess this software for the classroom. One review (Aydin, N., 2008) seemed to misunderstand or have mis-identified the software, attributing to it dictation or speech-to-text capabilities, which Audacity does not have. A large number of teachers writing about Audacity are describing how to use the software in education, encouraging other teachers to use it, or both. Only one in-depth review could be obtained for the purposes of this assignment.

Review Specifics

Universal Design for Instruction (UDI) at the University of Connecticut. UDI had the most comprehensive review of Audacity, including an end-user survey of both instructors and students. The questions for the instructors were as follows (where “e-tool” refers to “Audacity”); N=5:

  • The e-tool was easy to incorporate into my course (likert)
  • I used the e-tool to address the following: (check all that apply)
    • stating and explaining course requirements
    • reducing the physical demands on the learner allowing maximum attention to learning
    • facilitating communication with and between students
    • allowing students to demonstrate understanding or mastery of content
    • other
  • Do you think the e-tool exemplified the construct of UDI within your course? (UDI is defined as an approach to teaching that consists of the proactive design and use of inclusive instruction) (likert)
  • Please comment on the benefits of using the e-tool in your course (open-ended)
  • Please comment on any drawbacks of using the e-tool in your course (open-ended)
  • I will use the e-tool in another course (yes/no)
  • In the future, I will use the e-tool to address the following (check all that apply):
    • stating and explaining course requirements
    • reducing the physical demands on the learner allowing maximum attention to learning
    • facilitating communication with and between students
    • allowing students to demonstrate understanding or mastery of content
    • other

Similarly, the survey of student users was as follows (N=17, 2 with learning disabilities):

  • The tool was easy to use. (likert)
  • The tool assisted me with the following (select all that apply):
    • understanding course requirements
    • reducing the physical demands on the learner allowing maximum attention to learning
    • facilitating communication with other students in the class
    • allow me to demonstrate my understanding or mastery of the content

This review does not address most of the pre-pilot evaluation included in our assessment, but rather focuses on the results of the pilot run of the software in classes for the purpose of assessing its applicability. It is difficult to determine whether the scope of the assessment was as in-depth as ours, but based on the details of the survey, I would hazard to guess that prior to use the in the classroom, some person or group of persons assessed the technical and practical issues we attempted to cover in our assessment. As such, the primary evaluation (sections 2-5 in our assessment) are absent from this review.

Overall, this review agrees with our assessment, with some interesting open-ended responses in the “drawbacks” question. One instructor, who self-identified as lacking proficiency with software found file conversion (from .wav to .mpeg) to be difficult. This may be an important concern for implementation in the classroom. Another commented on a bug, while a third commented on file size presenting problems for attaching to email; a problem that could be solved through PD, accessing online tutorials or other user education. Given that 85.7% of faculty users found the software easy to use, these comments may have come from outlier responses. Likewise, as 100% of survey respondents indicated they would use the software again, this did not appear to have a lasting negative effect on the instructors’ perceptions of the appropriateness of the software. There are no clear differences between online review and our review of this software.

Emory University: Center for Interactive Teaching. This review is much less in-depth than the UDI UConn assessment, but was selected for its attention to education and implementation. The assessment is directed at faculty considering new software for implementation in the classroom and other education settings. It is hard to determine from the summary review how much research was put into this assessment. The primary areas of concern are as follows:

  • How is Audacity applied in learning environments?
  • What challenges are involved in using Audacity?
  • What are some of the benefits to using Audacity?
  • What is the required skill level (Beginner, Intermediate, Advanced) for using Audacity?
  • As a faculty member, why would I use and spend time to learn Audacity?
  • How have others used Audacity at Emory?

This summary assessment is focused on faculty at Emory University and gives a brief overview of how and why faculty might consider using Audacity for their classes. It would be nice if this overview included a link to a more in-depth list of information or considerations for faculty that may have a deeper technical knowledge. Overall, the assessment agrees with ours with no points of disagreement. Of interest, however, is the list of challenges. Here, the Emory reviewer comments that there is limited support when issues arise. While it is true that the user might not be able to call someone to walk them through the process, the online community, based on our review is active and responsive to users who require assistance. A key for new users would be understanding where to source technical assistance.

Conclusion

Considering the nature and content of the reviews examined for this assignment, it appears that software reviews can be helpful for the end-user teacher who is seeking new solutions to existing pedagogical problems or new ways of using software for teaching. The reviews online tend to be short and direct in either recommending for or against a specific title. In particular, many online assessments direct teachers in specific ways they can use the software to meet their educational goals. Access to more in-depth reviews and information for the interested instructor would be an improvement over the current information, which is rather shallow.

As many of the reviews appeared to be from end-users themselves, without stock or interest in the success or profitability of the software, I believe it is safe to consider these reviews unbiased.

 

 

References

Audacity (n.d.). Emory’s Center for Interactive Teaching, Office of Information Technology. Emory University. Retrieved from: http://ecit.emory.edu/teaching_tools/audacity.html

Audacity. (n.d.). Universal Design for Instruction (UDI) at the University of Connecticut. Retrieved from http://www.udi.uconn.edu/index.php?q=content/audacity

Aydin, N., (2008, November 25). Technology Assessment: Audacity. Retrieved from http://voices.yahoo.com/technology-assessment-audacity-2193624.html?cat=15

CN: My Vision for Education

Assignment:

The first step in preparing a needs assessment is to define your educational vision. Use your assignments from Unit 1 and consider all the relevant factors to create an educational technology vision for your classroom. (Remember that this vision statement should focus on education … technology is merely a way to improve the educational process. This being said, your vision is for a classroom of the 21st century, and therefore should reflect the role of technology that you envision).

Resources:

http://en.wikibooks.org/wiki/Technology_Planning/Vision

http://ictpd.net/techplan/

McNabb, M., Valdez, G., Nowakowski, J., and Hawkes, M., (1999) Technology Connections For School Improvement: Planners’ Handbook. North Central Regional Educational Laboratory U.S. Department of Education.

Notes:

Technology Planning: The Educator’s Guide:

  • should be forward-looking
  • should bridge present and future
  • should be the impetus for change

 

……

Vision Statement:

To create a holistic learning environment with student-centered learning approaches in which multiple learning styles are both recognized and accommodated. This holistic environment will address academic, mental, and physical growth and development. Students will be prepared for employment, citizenship and life-long health and wellness. This will be realized through partnerships with parents and the larger community employing appropriate technology, internships and other resources to benefit and enhance student outcomes.

CN: Week 1 Reading & Response

The assignment:

Read the following article: Evaluating Instructional Software. The article considers three general data sources for software evaluation: (1) experimental study, (2) expert opinion, and (3) user surveys.

Answer the following questions in the your Group’s discussion topic area.

  1. Of the three data sources for software evaluation, which do you think is best and why?
  2. Would this overview of software evaluation be appropriate for tool/application software like word processors or spreadsheets?
  3. Which approach–constructivist or directed–is more in keeping with this author’s look at evaluation? The directed versus constructivist methods’ page contains more information on these philosophies.
  4. Since we will be concentrating on expert opinion style of software evaluation, discuss what types of criteria you think would be necessary to look for in designing a software evaluation model?

 

Notes from the Article:

To evaluate software for educational use, there are three primary methods: experiments (was it effective?), expert opinions and user surveys. Utility of evaluation method is determined by the information it yields and its applicability.

  • Experiments:
    • most convincing, but time and cost prohibit widespread employment of this method.
    • previous experiments chalk up differences in outcomes to differences in teaching style rather than the software
    • “Effectiveness is operationally defined in terms of student achievement. This style of inquiry is referred to as a “black box” study. How a student learned or how to use the software is not addressed.”
  •  Expert Opinions:
    • Experts have lists based on their experiences
    • Failure to involve learners in the evaluation process results in a subjective opinion by the reviewer.
    • Conclusion: Fails to take into account student outcomes unless students are intentionally included.
  • User surveys
    • Interpretive Evaluation: performed by observing user interaction with the software
    • Reiser and Dick (1990) created a method that involved the learner, and found results different from the subjective (expert) results. Two years later  they published modified results that attempted to answer the question: “Do students learn the skills that the program is designed to teach?” Data sources include “pretests, immediate and delayed posttest, and learner attitude surveys”. The revision uses a representative student from low, average, and high achieving sectors of the student body. Software efficacy is a function of the delayed post-test outcomes. They conclude that teachers use criteria other than student achievement for their evaluations, and that the time and cost of in-class evaluation is worthwhile.
    • Anderson and Chen (1997) use an econometric model, focusing on user evaluation of features.
    • Castellan’s (1993) method is described as being thorough and advantageous to the reflective practitioner; but largely impractical for implementation.

Author’s Conclusion:

“Software evaluation should be conducted by the instructor. The evaluation method used should yield information about quality, effectiveness, and instructional use” (p. 6)

Answers to the questions:

Of the three data sources for software evaluation, which do you think is best and why?

It’s hard to answer this question without considering what qualities would describe “the best.” If concerned with educational outcomes, then the Reiser and Dick’s modified method (using information from user results) might be best.

Upon examining the study by Zahner, Reiser, Dick and Gill (1992), and considering the findings listed for the Experiment section of this paper, I wonder if some of the problems with retention and thus the appropriateness of the software had to do with teaching methods and student learning styles. It appears to have used a direct method of instruction, and there was no discussion of audio, visual or other types of communication which would serve learners who were not primarily linguistic learners. I am curious to know whether software with different instruction methods, perhaps one with problem-based instruction, would have performed better.

Would this overview of software evaluation be appropriate for tool/application software like word processors or spreadsheets?

I think this depends. I think tools such as word processors and spreadsheets are seen primarily as tools for organizing and sharing information whereas the software discussed in this paper were used for delivering instruction. I would argue that both need to be considered for their educational purpose and efficacy. In some cases, some educators may argue that learning to use software such as word processors and spreadsheets may be a purpose unto themselves as a preparation for life both in and beyond the classroom as such tools are prevalent in most offices around the world.

Which approach–constructivist or directed–is more in keeping with this author’s look at evaluation? The directed versus constructivist methods’ page contains more information on these philosophies.

The article itself doesn’t lend to many direct statements about pedagogy, but given that the focus of the articles I could find making no mention of contemporary, student-based pedagogies such as problem-based learning, and the description of the software used in Zahner et al, I would surmise that most of the pedagogies here have been direct-instruction based. The attention to standards and criteria alone, however, do not give adequate information as to the pedagogy involved as either a direct-instruction or constructivist pedagogy may be employed to the same ends.

Since we will be concentrating on expert opinion style of software evaluation, discuss what types of criteria you think would be necessary to look for in designing a software evaluation model?

Criteria should likely include not only hardware and software concerns, but attention to sound pedagogy and outcomes in concert with curricular design.

 

Reiser, R. A., & Dick, W. (1990). Evaluating instructional software. Educational Technology Research and Development, 38 (3), 43-50.