Nov-9-2003 7:41 PM Message 1 of 11 Go to >
Subject:
Discussion Part Three: How to Test the Effectiveness of the Models
From: DJRosen
In this part of the
discussion we hope to focus on how we would know if a model were effective, how
to test its effectiveness.
Nov-11-2003 5:58 PM Message 2 of 11 Go to >
Subject:
RE: Discussion Part Three: How to Test the Effectiveness of the
Models
From: mariedoerner
Reply to: DJRosen
[ Message 1 ]
I guess I will start
with the obvious or at least what is obvious in our program. We look at three
areas; number of GED passed or in progress; number of accomodations for folks
with disabilities and retention. Since we have been doing this, our program has
become more focused and more people are graduating overall.
Our ultimate measure is whether people pass the GED after being in
our program. Since we really began monitoring our GEDs, our numbers have risen
tremendously. The rise in GEDs is our measure, but you have to be careful.
Adult education is not like K-12 where we have everyone start at 5 years old
and end at 18. I think you need to look at what we can do to increase the
number over the long haul.
Second, since we work with learning disabilities here in San
Diego, we have been maintaining about 15% of our GEDs taken with
accommodations. This is measuring that we are serving those in our population
who need these services.
Third, retention. Students come if the program is working. They
vote with their feet.
Nov-11-2003
9:44 PM Message 3 of 11 Go to >
Subject:
RE: Discussion Part Three: How to Test the Effectiveness of the
Models
From: DJRosen
Reply to: mariedoerner
[ Message 2 ]
Marie,
This is very helpful. Thank you. I have a few observations and questions.
GED and retention, I agree, are obvious measures of program
effectiveness. I wonder, however, if you could write more about "number of
accomodations for folks with disabilities." I am surprised (and delighted)
to see that this is included in your program. I have not seen this as a measure
of program effectiveness before. Tell us, if you can, how this evolved in your
program, if it is supported by incentives or structures outside your program
(in San Diego or in California,) and why it is a measure of program
effectiveness.
I think I understand what you mean by "I think you need to
look at what we can do to increase the number over the long haul" but
could you spell this out? Do you mean a way of counting GED diplomas for those
who move in and out of your program (or programs in San Diego) over several
years, and who eventually get their GED diploma? Are you saying that testing
effectiveness of a model needs to account for this (common) learner
participation pattern as well as for those who get their GED within a year?
David J. Rosen
Nov-12-2003 11:41 AM Message 4 of 11 Go to >
Subject:
RE: Discussion Part Three: How to Test the Effectiveness of the
Models
From: BOP6751
Reply to: DJRosen
[ Message 3 ]
The Bureau of Prisons
launched a GED accommodations initiative three years ago. Presently we monitor
(1) the number of referrals and (2) those that are approved for one or more
test accommodation. What we have not been able to do yet is track the number of
successful test completers (who used accommodations) but that's the next step.
Since FY 2001, with good support from Bureau psychologists and special
education teachers, the number of referrals has tripled each year. In 2003
approx. 130 referrals were submitted, and about 75 percent were approved.
-Bill Muth
Nov-16-2003
8:31 PM Message 5 of 11 Go to >
Subject:
RE: Discussion Part Three: How to Test the Effectiveness of the
Models
From: mariedoerner
Reply to: BOP6751
[ Message 4 ]
Ahh!!!! Gettting GED
accommodations approved that is a research project and art form. We have worked
really hard to address this issue and it still remains a difficult part of our
process. As for why we care about how many students are served in our GED labs.
First, our adult education (continuing ed) is part of the community college
system not K-12 so we have mandated money for positions to help those with
disabilities, but it is a voluntary service. Meaning students must volunteer
and say "I want services." Our department saw a need for support for
students in our GED/High School program so I was hired using
"Matriculation Funding." We started a resource program in our school
which has been very successful. We have had over 50 people with disabilities
pass the GED. However, my boss is always looking to see where we, as staff, are
placed where we would be best used. In order to keep me where I am, I keep
numbers flowing across her desk. So the short answer is that I validate that I
am doing my job by looking at numbers.
It also reminds teachers to keep disabilities in mind. Last
semester, one class had 28 people with disabilities attending during the
semester, this semester we only have 18. Since I know the numbers, I tell her
what the status is and she refers more people. It keeps our door revolving.
As to the qquestion about "the long haul." The people
with disabilities passing the ged seems to go in waves. I work mostly at two
schools. One has 8-10 people who are in the process of taking GED exams. The
other school went through that phase a year ago and now we have students who
are another 6 months or so away from the GED. The year the gED changed, we had
20 or so people pass, but then it took a while to build up momentum again.
Right now we also have people for whom the GED is not a realistic goal. We try
to help them get their basic skills to a point where they can take vocational
classes and get a job. At the end of each year, we have a tally of how many
students passed the GED and a general idea of how many students are headed in
other directions. This means that you have to step back and keep count of how
many people passed the ged, how many people are in the process and how many
people do we serve who are attaining other goals. The GED is the best number to
keep because my dean and her bosses really respect that number.
Nov-21-2003
11:32 AM Message 6 of 11 Go to >
Subject:
RE: Discussion Part Three: How to Test the Effectiveness of the
Models
From: comingsj
Reply to: DJRosen
[ Message 1 ]
A specific example
might help focus the discussion. One subpopulation might be ASE/GED students,
whose reading skills are sufficient to pass the requirements of their ASE
program or the GED and whose goal is a job with good pay and benefits.
Research suggests this goal can only be met if students are
encouraged to build their skills and knowledge so that they can attain a higher
score (GED score predicts income) and helped to transition into postsecondary
education or training programs (adults with two years of postsecondary
education or training are more likely to attain good pay and benefits).
A program model that would help students in this subgroup reach
their goals might:
Intake and Orientation: help students understand the steps
necessary to reach their goal and work with them to develop a learning plan
that fits with the demands of their lives. That learning plan would outline the
instructional and support services they need to be successful.
Instruction: Most ASE/GED classes are a combination of indivudual
study, small group work, and whole class instruction. This approach developed
as a response to mixed enrollment and open entry/open exit classes. With
well-defined curriculum goals (the ASE requirements or the GED test) and an
acceptance of the needs of this subpopulation that lead to mixed enrollment and
open entry/open exit classes, development of an effective model of instruction
that draws on the experience of good teachers should not be difficult. Adding a
self-study component that utilizes the recent development of technology
applications should not be difficult either.
Since these students will have post-secondary education as part of
their goal, this curriculum would have to be augmented with content (such as
academic vocabulary) that would provide the skills and knowledge needed to be
successful in postsecondary education and training.
Support Services: ASE/GED programs usually cannot provide direct
support services (such as day care) but they can provide counseling and
assistance to students so that they can arrange those services for themselves
or find ways around these life barriers to participation. Programs can help
students develop the emotional support they need by establishing a program
climate and procedures that build support among students and between staff and
students.
Transition: Programs that help ASE/GED students make the
transition to postsecondary education do focus on skills and knowledge, but
they also focus on establishing a support structure for students once they
enter postsecondary education. The building of this support structure begins in
the ASE/GED program but should continue after the transition.
Cost: A program like this might cost $1500 to $2500 per student.
Evaluation: An evaluation of a program like this would have to
include a longitudinal component that followed students for 3-5 years. A random
assignment experiment or a treatment and comparison group study could look at
this model vs the existing model that costs quite a bit less. I believe the
evaluation would show that this more expensive model is cost-effective.
With this data in hand, our field could argue for policies that
support this kind of program for this population. If someone comes up with a
good idea that might improve this model (a new math curriculum for example) or
make it less expensive (a new technology tool for example), it could be tested
against this accepted model.
Other subpopulations (adults with very low literacy skills or
those seeking education to help their children in school for example) may be
more difficult to fit into this approach, but starting with the easiest
subpopulations might make model evaluation for these subpopulations easier.
Do you think this would work? How might you do it differently?
Nov-25-2003
6:30 PM Message 7 of 11 Go to >
Subject:
RE: Discussion Part Three: How to Test the Effectiveness of the
Models
From: JanetIsserlis
Reply to: comingsj
[ Message 6 ]
John and all,
I've copied out pieces of your post, and added my comments/questions
within
[j.c.]Intake and Orientation: help students understand the steps
necessary to reach their goal and work with them to develop a learning plan
that fits with the demands of their lives. That learning plan would outline the
instructional and support services they need to be successful.
[j.i.] It seems that programs do this first step; some more
formally than others, and/or in an ongoing manner.
[j.c.] Instruction: Most ASE/GED classes are a combination of
indivudual study, small group work, and whole class instruction. This approach
developed as a response to mixed enrollment and open entry/open exit classes.
With well-defined curriculum goals (the ASE requirements or the GED test) and
an acceptance of the needs of this subpopulation that lead to mixed enrollment
and open entry/open exit classes, development of an effective model of
instruction that draws on the experience of good teachers should not be
difficult. Adding a self-study component that utilizes the recent development
of technology applications shouldnot be difficult either.
Since these students will have post-secondary education as part of
their goal, this curriculum would have to be augmented with content (such as
academic vocabulary) that would provide the skills and knowledge needed to be
successful in postsecondary education and training.
[j.i.] - again, how is this different?
The model you describe seems to encompass what good programs
already do - some to a greater, some to a lesser extent.
I wonder if this list is quiet because of the pending holiday, or
if some of us are just feeling tired beyond measure because all you propose
here makes sense, but the funds aren't in place to implement and evaluate
programs in the way you suggest, or the support for professional development
and the respect for the fact that teachers do learn and want to learn is absent
in many adult learning programs.
Most of us don't have the wherewithal to develop the kind of
evaluation plan you describe and it seems that for YEARS we've had some tools
for advocacy that have nonetheless been ignored by those who have found, say 87
billion reasons to discount our work. This doesn't mean that we don't try to
evaluate our work, and look at evidence, but how much evidence is enough?
Lest this turn into a total rant, I think what I'm really trying
to articulate is the fact that yes, you've proposed a fine model; yes, programs
likely incorporate many of the elements you describe - if not all of the pieces
you propose, but no, we can't seem to bridge the gap between the kinds of
evaluation and analysis that Big Research seems to demand of us, while ongoing
small and participatory measures tell us that people are learning.
I don't mean to set up a binary, either/or argument here, but am
frankly unsure how to move beyond the rhetoric of what counts as evidence and
how to use that evidence to support learning and teaching.
respectfully,
Janet Isserlis
Nov-26-2003 10:23 AM Message 8 of 11 Go to >
Subject:
RE: Discussion Part Three: How to Test the Effectiveness of the
Models
From: comingsj
Reply to: JanetIsserlis
[ Message 7 ]
Thanks. Yes this is
what good programs look like and I'm suggesting that we do the "big
evaluation." Once that evaluation is done and identifies impact, which I'm
confident it would, then the field would have the evidence it needs to justify
funding for this subgroup and accountablity could move from measuring outcomes
to measuring whether or not programs conform to this model of good practice.
Future research for this subgroup would take place within programs that conform
to this model.
Is there more to say about how to refine the model?
Or, maybe it is time to move ahead and see what happens.
I hope everyone has a happy Thanksgiving
Nov-26-2003 10:27 AM Message 9 of 11 Go to >
Subject:
RE: Discussion Part Three: How to Test the Effectiveness of the
Models
From: bonniesophia
Reply to: JanetIsserlis
[ Message 7 ]
While I understand
that the most obvious way of measuring a program's effectiveness is to document
that which would be easiest to measure (GED completion, transition to
post-secondary education), I am concerned that there is a kind of triage
operating here. And inevitably so, perhaps: as mandated programs need to
increase positive outcomes they would tend to focus on those students whose
success is most likely (i.e. those already with learning proficiencies). The
challenge is of course to develop ABE students' intrinsic learning motivation.
I believe that standardized questionnaires/checklists to determine not just
goals, but learning needs (EFF has something like this, I believe). Develop
Intake that incorporates not just assessment but interviews, and develop
emergent curriculum and modify existing curriculum based on these findings.
Implement the Teaching/Learning cycle, but keeping in mind that skills content
is imperative. My program is successful in integrating technology into basic
skills learning, a kind of reinforcement of learning modalities. Documenting
success would involve (as CT does) matching student goals with outcomes;
documenting which level learners have the most increase in both reading and
math; attendance: frequency and total hours, and attrition. We are also
successful in retaining students with learning disabilities and some mild
mental health issues. I have to admit that I don't do all of this now, simply
because students do become annoyed with a lot of up-front assessment. And our
attrition is most likely based on not having accurately identified the kinds of
students who would do well in our program, or not successfully determining
appropriate levels and activities that would facilitate mulri-level, small
group and individualized learning. Integrating some of the intake questions
into "Why? and "Why now?" and a kind of
literacy/language/learning inventory into active goal setting and refining
might give a much more clear picture than CASAS competencies checklists and
test score performance indicators.
Bonnie Odiorne, Ph.D.
Working Smart
Waterbury, CT
Nov-30-2003 3:21 PM Message 10 of 11 Go to >
Subject:
RE: Discussion Part Three: How to Test the Effectiveness of the
Models
From: ngriffis
Reply to: bonniesophia
[ Message 9 ]
This post should
rightfully be place in Part 2. I post it here so that it will better come to
people's attention.
Students By Grouping Within My Lab:
Advanced Short Timers (Those student soon to get their GED) (and
no, I do not run a prison based system although some of the teenage students
might disagree)
Elderly
Post ESOL Students
Working students whose job mandates a GED
Learning Impaired Students
Teenagers
Court and Welfare Mandated Students
Pre-literate Students
The above groups are listed by most able to sustain attendance, as
well as most able to participate with an active mind. It is one thing to be in
the seat and quite another to be working at the curriculum. Please note that I
refer all pre-literate students to another program.
These groups can also crossover into each other. Elderly students
very often have some learning or memory impairment. Teenagers can be part of
the court-mandated group. You get the idea.
The course that students enroll in is titled, ÒGEDÓ. They come in
our door because our society puts economic and social emphasis on the
acquisition of this document. Whatever life skills, leadership skills, and
learning moments that come from enrollment is fine with my students, just so
long as I help them to acquire their GED. The GED to them means job opportunity
and greater pay. The GED diploma is the sole reason why most of my students are
in class.
Which model and how to implement it?
I believe teachers put together their own set of tools that they
have picked up along the way. They use what they perceive works for them and
their students. Teachers are the soldiers in the trenches. There will be no
ability to help them if there is a disconnect between the learned scientists
and the people who have to carry these models out in reality. That is why a new
model must focus to make the teacherÕs job easier not harder.
The ideal model would not only focus on an academic best practices
curriculum but on administrative tools so that teachers can teach more and
bureaucratically administer less. Adult Education has its own peculiarities. These
pose sometimes-unmanageable administrative busy work. I have worked hard to
over come some of these and would be happy to post the solutions that I have
inculcated into my model as well as my Christmas list of imagined tools and
templates.
Teachers for the most part teach in isolation. They have no time
to plug into anything else other than what is on their plate. They all are
recreating the wheel as best they can. That is why a forum such as this is so
important a first step to bringing Best Practices to teachers.
I would like to see the teachers of this group post, in whatever
file format is possible, their Best Practices Curriculum and Administration
modules, their notes, their papers, whatever special knowledge they consider as
their best practices. Let us see what teachers perceive works for them so that
we, as a forum, can come to an agreement as to what some of the elements of our
best model would look like. David could tell us how best to go about posting
and reading this input.
My own desire is to build a model that creates a Fast Track to the
GED for as many of my students as possible. I am trying to put together a
package that would effectively and efficiently bring students to their GED. In
short, I would teach to the test so that my students could get what they came
for with the least amount of time and effort on their part. Along that path, I
would add tools students could use in their life to be more capable. The two
goals are not necessarily exclusive.
It is time to answer the ladyÕs question of ÒWhereÕs the beef?Ó
Nick Griffis
Inlet Grove HS
Adult Education
Riviera Beach FL.
ngriffis@bellsouth.net
Dec-2-2003 5:43 PM Message 11 of 11 Go to >
Subject:
RE: Discussion Part Three: How to Test the Effectiveness of the
Models
From: mariedoerner
Reply to: ngriffis
[ Message 10 ]
I agree that any
evaluation or research effort needs to serve the needs of teachers. In light of
this, I would ask for a clean, easy way to measure student achievement. From
all the teachers that I have spoken to, this seems to be very difficult. It is
such a problem that most teachers have given up on formally testing their
students more than mandated.
What to do about this? Where to start?
It is fun working with GED students because you know you have
succeeded when they pass the test. Also they have practice GED tests that help
us focus our instruction and know when the student is ready to test. It would
be great to have something similar with measuring ABE students learning. The
available tests seem so broad that they do not show small steps in learning.
The other problem is curriculum based tests are too narrow. They measure what
the student is doing now, but not the cumulutive effort.
We have a basic "Math Placement Test." My proposal is to
make 10 forms of this and on the first of the month give very similar tests to
students to see if we can measure progress.
In a dream world, I would like something online that would measure
students monthly both in reading and math. It would be great if it had a report
on what needed to improve and what improvements had happened.