Oct-14-2003     7:50 AM Message 1 of 55 Go to >

Subject:

Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Discussion Part One: How to Improve the Process for Developing an Evidence-based Adult Education System

Tuesday, October 14 – Friday, October 24

There are two questions which we hope you will address in the next ten days. Let's start with the first one:

1. How could the process for developing the system, set out on pages 10-15 (Building an Evidence-based Adult Education System, Baseline Program Models for Adult Education, Defining, Testing, and Using the Baseline Program Models, and Establishing an Evidence-based Education System), be improved?

Please review this section of the paper and, to add to the discussion, choose reply. You may wish to _preview_ your message, and edit it before you post it. Messages posted in this discussion are part of a public dialogue. Some may be quoted or summarized at the end of the discussion and made available for a broader adult education audience to comment upon.

Please note that each question we post will have its own folder (or thread) so please be sure you reply within that thread. A new folder will be opened for each question as the discussion progresses.

Thank you for joining and participating.

David J. Rosen

EBAE Discussion Moderator

Oct-14-2003     7:53 AM Message 2 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Reply to: DJRosen    [   Message 1  ]

Someone will need to help by posting the first discussion message, breaking the virtual ice, so to speak. We will all be grateful to you.

: - )

David J. Rosen

EBAE Discussion work Group Moderator

Oct-14-2003     8:47 AM Message 3 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: andresmuro

Reply to: DJRosen    [   Message 2  ]

David J. Rosen

EBAE Discussion work Group Moderator wrote:

"Someone will need to help by posting the first discussion message, breaking the virtual ice, so to speak. We will all be grateful to you.

: - )"

You know, I am kind of shy about doing this, however, here I go. In general, when we talk about evidence we refer to improvements in language communication skills measured in some standardized form.

A while back Mary Beth Bingam and Olga Ebert from U. of Ten. where working on a paper on Literacy and quality of life. I liked their papaer and became very interested in this concept. Is there a relationship and can it be measured?

related to this and the idea that literacy is relative, Tom Stich always brings the point that while the NALS suggests that 90 million Americans lack literacy skills, a lot of these don't feel that they lack anything. If their literacy meets ther needs to have a reasonable quality of life, then they feel that they are ok. Only when their environments change they feel a lack.

When I make health literacy presentations I always talk about a woman with third grade education who walks to the grocery store, buys all the ingredients necessary to prepare a several course meal for her entire family, goes home and prepares a healthy meal in no time. On the other hand, it is possible that a male Ph.D cannot do this task and orders pizza instead. The question is, who is more literate and what evidence do we have that one is more literate than the other.

Their literacy skills change, in the eyes of society, as the rules change, but are tied to economics. For example, with changes in the health fields, the time spent talking to a pharmacist, doctor, nurse, health worker, have been reduced. So patients are expected more and more to fill out forms and understand written instructions from doctors, pharmacists, etc. So, while people may be much more literate today than 50 years ago, in that they can decode more morphemes, the literacy expectations are greater.

I just woke up, it is early, and I have to go catch a plane, so these comments/questions are not elaborated very carefully, but I hope that they contribute towards starting the discussion,

Andres

  Oct-14-2003     8:56 AM Message 4 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Reply to: andresmuro    [   Message 3  ]

Thanks, Andres, for getting the ball rolling.

I have a request for you -- and for others who reply. Please try to focus your comments now on the specific question -- how to improve the process for developing an evidence-based adult education system. There will be plenty of opportunity later, both here -- and in an e-list ("listserv") that I will create in November for a broader audience discussion of evidence-based adult education.

Oct-14-2003     9:54 AM Message 5 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: gvaughn

Reply to: DJRosen    [   Message 1  ]

I think that it would be a good idea to consider the adult age group when we look at assessing the needs and goals of participants entering into the program. For example is the motivation work related?

Oct-14-2003     12:47 PM Message 6 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: joadurkee

Reply to: gvaughn    [   Message 5  ]

First of all, thank you for this opportunity. In addition to participating in an important dialogue, I appreciate obtaining more experience with virtual discussions.

I just have a question at this point. The introduction of Comings paper makes reference to ASE programs, yet the next steps refer to GED students only. A significant part of our student population are ASE students who are working towards earning a high school diploma. Can they/should they be included?

Oct-14-2003     2:03 PM Message 7 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: comingsj

Reply to: joadurkee    [   Message 6  ]

We meant all ASE students, not just GED. Thank you for catching that.

Oct-14-2003     2:26 PM Message 8 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: joadurkee

Reply to: comingsj    [   Message 7  ]

Thank you for the clarification about GED and ASE students. Given this, the second bullet under Next Steps should include CASAS as a specific tool for measuring achievement outcomes.

The third bullet talks about choosing groups to begin with. How is "groups" defined here? Is it referring to choosing a few from the list of six in the first bullet?

While it may seem off topic, I have a nagging concern. So many programs are designed around funding constraints and limitations due to how students make themselves available to programs. In one classroom, the range of student needs, academic levels, etc. is often quite broad, resulting in a less than ideal instructional delivery model. How would this impact the selection of 'groups'.

How do you see this system interfacing with what is being done by Equipped For the Future?

Oct-14-2003     2:52 PM Message 9 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: JanetIsserlis

Reply to: joadurkee    [   Message 8  ]

Being mindful of the question at hand - how to build an evidence-based system, I want to raise a question about the use of medical models/examples within the current paper. While I understand that medicine is useful in understanding evidence-based research, it is also troubling that it appears to be just about the only other discipline discussed in explicating what "evidence-based" means. As we continue to work on the content that is relevant to adult education, I hope we can find other fields from which to draw upon in framing examples and models for our own use.

I'm also curious about connections to EFF as a means of constructing an evidence-based system that allows for a broad range of content and approach.

Janet Isserlis

Oct-15-2003     7:11 AM Message 10 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Reply to: JanetIsserlis    [   Message 9  ]

Janet and others,

Imagine that it's five years from now and that we have what you would consider an exemplary evidence-based adult education system, one that includes baseline program models for adult education, defining, testing, and using the baseline program models. a system built. At a concrete level, what would this look like where you live and work, and in other parts of the country? What would be happening? What would be different from what we have now?

Oct-15-2003     8:22 AM Message 11 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: pmcguire

Reply to: DJRosen    [   Message 10  ]

Hi all. I am, in general, really pleased by the careful and thoughtful process being set out in this paper for building an evidence-based system. I am especially pleased by the call, repeated in several ways, for constructive inquiry, dialogue and consensus-building among the multiple stakeholders in such a system. Having said that, there is at least one area where I don't see that call, and it concerns me. The process calls for stakeholders to define the expected results of program improvement (persistence, achievement, impact), but where do these folks get to come to agreement on a working definition of "achievement"? I guess I was hoping to see a step in the process that allowed for discussion and critique of basing definitions of achievement for adults primarily on scores from currently available off-the-shelf standardized tests, or to frame adult achievement by K-12 grade levels.

Seems to me that our broadly-agreed-upon definition(s) of achievement would have significant impact on the "Next Steps" put forth in the paper. For instance,

1) If we intend to purposely define groups of students according to a narrow range of needs/goals, are we content to limit our understanding of their needs/goals to test scores and grade levels?

2) Our definition(s) of achievement will, and must, have an impact on our choice of tools to validly/reliably measure that achievement.

So at what point in this process do we get to have what would promise to be an edgy and exciting and very important conversation about appropriately defining and adequately measuring adult achievement?

Thanks for the opportunity to weigh in!

Peggy McGuire

Oct-15-2003     10:50 AM Message 12 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: HSilverP

Reply to: pmcguire    [   Message 11  ]

I share Joanne's concern that we are not structuring enough exploratory research and discussion up front to really dig in and discuss what are the appropriate outcomes we will set up further research to investigate. Scores on achievement tests are not going to capture the breadth of what we do and what our learners do in our programs. Victoria Purcell-Gates's work with NCSALL and Sheehan-Holt and Smith (Reading Research Quarterly, 2000 Vol 35) offer an example of conflicting conclusions on what counts and how to count it. I'd like to see preliminary steps added that include quantitative, exploratory research to address these epistemological issues.

I also echo's Janet's concern that the medical model is being applied here. I understand that this is the language of the DOE's Institute, but there are more differences than similarities in our fields of research and expected outcomes. I worry about the ramifications of trying to adopt a medical model.

Oct-15-2003     11:14 AM Message 13 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: comingsj

Reply to: pmcguire    [   Message 11  ]

This paper is a starting point, not a final plan, and so when we identified the groups, we constructed categories that were big and easy to describe. Once a system like this started, the number of groups could grow. Still, a lot could be learned within the large groups that would be applicable to most if not all subsequent student groups.

This research would make the case that instructional groups in programs should be made up of students who fall into defined categories, rather than multi-level groups. This would require greater funding.

If members of the list think this approach will not work, can they suggest an alternative?

Could anyone suggest a different set of groups?

I think groups 1 (beginning ESOL students literate in their native language) and 3 (ASE students with higher level literacy and math skills) are good groups for the start of this line of research. These students come to our programs in significant numbers, we know a lot about how to serve them, and we have good outcome measures for them.

Which groups do members of the list feel might be best for the beginning of this line of research?

Oct-15-2003     1:53 PM Message 14 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: mariecora

Reply to: comingsj    [   Message 13  ]

Dear everyone, I think that clearly defining the outcomes is a monumental task, even with the existing standards and principles noted in the paper. I believe that this chunk would take quite a lot of time and effort, and I don't see a lot of discussion around this piece in the report.

Defining groups of students to focus on: while I understand that in ABE we now identify ABE levels using SPLs and GLEs, we in assessment in Massachusetts (at any rate) are trying hard to lose this way of labeling advances in student gain, we find it inappropriate. Is this important to the Feds or to funders? I'm unsure, but it is to assessment people in ABE in this state (I assume it might be important to others, and to folks in other states as well). At the same time, I understand that we need to be able to articulate learning gain in ways that are understandable to people not necessarily working in education (hence, using K12 definitions and trying hard to align the content of what adults know/are learning at particular levels). As well, some of the tools for measuring student gain are simply not model themselves, AND they will NOT align with the curricula of a program unless the program aligns itself to that test. So how would that affect the Baseline Program Models? The more interesting models of measuring gain (EFF and family literacy) are not fully developed yet (I think?) and are also not neat packages (as opposed to saying 'you are an spl 4 according to BEST'). So I guess I'm asking how to go forward developing this new system even though components of ABE are flawed, or those flaws are being worked on? It's clear that our lack of cohesive system is a flaw, but I'm concerned that if we try to build a new system, we'll get really hung up on fundamental components that need to be very different but aren't. Can both efforts happen simultaneously?

Finally, I'm curious if the authors have considered some timeframes for such an endeavor? I assume time and timing would be of interest to funders as well. marie cora

Oct-15-2003     6:19 PM Message 15 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: mcorley

Reply to: mariecora    [   Message 14  ]

Kudos to John Comings and the primary working group, as well as to David Rosen, et al., for initiating and facilitating this process.

My comments: Although page 8 of the NCSALL report lists various published sources for principles for each element of the baseline program model, I think that this (defining the baseline program models) is going to be the most difficult piece of the process, as proposed in the NCSALL paper. It seems that the concept of the baseline models is still too fuzzy for most of us to comment on, but it’s a necessary starting point for the entire process.

Sure, we can all review the TESOL program standards and the Kruidenier report, and the other documents listed on page 8, but we won’t all agree with all the principles contained therein. The NCSALL report proposes that “a group of academic and practitioner experts would help identify sources, judge the quality of the evidence, and make decisions about which principles should define the program models.” Perhaps before that happens, someone can begin by proposing one or more elements of one or more baseline models along with some of the defining principles for each element, and then this community discussion group can sink our teeth into a discussion of specific elements and principles? Otherwise, we’re speaking in generalities that many of us will have trouble connecting to. As you can see, I’m having trouble moving beyond the “defining the baseline models.” Perhaps members of the NCSALL working group can help ease our way into such a discussion?

Thanks,

-Mary Ann Corley

Oct-15-2003     9:47 PM Message 16 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: 4128st

Reply to: mcorley    [   Message 15  ]

I feel that models come and go in education. Most do not last long enough for good outcomes.I like the persistance part. I recently talked to a young woman wanting to get her GED and lacked math skills to get it. Mainly doing fractions. I encouraged her to go back and get help and get it done. She needed some confidence and it wasn't there at first try. Do we lack follow up on some adult learners? Do we give them a pill? Not trying to be smart on the medical front. Somewhere we tend to leave kids behind in k12 and it carries over into adult literacy. I used to teach and it always bothered me when students left behind knew it, and it was hard to bring them up to speed. Persistance didn't carry on when the kid went on to the next grade. Perhaps more effort on carry over was needed in some skills and not the old adage I was guilty of "theres not enough time"

Oct-16-2003     11:04 AM Message 17 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: bercovitzL

Reply to: 4128st    [   Message 16  ]

Hello everyone, I'm Laura Bercovitz and my colleague, Laurie Martin is sitting with me. We'd like to comment on a couple of topics that have been discussed.

1) Concerning assessments being used. We feel it would be important to use all assessments that are approved to be used for NRS reporting. If the results of this Evidence-Based Adult Education System are to reflect "the real world" of adult education in the field, then it would seem that all approved assessments need to be included for ABE/ASE and ESL.

2) In setting up the research sites, how will the number of variables be controlled? How many models can one study have?

As stated in the occasional paper, it is recognized that time of day, experience of teachers, curricula being used, number of classroom hours are all separate variables. Are there enough established research sites available to set up classrooms where the number of variables can be limited? Might it be necessary to recruit additional sites so that there are enough classrooms to provide the needed critical mass of data? If so, what might some criteria be to qualify?

3) In the next steps the term "practitioners" is used. Does this term refer to practioners from the research sites or practioners in the field? We are concerned that research site practioners may not reflect real world experiences. Often times their education exceeds that of average adult education instructors and their experience base may differ from the average instructor (e.g., better resources and a support system).

Oct-16-2003     12:51 PM Message 18 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: agopalak

Reply to: bercovitzL    [   Message 17  ]

Good day. Thanks for this opportunity.

I like the idea of baseline program models because they establish minimum program quality that the community deserves.

It appears that the baseline model will be tested on the agreed upon outcomes. The example on page 6 talks about the impact of managed versus open enrollment (the baseline model) on learning gains (the outcome). Should the steps on Page 15 make specific mention about the process of developing those clearly defined outcomes? I see that the Next Steps on page 17 does make reference to agreement on the outcomes.

I, like Laura and Laurie, wonder about the potential disconnect between research sites and the “real world”. For example, an adult education program generally operates within the policies established by the local, state, and federal government. Would being a research site change any of that i.e. “exempt” them from such policies? Can NCSALL provide some example criteria on how research sites could be chosen and how many would be needed?

Thanks.

Ajit Gopalakrishnan

Oct-16-2003     2:51 PM Message 19 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DebbieYoho

Reply to: agopalak    [   Message 18  ]

Hello friends. I have subscribed to this list primarily to keep an eye on the process with admitted heavy skepticism and deep philosophical reservations. My primary concerns come from much observation of the inability of educators to control so many, if not most, of the variables that impact on the effectiveness of our practice, especially in adult programs conducted in a democratic, and competitive, society. However, I am highly interested in a discussion of these variables and how we might seek to control them despite the constraints that John Comings mentioned. To improve the proposed process, I would like to see a thorough listing from practitioners of what those constraints and variables might be. One way to do this might be to use the tried and true SWOT approach--brainstorming Strenths, Weakness, Opportunities and Threats already discernible in the current system. Perhaps someone has already published such an analysis. I would not expect such a listing to be something we would all agree with. But I believe that without such a beginning the proposed process for building a new system might throw out the baby with the bathwater.

Oct-16-2003     3:59 PM Message 20 of 55 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: llwest

Reply to: DJRosen    [   Message 10  ]

Some thoughts in response to David's question

"Imagine that it's five years from now and that we have what you would consider an exemplary evidence-based adult education system, one that includes baseline program models for adult education, defining, testing, and using the baseline program models. a system built. At a concrete level, what would this look like

..."

Since I'm an information specialist as well as an adult educator, I'm concerned with making the proposed evidence-based adult education system available to all stakeholders - policy makers, researchers, and practitioners - so that effective models are utilized in the field not confined to academia.

I envision a web-based data system with the targeted program areas as the organizing principle. Based on the concepts in the occasional paper, the major content in each area would be program model descriptions (including video samples where illustrative) linked to the research reports on which they are based. The continuous improvement cycle part of the system would feature evidence from practitioners about using (evaluating) program models, discussion of the evidence, and modification to the models. I don't envision as much of a separation between "program model evaluation/testing" and "practitioner knowledge to improve implementation" as the diagram on p. 3 implies.

I share the concerns of others about the disconnect between researchers and practitioners and the suspicion that the separation of the two roles is a faulty paradigm. Ideally the researcher should be a classroom teacher, but there is an obvious need for a support system for the teacher-researcher.

In California there are federally funded statewide projects (CASAS and OTAN) that have some experience with action research in the areas of instructional technology and learner retention. Connecting with a national system that includes identifying research goals, selecting appropriate methodology, collecting and analyzing data, and developing and sharing reports would be welcome.

Linda L. West, Managing Director

Outreach and Technical Assistance Network

Sacramento County Office of Education

10474 Mather Blvd

P.O. Box 269003

Sacramento, CA 95826-9003

(800) 894-3113 CA Only

(916) 228-2580 228-2563 fax

http://www.otan

Oct-17-2003     1:12 AM Message 21 of 62

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Reply to: pmcguire    [   Message 11  ]

Peggy, you -- and perhaps others-- have raised concerns about the state of the art in our field of the standardized instruments we use to measure adult learners' achievement.

Do you, and do others, believe we need better assessment instruments as part of an evidence-based adult education system? Is the current state of the art --the range and quality of the assessments -- an obstacle to our doing good research? If so, what kinds of new instruments or improvements in current instruments are needed?

Oct-17-2003     9:16 AM Message 22 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: pmcguire

Reply to: DJRosen    [   Message 21  ]

David (and all): Thanks for asking! Be warned that my response here is a bit long.

I believe that an effective, evidence-based adult education system will result only from collaborative, good-faith efforts to align all components of the system -- what gets taught, what gets learned, and what gets assessed, along with all the ways we need to support quality instruction and assessment -- with what all the stakeholders agree are the most important goals for that system. In my mind, the question of how we define "achievement" is at the very heart of the matter. What is important for adults to achieve through their participation in our programs? What do aduts need/want to know and be able to do in order to meet their goals in their important adult roles, goals for "right-now" as well as for lifelong learning? In a well-aligned, evidence-based system, I think this "content" question will lead naturally to the "assessment" question:Once we agree on what is important to teach and learn, how will we appropriately measure learning of important content? (And, by the way, how will we support teachers in improving delivery of instruction focused on this important content?). Then we can look at currently available standardized tests, identify what content they do in fact measure, and decide how well that matches up with what content we think is important to measure. At that point we may be in a position to adopt currently available tests, and/or change them, and/or develop new ones, so that we end up with an assessment system that actually tells us what we want to know about adult achievement.

I don't believe our current system works this way. I don't think it can -- because we are not working with a consensus about what is important. And since we don't have that consensus, we can't really address the question of whether currently "approved" standardized tests adequately measure what we think is important. We think that each of these tests may give us valid and important information about some aspects of adult learning -- "parts" of what is important, but we know that the tests are based on different sets of assumptions about what is important, as opposed to broad agrement on what's important, so they do not give us the same information. So my concern about defining adult achievement -- and groups of adults, for that matter -- based on the scores from currently available standardized assessments is a concern about misalignment between what we believe is important and what we measure. Given this misalignment, how can we possibly hold learners, teachers and programs "accountable" for the results?

Now let me be transparent about my position here. I've worked with the EFF Assessment Consortium for almost 5 years now; for 15 years before that I taught ABE in a small community-based organization in Philadelphia where I eventually became the Executive Director. I'm incredibly fortunate -- I've gotten to be teacher, administrator, trainer and researcher. And I absolutely believe in the necessity of mutual accountability for results that matter to adults in our system. I believe that the work that EFF has done for almost 10 years now has been, and continues to be, about making mutual accountability possible (and more likely) through the alignment of our system around important and broadly-agreed-upon learner and policy goals. The key to this alignment are the 16 EFF Content Standards -- the expression of a participatory and long-term consensus-building process around what is important for adults to know and be able to do. And now, the EFF Assessment Consortium is a few months away from releasing a handbook on assessment that is aligned with the EFF standards, along with an assessment prototype focused on the EFF standard Read with Understanding that will provide tools, guidelines and resources needed for assessment of the important applied knowledge and strategies that are embodied in the standard. I've been privileged to be a part of the development of these products and to work with some of the best and smartest and most committed people I know to produce them. Perhaps more important, I've come to believe that standards-based educational improvement really makes sense and can work, if it is developed and implemented with integrity by people of good will working together.

This isn't the end, but barely the beginning of development of the system I imagine, David. A critical beginning, though, that just might allow us to define and measure achievement in a way that makes sense to our most important stakeholders -- the adult learners whom we are here to serve.

All the best!

Peggy McGuire

Oct-17-2003     10:03 AM Message 23 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: comingsj

Reply to: pmcguire    [   Message 22  ]

I agree with Peggy's posting. We do need consensus on the outcomes of our programs and EFF does offer an approach that could develop outcome tests that are more useful for some or even all of our students. EFF still needs more resources and time to develop the assessments, and the field might want to improve existing instruments or develop new ones.

The EBAE paper suggests a way to move forward before that consensus is reached and the instruments are developed. That is, start with one or two populations that have goals and needs that are well defined and outcome measures that may not be perfect for accountability but are acceptable for research purposes. So, are there subgroups of students for whom we already have a consesus on outcomes and good measures of those outcomes?

NCSALL research indicates that a significant number of our students (maybe up to 30%) come to our programs seeking a GED or its equivalent and have reading skills sufficient to pass the test or meet the requirements of their ASE program. These students need help preparing for the GED or meeting the requirements of their ASE program, help making the transition to postsecondary education or training, and instruction that builds their "academic vocabulary" (the reading skills that are particular to the academic environment).

I believe we have the tools needed to identify these students, know how to serve them well, and have good outcome measures to judge program success. Why not start our research with this group? Are there other groups that are ready for model evaluation?

I also agree with Debbie that the constraints on research in education in general and in our field specifically are daunting, but that doesn't mean this research is impossible. We would probably need a minimum of 8 research sites. I'm not sure a random assignment experiment would be the best approach to evaluating a baseline model, but the methodology would have to be rigorous and the research sites would have to be replicable within reasonable budget levels.

Oct-17-2003     3:44 PM Message 24 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: rkmcknight

Reply to: comingsj    [   Message 23  ]

Peggy is right on target in her response. Before I elaborate, let me just say that this forum is a model for using technolgy to facilitate communication and engage in identifying barriers and finding solutions. I applaud this initiative! Now, to the crux of the discussion. We are seeking models ... Although we know that content or curriculum must align with performance expectations or outcomes in order to be effective, we have come to the development process backward. This is, in part, the result of the NRS mandate for performance standards without content standards. This has led to a lot of confusion. A way to proceed in the process of developing evidence-based models is, as Comings noted, to draw from other disciplines. One such discipline, concerned with and well grounded in research is the field of instructional design. Using traditional instructional systems design, learners are analyzed to determine characteristics such as prior knowledge and entry-level skills. Next, goals and objectives of instruction are specified. Subsequently, content and instructional strategies are developed to meet the specified goals. Finally, assessments are conducted to determine the level of performance achieved and the overall success of the instructional design process. Regardless of the pedagogical approach, learners must be informed of the content to be assessed. The cyclical nature of instructional systems design ensures that performance assessment is focused on educational goals, content, and teaching strategies in a continuous feedback loop. For this reason, content standards are an essential precursor to performance standards. How can we specify a level of performance if we haven’t specified what to measure? The principles of instructional systems design are violated when we start with performance measures and move backward. Once content is identified, appropriate performance measures can be developed. And many performance measures will need to be developed for the adult educational system. No "one size fits all" approach will suffice.

Oct-17-2003     9:55 PM Message 25 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: 4128st

Reply to: rkmcknight    [   Message 24  ]

So far I am almost lost in the educational jargon being used. I think as over the years of teaching did instruction being used in the classroom chase the assesment-test makers or for the the learners sake. If we teach for the test, much is missed and it becomes less creative or we missed the teachable moment. Iowa basic skills is the norm I used in the real world. It had some drawbacks. I always felt the test did meet some standards but had geographic, social and economic weaknesses. Stakeholders the parents, school board, and community can say we went up or down district wide. That leaves out the climate of the learning environment. If we teach for the test in schools it becomes a competative atmosphere somewhat and not necessarily good for live long learning. Some parents always worried of their children not staying abreast or not being on level. All the assesments were taken and and gone over in group-parent conferences with both teacher and parent apprehensive about some score on a test. I am just trying to put a face or person on all this and not meaning to put down researchers,testmakers or academia. Off my soap box now!

Oct-18-2003     5:07 PM Message 26 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: andresmuro

Reply to: 4128st    [   Message 25  ]

Supposed that the expected outcomes of a certain program are increase in literacy behaviors. To me some literacy behaviors may include: using an aspect of the health care system, helping children with homework, ordering a prescription online, understanding a prescription, voting, using the internet, going into a chat room, transitioning to college, getting a job, eating healthier, donating blood, moving to the Shelter for Battered Women, applying for political asylum. To me, these are examples of outcomes that I hope for my students as they attend and complete our program.

How can I measure these? The only way I can think of, is by documenting them if and when they are reported. However, different students may engage in different behaviors. Also, some may not report any, but that does not mean that they are not engaging in them. How does establishing programs that seek to do this and document this can be accomplished?

I feel that anytime you have a standardized outcome by which a program is judged, it detracts from the instructor’s ability to do the above. Think of it in this way: A student is at high risk of diabetes or has diabetes, but doesn’t know it. What is a more important program outcome, passing the GED or accessing and seeking treatment?

In a lot of classes, there are possibilities that some of our students are in this situation. What shall the teacher do? Could we report as a positive outcome that a student started receiving diabetes treatment?

Oct-19-2003     10:00 AM Message 27 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: pmcguire

Reply to: andresmuro    [   Message 26  ]

Andres (and all). What I think we can do is to help adults develop and apply the knowledge and skills they need in order to effectively engage in one or some or all of the literacy behaviors you list. Once we identify what those adults need to know and be able to do in order to meet their real-life purposes for learning, like the ones you mention, we can teach this "important content" in contexts that give learners opportunities to apply and practice what they are learning in meaningful activities (reading with understanding some information about diabetes treatment options in order to prepare for a visit to the doctor? Speaking in a simulated doctor visit so the doctor can understand one's concerns about getting the best treatment?). In fact, the evidence from cognitive research into how people learn and develop expertise tells us that teaching in this way makes it more likely that learners will be able to transfer what they learn into real-life situations, to meet real-life needs. And here's the really good news: it is possible, and in fact important, to measure this kind of contextualized knowledge/skill development and application in an assessment system that includes performance-based assessments. These assessments allow us to collect -- in sufficiently standardized ways --evidence that learners have mastered the knowledge and skills needed to successfully perform some meaningful task. So in the examples above, we would be measuring ability to Read with Understanding, or to Speak so Others can Understand, in the context of real-life health-related activities.The focus for instruction and assessment is the same important content, and the evidence of learning speaks to adult purposes. Andres, with this kind of evidence in hand, I think we have a sound basis for linking learner achievement to the kinds of important outcomes you have discussed.

Oct-19-2003     12:08 PM Message 28 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Reply to: pmcguire    [   Message 27  ]

While the discussion on the first question may continue -- and I hope it will -- here's the second question which you may wish to comment on:

How could the initial steps, set out on pages 17-18 (Next Steps), be improved? You may wish to ask questions about, or refine, the steps suggested, or you may wish to suggest additional steps.

Some postings have begun to address this second question, which is fine.

Thank you to those who have already responded. Your contributions are thoughtful and focused, and the quality of the dialogue is excellent. I also want invite others to join in the discussion who have not contributed yet.

Oct-19-2003     1:23 PM Message 29 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: GeorgeDemetrion

Reply to: DJRosen    [   Message 28  ]

Hello All:

I've read through most of the commentary on this question. This is an impressive group!

I share with Debbie and others the difficulties (if not practical impossibility) of the task. Still, I'm here.

The underlying issue many seem to be raising is not so much the difficulties in establishing evidence-based criteria that provide good models for program improvement, but the legitimacy of the evidence drawn upon.

As we know, this is a sticky matter that cannot be separated from the politics of literacy and the discjuncture between the neo-conservative beliefs of the current Department of Education and much of the progressive educational literature that many of us have been reared on, which shape our identity as literacy practitioners and researchers.

In terms of definition of what evidence-based educational research consists of, I think it's essential to deconstruct the hierachy which privileges research based on experimental design as ideally the highest form of research and placing case study at the bottom range of the pyramid (p. 5). This is especially a problem when it becomes difficult to tightly isolate independent variables, which is often the case when considering the various influences that intersect on the quality of programs.

What I haven't seen (not that I'm going to take it on) is a systematic review of the literature that does exist pointing to program quality. A systematic and synthetic review of that body of work would represent one important baseline in identifying the core issues the field is grappling with, which then can be subject to further refinement.

In addition, as practitioner-researchers, what would also be valuable is a field wide query through the lists and other venues on the three queries posed by page 12.

What concerns me in the current document is the privileging of the researcher (and it is a neo-postivist researcher that is idealized) as the ultimate expert, though one attuned to the "folk" wisdom of practitioners, an attunement that is taken very seriously in the NCSALL document. Drawing on Cochran-Smith and Lytle's Inside/Outside: Teacher Research and Knowledge, I want to push hard on that assumption and deconstruct a privileged role either for the academic researcher or the practitioner and state that both have important provileged information and insight that needs to be factored in on equal terms.

In addition, I want to open space for other intellectual traditions than positivism that have shaped educational scholarship for a century. These various paradigms of social science research are detailed in Donna C. Mertens' important text, Research Methods in Education and Psychology: Integrating Diversity With Quantitative and Qualitative Approaches. I also deal with research traditions in Chapter Nine of my forthcoming book, Conflicting Paradigms in Adult Education: In Quest of a U.S. Democratic Politics of Literacy. The Chapter is titled Research Traditions: Problems, Paradigms, and Polemics.

What Mertens, I and others are attempting to do is to open space for a diversity of research traditions in order not to privilege a single interpretive scheme in answering such questions as listed on page 12 of the NCSALL Report.

To cut to the chase:

a) Look closely at the framework that gives shape to one's research paradigm, including the one that underlies the NCSALL paper.

b) Deconstruct the hierarchies in what counts as legitimate evidence.

c) Include a systematic review of the literature that pertains to the queries posed on p. 12.

d) Include practitioners, researchers, and theorists as equal dialogue partners in the joint grappling with the issues identified on p. 12.

e) Open up the query to the broader field via the listservs and others means

f) Do not allow the the issue of defining "evidence-based" research to be shaped by the US Department of Education

g) Seek answers to those quesries on p. 12, using the best avaliable sources of information and research methodologies regardless as to what is legitimized or not in Washington D.C.

George Demetrion

Literacy Volunteers of Greater Hartford

30 Arbor Street

Hartford, CT 06106

(860) 233-3853

(860) 236-1640 fax

george.demetrion@lvgh.org

gdemetrion@msn.com

Oct-19-2003     9:38 PM Message 30 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: durhamlit

Reply to: DJRosen    [   Message 28  ]

Down in the Trenches Reality Check -

Less is More appeals to me as a practitioner who everyday faces the challenge of insufficient resources - time, money, and sheer energy - to meet the immediate challenges facing students, staff and volunteers down here in Durham, NC. Any participatory research study should acknowledge the stake and expertise of practioners, but also the limits of those practitioners' capacity to take on a big project.

Setting up the research partnership so it works on the operational/partner level is crucial. A couple of priorities I see are:

1. Finding funding to provide "release time" to participating practioners to ensure they can be fully involved.

2. Taking an asset-based approach that builds on best practices already in place (not just those proposed in theory). Thus the purpose of the literature review would be to create benchmarks for literacy outcomes or correlates of those outcomes (average number of hours of instruction logged/student; average standardized test gains; certifications gained (GED, vocational test, driver's license, citizenship, etc...); self-reported learner/teacher satisfaction levels.

3. Inviting programs to self-nominate based on good outcome data or an investment in improving outcomes.

4. Involving researchers as outside consultants who

help practitioners structure research questions, gather and analyze datal, and then provide the umbrella function of seeing and communicating the common themes emerging from the well-grounded, practioner led study teams.

5. Envisioning the development of assessment tools that complement (rather than supplant) standardized assessments, . If we want federal dollars, we can't whine about all standardized testing being unfair - or expect that a system for evidence gathering become too idiosyncratic or time-consuming. A good example of such an assessment tool is the competency checklist. REEP in Arlington, Virginia conducted a small study in which they documented that the evaluations of teachers using checklists produced resuts confirmed by standardized tests. In so doing, they made a case for using a more nuanced form of student assessment that offered more to their students and teachers, while also passing muster in terms of reliability and validity.

I would also caution against too ambitious a scope for the initiative. With limited budgets, the return on investment when it comes to following up with students who have left the program seems relatively small. That said, I would love to have some researcher do a longitudinal study of families to determine what the value added for children is of a mother's participation in an adult ed program (provided that participation is of sufficient intensity/duration to be expected to matter).

Lucy Haagen

Oct-20-2003     12:08 AM Message 31 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: rkmcknight

Reply to: durhamlit    [   Message 30  ]

All of Lucy's comments in Message 28 are pragmatic and realistic. We can, once again, find examples of effective standardized testing within other disciplines to inform adult education. Although these assessments are standardized, they utilize observation of performance.

For example, in nursing, which necessarily requires certification prior to licensure, many forms of assessment are utilized. In the certification process for nursing assistants, students take a written test AND must perform certain skills such as bed-making or measurement of vital signs. These assessments are done through observation using checklists that are standardized to assure validity, reliability.

Performance-based checklists are a pragmatic way to accomplish standarized assessment through observation of applied knowledge. They allow for context-specific application of learning. Another advantage of observational assessments such as checklists is that they do not place unrealistic demands in terms of time at the program level - a burden that many practitioners may find unworkable.

This type of testing requires training to assure that assessments are uniformly administered. Testing of nursing assistants is, in fact, administered independent of their educational programs, further assuring that bias does not enter into the assessment process.

Roberta McKnight

Virginia Literacy Institute

Oct-20-2003     10:10 AM Message 32 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: bonniesophia

Reply to: rkmcknight    [   Message 31  ]

I am impressed with the quality of discourse and thoughtful replies I've seen in this discussion so far. I would like to articulate a few concerns about the theoretical base for the article. I think George Demetrion's comment about a review of the literature, and Agit Gopalakrishnan's observation about baseline standards, are pertinent: "practitioners" need a resource of commonly accepted assumptions from which to "practice." The medical model explicitly proposed implies that not only can there be a "diagnosis", there can be a "cure" based on existing knowledge and shared standards of practice. The research-based EFF model allows for a great deal of diversity in program practice in applying frameworks and standards, and, yes, needs more time for assessment models to test program improvements (an aggregate of individual improvements?). What I reacted to most was the (almost in passing) remark that the evidence-based model is "normal science" as defined by Kuhn, which, obviously, doesn't allow for different fundamental assumptions, "paradigm shifts", or for questions about whether fundamental principles of another area of educational research (K-12) would apply, say, to ESOL, or that principles of ESOL would apply to ABE. This is just a shot in the dark, but might address some of the questions of implementation that have been raised.

Bonnie Odiorne, Ph.D.

"Working Smart"

ABE/ESOL, Employment and Technology Training

C4K

Silas Bronson Library Information Technology Center

267 Grand Street

Waterbury, CT 06702

203-591-1714

bodiorne@c4k.org

Oct-20-2003     10:29 AM Message 33 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: agopalak

Reply to: comingsj    [   Message 23  ]

John suggested perhaps starting with GED and Secondary group first.

Earning a diploma (especially the GED credential) is very well associated by the general public and legislators with the overall mission of adult education. However, English literacy is not seen in the same light. I believe that though research is a method for inquiry and knowledge creation, it cannot be separated from the context and politics of its times.

As a result, I think we should not focus our first research effort solely on the secondary goal, and include some focus on ESL as well. I am not sure which sub-group within ESL can be the focus.

Nationwide enrollments Statistics from report to congress for PY 01 suggest:

ABE – 37%

Secondary – 21%

English Literacy – 42%

Moreover, with regard to the secondary goal, it seems to me that focusing on adult high school diploma does complicate matters because of varying state and local standards for diploma acquisition.

Ajit Gopalakrishnan

Oct-20-2003     3:07 PM Message 34 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: abe102

Reply to: agopalak    [   Message 33  ]

In response to DJRosen's question (10-17) about instruments to measure adult learners' achievement: It seems there may be achievement tests (e.g.reading) and then perhaps "subtests." It depends on the outcomes. There are several skill areas to consider in becoming a better reader, as pointed out in Kruidenier's book. If we test only on a test like CASAS or TABE to determine reading level, how do we know what reading difficulties to address in instruction in these important areas without giving additional reading diagnosis tests? It would seem to me that in terms of outcomes for the "graduating" learners or learners who leave, we want them to get a higher "achievement" score as measured by a standardized test, but also want them to be better readers -- they are not the same.

Oct-21-2003     8:39 AM Message 35 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: pmcguire

Reply to: abe102    [   Message 34  ]

Hi everyone. With reference to the proposed next steps, I've already expressed my concerns about defining groups of students to be the focus of initial baseline program models/evaluations as though the range (even a narrow range!) of needs and goals of adult learners can be adequately expressed in scale scores from currently-approved standardized tests or in K-12 grade levels. But I do want to say that I also appreciate John's concern that we get started somewhere, to try out the process by developing and testing a few models even before we have a fully-realized system in place to align Adult Basic Ed instruction and assessment with what we agree are important adult achievements. It makes sense to me that, in order to get started under these conditions, we would want to choose a group of students that is relatively "easy" to define by one fairly concrete and explicit (and very familiar!) goal -- to "get my GED" or "get my diploma". But it seems to me that the reason the goal is so conveniently "narrow" in these cases is that it is directly related to gaining a particular credential -- what some group of people has decided will constitute (as in a local ASE program), or can be equated to (as in the GED), high school completion. The decision about what "high school completion" means has guided development of the tests, and what's "in the test" has guided instruction for folks who are preparing to take the tests. Oversimplified, of course, but my point here is that the "credential-as-goal" is what would make this group relatively easy to start with, right? Then, here's my research question (and it's an honest question -- no other agenda than wondering!): If we choose this group to begin with, how sure can we be that what we learn about program development and evaluation design will be useful in future development of program models/evaluations where the student group is not so neatly defined by a credential? How well will our learning "transfer" as we begin to address the rich variety of learner goals and needs in our definitions of adult achievement? Or is it even important for what we learn from one model to inform development of others? Can we say the process "works" or "doesn't work", independent of/distinct from any particular student group, defined by any set of needs and goals, that we choose as its focus?

Peggy McGuire

EFF Assessment Consortium

Oct-21-2003     11:48 AM Message 36 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: comingsj

Reply to: pmcguire    [   Message 35  ]

I'm not sure that what we would learn from testing a model for ASE would inform models for other groups, but I suspect that it would. Even if it did not, we do need a model for ASE that not only helps people acquire a credential but also helps them build basic skills and knowledge and transition to postsecondary education and training.

This would be a research project, and so we would be able to use measurment tools that might not be appropriate or too costly for use in an accountability system. Multiple measures of outcomes and logitudinal studies of impact are possible. The model for orientation, instruction, support services, and transition would, however, have to work within the constraints of a reasonable per-student cost and of our student's lives.

One problem with discussions in our field is that we serve so many different kinds of students. Thinking about this specific group, ASE students whose reading test scores are at or above grade-equivalent 8, might make our discussion about the approach to building an evidenced based system easier. Later in the discussion we could talk about what that model might look like and how we would judge its effectiveness.

With this specific group in mind, do youhave advice on how to improve the approach and the next steps described in the document.

Oct-21-2003     2:28 PM Message 37 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DebbieYoho

Reply to: comingsj    [   Message 36  ]

Here's where my limited understanding of research design prompts me to ask some questions. What I don't understand is how it is possible to apply a standardized method of assessment for research purposes without standardizing the student group in the process.

Let's say we do a literature search and identify XYZ methods/content/etc. that has already shown to "work". We construct a model that includes what we already know "works", plus some additional techniques/content that we want to evaluate. Let's say we include both researchers and practitioners in designing the model. Suppose we agree to concentrate on ASE, and construct a program for learners whose entry level is at an RGE of 8 and who want to pass the GED. Let's say we have dutifully identified all the content/skills needed to pass the GED (that's already been done by the writers of the GED, anyway.) Let's say we agree to administer Test(s) XYZ to establish what each learner's baseline (entry) level is. (As for David's question about improving assessment I'll leave that for another post. To continue...) Okay, we test a group of learners who "qualify" according to the criteria we've set to define a group to work with. Next we apply XYZ program model, Then we test the learners again to find out if it worked. But some of the learners will enter the program after the initial testing and will not have been tested on the same day, (under the same conditions) as the rest. Some will have missed class for an infinite number of reasons, and will miss content. Some will drop out, etc. etc--all those variables I alluded to in a previous post. The group we end up post-testing will not be the same group we pre-tested, though there will be overlap. So okay, take the data for those who did participate under a given set of conditions. Let's say the tests show improvement, let's say for a majority of the learners. Conclusion: the model worked for those who participated at whatever level, under whatever conditions. We know that the model worked with certain learners under certain conditions. But we don't know WHY it didn't work with those who don't show progress. Was it the model, or one of the myriad variables we all know effect the learning process? And even if we know under what conditions the model did work, how do we know that those conditions can be recreated?

But let's forge on. Suppose we arrive at the conclusion that this or that model works for say, learners whose beginning point is RGE 8, and who participate in a class of no more than 10 learners for two hours a day every day over 10 weeks. So how do we apply this new knowledge? By setting up the same conditions. Okay, fine. Now what about all the other people who don't match this type of learner we have "standardized"? If I understand John's paper correctly, what we do now is search for a model to help that group, then search to help another group, and so forth.

But throughout this, no one has asked the learner what he/she wants to learn, why he/she wants to learn it, or how. But let's put those considerations aside, and agree that what we are looking for is the greatest good for the greatest number of people. That's why we're searching for models to begin with, right?

Okay, let's say lightning strikes and we identify such a model. We still haven't even begun to deal with even some of the variables involved from teacher to teacher. We already know from the K-12 world that two teachers can use the exact same model with students with the same characteristics, so far as we identify those characteristics, and come up with different results.

Let's say we even conqueor that challenge. In fact, I could even make the case that this has been done in K-12. A majority of the kids do graduate in 12 years, a majority do function "on the job and in society", etc. etc.

But the overiding problem we have is not with those kids, but the adults the OTHER kids grow into. Yet here we are trying to construct an adult system with the same intent--the "greatest good for the greatest number".

I will stop here and resist preaching. I am genuinely asking, genuinely hoping that the process outlined in the paper can lead to something meaningful. I do not oppose spending time and money on such an endeavor. I raise these questions as a "devil's advocate", because clearly the devil is in the details, and maybe my questions will contribute something to refine the process. And if there is anything else I can do to improve the process, let me know. I think we are searching for a Holy Grail, but heck, I've chased a lot of them in my career.

Oct-21-2003     5:35 PM Message 38 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: AWilder106

Reply to: DebbieYoho    [   Message 37  ]

Hi all,

This is about the medical model. i may be the only person in the discussion who has participated in a double-blind medical model study. This is what happened to me: while I got happier from all the attention, in fact I was getting sicker. Afer the study ended I went to another doctor (the study was in a hospital, a doctor was in charge) and got treatment that actually helped me. I found out a simple test would have proved the value of the first treatment, if there was value, a couple of weeks out.

I conclude:

1) The medical model is not by itself perfect

2) The most current information should be used as content

3) The process should be continuously monitored by outsiders not wedded to a particular theory, and by skeptics

Oct-21-2003     7:04 PM Message 39 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: AWilder106

Reply to: AWilder106    [   Message 38  ]

Debbie,

My understanding is that with a truly randomized study you need a very large group so group differences wash out and central tendencies emerge.

Often a pilot study is undertaken to see whether the measures have any validity before going on to a larger study.

Oct-22-2003     11:43 AM Message 40 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: HSilverP

Reply to: AWilder106    [   Message 39  ]

In response to John's suggestion to begin with a definable group (in his comments, a high scoring, ready to test and transition to comm. college), I think the other end of reading abilities spectrum would/could be another definable group. If we go looking for another group, I'd suggest we have a lot of good work waiting for us for those adults who score below RGE 3. While the group is fairly definable by extreme low scores on existing standardized tests, several good research opportunities await (in no particular order):

1) instructional models to support emerging literacy

2) assessment measures to capture literacy growth (nothing we are using now is sensitive in this range)

3) professional development needs for instructors to teach the models

4) personal development/transformational goals and experiences of adults learning literacy

5) some psychoeducational testing for all to get a sense of the cognitive profile, how many of these "reading disabled" students are truly learning disabled and how their learning disablities profile/s are similar or different from what we know in the adults with LD research (I proposed this part as a grant to NICHD but was turned down, darn it, still seems like a good idea!)

Also - I am a big fan of chasing down students who have left programs in order to learn from them. We absolutely MUST learn from those who leave or we will continue on designing programs for new students, not supporting others through a long-enough commitment to meet their goals. Alan Quigley has talked about this, too. As we start baseline programs, I encourage us to think about a group of learners being the program, not the institutional base. If we can focus on WHY learners come and go, what is going on in their out-of-class lives, we might be enlightened.

Oct-22-2003     4:16 PM Message 41 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: alangdon

Reply to: HSilverP    [   Message 40  ]

I agree with the last post about tracking down learners who have left programs. I am still trying to figure out the most effective way to gain feedback from our students, current and former. I think for any program to truly develop and come closer to truly meeting the needs of students, it is key to gain insight from those who are no longer in our programs.

Continuing on the discussion of definable groups -- As a practitioner and program director of a large adult ESOL program in DC, I have found that, especially within ESOL, the learner groups need to be not only identified and served based on their performance on a test but also on their cultural, geographical and employment demographics. Especially with regard to culture, I have found that learner attitudes toward education vary greatly across cultures and must be taken into account, not only at the classroom level but also at the program level. While some programs try to be as flexible and as accomodating as possible, we have found that our set registration times, standard assessments and consistent policies help create a context in which our learners are invested and feel they are accomplishing their goals. We have policies that some deam "old school" or characteristic of elementary school, but given the fact that many of our students have had interrupted or limited educational careers outside of the U.S. , they desire and thrive in our structured environment.

As far as geographic and employment demographics, in the case of ESOL programs, resources, teacher pools, and options in services all differ greatly between urban, suburban and rural settings. Employment options for our learners also differ greatly and need to be considered in program models as well.

In summary, I am suggesting that when actually going out and doing the "experiments" the definable groups need to be defined even more to get results that can be used by a large number of practitioners.

Oct-22-2003     6:06 PM Message 42 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: ralfstc

Reply to: alangdon    [   Message 41  ]

Due to growing sense of discomfort, I'm afraid I'm going to step out of line here. As with many online discussions, it seems as if the assumptions guiding the answers have been put in place before the questions are asked. The basic assumption I would challenge is the notion that standardization is actually desirable (in the sense of baseline programs, for example). Why? Ironically, evidence would seem to suggest that evidence based instruction doesn't actually make that much difference to outcomes-- if anybody has evidence that it does, I'd love to see it. Surely, at some level, the notion of evidence based best practices (a limiting and ambiguous notion) contradicts the learnercentredness adult educators, at least theoretically, embrace. Applying such practices essentially argues that moving the mean score of a group of learners is more important that the ongoing educational relationship with the person in front of you. Or are we abandoning the notion of learnercentredness in our enthusiasm to create a brave new world of literacy programs. What we do now is, of course, not perfect. But are we on a fool's errand in our attempts to create a system that is? The North American approach to percieved problems is all too often to throw more technology at it, and evidence based instruction is yet another technology. Can't we do better?

Oct-22-2003     8:59 PM Message 43 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Reply to: ralfstc    [   Message 42  ]

RALFSTC wrote: "The basic assumption I would challenge is the notion that standardization is actually desirable (in the sense of baseline programs, for example.)"

The principles of Baseline Program Models have deliberately not been defined. ("The principles for each element of the baseline program model are not identified in this paper. These principles should come out of a comprehensive process of inquiry, dialogue, and debate among academic and practitioner experts. ") But this may make the concept of a Baseline Program Model vague and confusing for some readers. Perhaps some examples are needed of what these principles might look like.

If I were to participate in a dialogue to define the principles, I might put forward the following:

1) a program approach/philosophy which is articulated, understood and widely agreed upon within the program (not necessarily one that I would embrace, but one that is clear and that the program staff and students embrace;)

2) program stability (low or reasonable staff turnover; reasonably stable funding; adequate instructional materials) and

3) sufficient intensity of instruction such that a reasonable percent of students meet their goals and/or make learning gains.)

Whatever these principles eventually are agreed to be, they will define the baseline for the education programs to be studied -- not necessarily standards for all adult education programs. As I understand the concept of Program Baseline Model it is a way of defining a threshold for the programs that would be studied so research resources would be spent wisely. It doesn't make sense, to me at least, to study programs whose models are known to be poorly designed, resource-starved, conflicted in philosophy or goals, or with high student dropout rates.

David J. Rosen

Oct-23-2003     8:27 AM Message 44 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: jessicadilworth

Reply to: joadurkee    [   Message 8  ]

I think the idea of designing programs based on evidence from 'what works' is exciting and can lead to more effective education for adults. For example, in the past few years adult education program directors have been trying to respond to reading research and performance-based funding issues in making programatic decisions and in facilitating the training of our teachers. This is all weel and good, because once we know what the bottom line is, we can respond. The problem I have found is that 'what works' is not often measurable by the tools we currently have or are mandated to use. On page 9, it states that we must clearly define outcomes and "identify tools that measure those outcomes". This is one of the biggest tasks in front of us. The adult education standardized assessments we have used in our programs do not measure what students are learning; they oftentimes just add frustration to the student/teacher relationship; and they lead to program decisions that are responding to a test rather than the education of whole people with big lives. The GED test, while a good benchmark, does not actually lead to the life changes students are looking for when they talk about their real goals. I'd like to see some research addressing student outcomes and how to measure them as a first step. Equipped for the Future has determined that there are four purposes that frame the reasons adults enroll in adult education classes. If these purposes are true, where is the assessment for us to determine whether we are designing our programs to meet these objectives? If the adult education 'system' takes at face value the performance assessments programs are currently using as the indicator of student and program success, any attempts at focusing on program design based on those faulty tests is also be faulty.

Oct-23-2003     12:15 PM Message 45 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: HSilverP

Reply to: jessicadilworth    [   Message 44  ]

Hey Virtual Jessica (who is in reality just down a dusty desert road from me!) and all others -

Two things pop out at me this morning from yesterday's discussion:

David Rosen specifically said that we are not looking for "resource starved" programs to study...well, we are going to have a heck of a time convincing any of ourselves, first, and anyone else that our budgets of approx. $200 per student per year does not constitute starvation. We've been starved so long we're anorexic with all the mental illness that resource starvation engenders. Most of us don't even know WHAT resource healthy living and teaching would look and feel like. What are the NCSALL lab programs running on? Are they resource healthy? Do they want to join this part of the conversation?

The other comment was related, the value of looking at and looking for standardization. One thing that I worry about, obviously, is that if we are only studying what we are already doing, we will miss a whole spectrum of possibilities that we have not conceived of due to our constraints and assumptions. Part of our exploration of what a great program can be and do should be to look outside of our own federally and state funded system. What's going on in Canada? Australia? The UK? Other non-English speaking countries? We used to look to Denmark for adult folk schools, but I haven't a clue about what's happening there these days. We need some fresh ideas!

Oct-23-2003     3:56 PM Message 46 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: anpb

Reply to: DebbieYoho    [   Message 37  ]

I think there is some danger in starting on a course of research without, at least, having a more comprehensive research agenda in place first. I am concerned about the way preliminary research could be used, in this policy climate, to make quick decisions based on efficiency and narrowly-defined outcomes, without waiting for all the supplementary questions to be answered.

I believe this research agenda needs to be built on an agreed upon set of purposes for our system that go beyond individual achievement and beyond the social goals we've been handed by WIA, getting to some of the outcomes about self-efficacy and civic engagement that Andres and others have raised. Our research questions could then reflect the full range of purposes we're trying to achieve as a system. In my mind, this does not lead to the current sequence of steps as outlined on pages 16-17, but rather starts by identifying that range of purposes (increased health awareness, increased understanding of and voice in children's education, increased access to living wage employment, etc.) and then researching the program factors that effectively support this. And this might mean examining non-traditional models (such as programming organized less by "level" and more by goal areas) right from the start.

Oct-23-2003     4:02 PM Message 47 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: anpb

Reply to: anpb    [   Message 46  ]

Sorry, I thought my name would get automatically generated on the message. Message 46 is from Andy Nash, World Education.

Oct-23-2003     6:21 PM Message 48 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Reply to: anpb    [   Message 46  ]

This is very intriguing. I wonder if you, anpb, or if anyone else participating in the discussion could say more about program models which are designed to address goals like you have listed: "increased health awareness, increased understanding of and a voice in children's education, increased access to living wage employment."

Examples which come to mind include:

• a health empowerment program at El Paso (TX) Community College which Andres Muro has described on NIFL e-lists;

• a program in Lynn, Massachusetts called Operation Bootstrap, which has had adult learner leadership and a student health promotion team as part of its model; and

• several community-based programs in Boston, Massachusetts (Mujeres Unidas en Accion, WAITT House, and others) which have for many years had student leadership and empowerment embedded in their program model.

Are there other examples of program models (not just programs, but programs with models) specifically designed to address student purposes and goals?

More about Program Models in my next post.

David J. Rosen

EBAE Discussion Work group Moderator

Oct-23-2003     6:29 PM Message 49 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: BOP6751

Reply to: anpb    [   Message 47  ]

Reading this discussion thread, I am struck by two distinctive attitudes: one that seems ready to go forward with some kind of a "normal" science approach, and one that stresses the need to first explore the basic assumptions and definitions of literacy. But I wonder if there is a more fundamental question here:

Can we find a bi-cultural way to move forward?

This would acknowledge the legitimacy of both kinds of inquiry, but more importantly, it would also require us to recognize the importance and the limits of the numerous contextualized and decontextualized approaches to instruction and learning. I do not think it is so important to reach a "concensus" on these very different ways of knowing about learning, so much as a certain level of respect for, and acceptance of, the positive tension their juxtaposition creates. Then we could shift the discussion to how to go forward: Are there bi-cultural program models out there? (To me this means programs that embrace learner-centered support and systemmatic skill instruction in various degrees, depending on the needs of the adult learner.)

Others may not agree that a bi-cultural program is possible. If that is true, then perhaps NCSALL would need to establish two separate inquries. As a practitioner in prison literacy programs for 23 years -- as a teacher, school principal, and now at the policy level -- I do belive bi-cultural programs are possible, and I am working hard to develop a model for our system. But more importantly I value the contributions from both communities, and hope that both will feel well represented in whatever approach is finally adopted.

-Bill Muth

Federal Bureau of Prisons

Oct-23-2003     6:31 PM Message 50 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Reply to: DJRosen    [   Message 48  ]

More about Program Models

An evidence-based adult education system, one in which researchers and practitioners together (including practitioners who are researchers) closely examine program models would require that we have program models.

But what is a program model? How would we distinguish a program which had a program model from one that did not? Perhaps someone in this discussion has wrestled with this, or can cite an author who has. If you know of a document which you think addresses this well, send us the reference. Better still, if it's on the web please send us the link.

David J. Rosen

EBAE Discussion Work Group Moderator

Oct-24-2003     9:58 AM Message 51 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Reply to: DJRosen    [   Message 50  ]

Subject: Madeline Hunter

From: 4128st  

This is the only ed. model I was familiar and used yrs ago. It had a pathway on how to present and the content was up to you. Maybe old but good.

(Please note: this message was written by "4128st," not by David Rosen, but was re-placed in this discussion thread by David Rosen, the discussion moderator.)

Oct-24-2003     11:30 AM Message 52 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: andresmuro

Reply to: BOP6751    [   Message 49  ]

Hi bill, et al:

When NCSALL was first established a woman called Juliet Merrifield wrote a paper that I consider transcendental in the field. One of her main arguments was that before we moved anywhere, we had to envision what we wanted to get out of literacy, and we had to build a consensus. Since then we still don't agree with what we envision as outcomes of literacy education, nor do we have consensus on how to measure outcomes. Yet, some people want to move forward without that consensus.

I envision literacy as a tool to fully participate in all aspects of family, education, vocation and community life. I envision outcome of literacy education as literacy behaviors that reflect participation. I can list examples of behaviors, but I cannot predict all behaviors that reflect this.

What I do not want is a model that restricts the ability to help participants participate in the above mentioned contexts. This is unfortunately what NRS has done. We have an EL civics grant and its purpose is to prepare participants to fully participate in civics life (whatever that means) However, the only outcome that NRS cares about is that participants show progress in the BEST within prescribed intervals. So teachers get caught in preparing students to show progress in the BEST within those timelines, instead of helping students to fully participate in civics life. Some people claim that both aims can be intermeshed. I say BS.

By the way, in our program we had 305 students obtain the GED last year. This is a record for us. Aside from passing the GED when they are ready, we do not require our teachers to use any kind of standardized assessment tool. However, we require our teachers to document, for our purpose, literacy behaviors that students have engaged in. In fact, we require our teachers to facilitate instruction that will facilitate acquisition of new literacy behaviors. We have tons of behaviors that we document all the time, so even if students do not pass the GED, they demonstrate all kinds of things that are valuable to their quality of life. Unfortunately, even if these behaviors are much more valuable than passing the GED, nobody cares about them.

examples:

1. We have had students who have dumped their abusive spouses and have applied for asylum

2. We have diabetic students who din't know about their conditions, and began to receive treatment

3. We have students that got glasses

4 We have students that registered to vote

5. We have students that have signed up for primary care insurance for themselves and their families.

6. I can go on forever...

I consider these outcomes just as important as passing the GED, if not more, and certainly, more important that showing progress in a standardized test such as the BEST, TABE, CASAS.

Andres

Oct-24-2003     4:33 PM Message 53 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: sanstr

Reply to: andresmuro    [   Message 52  ]

Dear Colleagues,

I’ve very much enjoyed following this phase of the discussion and would like to offer a few comments before we move on to the next topic.

First of all, I’d like to comment on something that “ralfstc” said (sorry, there was no name, so I’m not sure who you are), “evidence would seem to suggest that evidence based instruction doesn’t actually make that much difference to outcomes.” I’d very much like to know what evidence is being cited here. Though I’m not a researcher, this perspective isn’t really consistent with my experience as a program director and practitioner. For the past seven years, 15 members of my staff and I have met monthly to look at research and program data and use it as a guide for improving our agency performance (enrollment, retention, pre/post rate, learning gains, GED attainment). It’s been very successful and our performance has improved dramatically in these areas. The bigger question we grapple with is one Peggy alluded to -- what do these outcomes really tell us? If 67% of our GED students now pass the GED test each year, can we really assume that these adults now have the knowledge and skills they need to succeed in their adult roles? As a practitioner I have no doubt whatsoever that an evidence-based system would improve outcomes. If there is agreement on the outcome, and practitioners implement research-based practices and watch their performance data, those outcomes will improve. The thing I worry about is who’s determining which outcomes and the impact that has on the evolution of practice.

Student grouping is also very interesting to me. Yesterday, one of my staff asked me to read an article in the August 2003 CLASP (Center for Law and Social Policy) Policy Brief, “The Language of Opportunity: Expanding Employment Prospects for Adults with Limited English Skills.” In this article the authors (Heide Spruck Wrigley, Elise Richer, Karen Martinson, Hitomi Kubo, and Juli Strawn) argue that our current WIA system (including Title II) is not producing the employment outcomes that adult language learners need and want. Further, they argue that the economic well being of our nation will suffer if we don’t do a better job of transitioning language learners into all levels of the workplace. At the local level I’m seeing things that appear to confirm this. Right now we group language learners based on native language literacy, reading ability, and spoken fluency, but that doesn’t seem to be helping them to succeed in Title I-sponsored training programs designed around job clusters. I want to know more about which types of learner groupings are most effective in helping learners achieve specific outcomes. For example, if I have a learner who wants to pursue a career in the construction industry, do I group him/her with other learners with a similar goal regardless of academic level (what our learners say they want) or do I continue to group by reading level and spoken fluency (as my teachers suggest)?

I completely agree with John Comings that it doesn’t make sense to wait until we have consensus on outcomes and measures before moving forward toward an evidence-based system. Having said that, if we’re going to choose a starting point for evaluating a baseline model, I would suggest that the subpopulation, outcome and measures selected be something that is significant to our learners, agreeable to our legislators, useful to local programs, and meaningful to our field. I worry about starting with high functioning GED students for a couple of reasons. First and foremost, the question that begs to be asked of our learners is GED for what? At the local level, we spend a lot of time talking with learners about the GED not being the silver bullet of future success. Starting with GED students reinforces the GED as destination approach to our work. Secondly, as a practitioner, I would say that high-functioning GED students are the population I need the least help in serving. I’m certainly not saying there isn’t a great deal to be learned, but I am suggesting my need is much greater in other areas (such as ESL for the workplace). If research to practice is going to work, it has to be research practitioners want and need. Finally, I would like the research coming out of these lab sites to strengthen our position in the workforce investment system. Yes, ESL students seeking family-sustaining employment is a much harder group to work with than good readers who want a GED, but it makes much more sense, from my perspective.

Ajit Gopalakrishnan asked, “An adult education program generally operates within the policies established by the local, state, and federal government. Would being a research site change any of that i.e. exempt: them from such policies?” If that statement doesn’t underscore the potential disconnect between researchers and practitioners, I don’t know what does. From a practitioner perspective, there is only one set of rules – the rules that allow us to receive continued funding for our learners. If you’re going to set up lab sites that live in some other reality, their findings become meaningless to us. To me, research to practice means we all get our hands dirty trying to figure out the really hard stuff. I agree with Lucy Haagen, it’s important not to be too ambitious with the scope of the initiative. However, let’s not take the easy way out, either.

Sandy Strunk

Oct-24-2003     6:02 PM Message 54 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: BOP6751

Reply to: sanstr    [   Message 53  ]

Andres, I certainly understand the need in your EL civics program for other outcome measures. And I hear quite clearly that you do not think traditional measures can be intermingled with measures of broader literacy practices. But in my own experience, I can recall a literacy learner whose purpose for being in the program was to learn how to spell so he would not be ashamed to write letters home. To him the quantified gains on the ABLE test gave him the confidence to write. I have used embedded phonics programs in fine arts programs with youth, and measured success by both skill level gains and gains in their ability to articulate, and reflect on, their own voice. I have seen beginning adult readers use language experience stories to generate word lists, and from their lists, build fluency, word recognition, and vocabulary skills while at the same time work through thorny family issues from afar. My point is, I have not found these different ways of knowing about literacy to be incompatible, at least not in all contexts.

-Bill Muth

Oct-25-2003     10:10 AM Message 55 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: andresmuro

Reply to: BOP6751    [   Message 54  ]

Bill:

I agree that for your student, as for many of mine, writing letters home and improving spelling are important outcomes that we need to document. However, using a standardized test to measure this is not a natural way of doing it, nor it is the most effective way t do it. It is possible that students perceive that standardized testing is the way to measure their improvements. However, this is not because it is, but because traditional educational systems may have ingrained that perception in them.

You can, for example build a glossary of terms with students, do dictation excersises, do vocabulary searches, and do written and oral evaluations of their improvments. If students rely on tests rather than on their teachers for feedback on their progress. then something is seriously wrong with our educational system.

Andres

Oct-27-2003     3:26 PM Message 56 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: AlisaBelzer

Reply to: andresmuro    [   Message 55  ]

I must start by saying that I have serious concerns about the possibility and advisability of establishing an evidence-based adult education system. Yet, I understand the political necessity of doing so, and also want to think optimistically that doing so (well) actually would improve outcomes for learners. At this moment, in the current climate, we really can’t afford to say this is philosophically a bad idea and/or it can’t be done. While we may continue to advocate for changing the way things are, we better jump in and make things as good as we can given the current reality. If we don’t, someone else will and s/he won’t have a background in adult literacy education!

With that said, I must say I have great difficulty with the whole idea of baseline program models. In its most simplistic definition, I get the concept, but when I start to think about the details it becomes a problematic notion. Given the vast range of programs, how many models should there be? If the models are made too general to decrease the number, they aren’t very useful. If they are tailored to the multitude of program contexts, formats, purposes and goals they probably become too numerous to be useful. Furthermore, it seems that by the time baseline program models have been established, using the process outlined in the white paper, we would be well down the line toward identifying evidenced based practice. This isn’t a bad thing, but it makes me wonder if the process outlined isn’t too redundant and overlapping. Is there a way simultaneously to begin establishing an evidence-based system while identifying program models? Can we be maximally efficient here? Just to complicate things further, I feel that baseline program elements number 2 is hugely broad and would have to be broken up into smaller elements to be useful.

I wonder how we can acknowledge in this plan the complex variation in every aspect of our system—from stakeholders, to definitions of literacy, to contexts and purposes, to formats and professional expertise, etc., etc., etc. Much of the white paper is written as if the complexity isn’t there or doesn’t matter as much as it does. I know that everyone involved knows it is there, but I wonder how the white paper could be written to be more reflective of this reality. I like Bill’s use of bi-culturalism. Unfortunately, I feel that it’s way more than “bi.” I would like to see the paper somehow more reflective of how little is monolithic in our field—and yet still assume the importance of coming to some agreed upon consensus about some things.

Regarding the matter of consensus, like many others, I feel leery of embarking on this process without a consensus of purpose and without good measures of achievement once we agree on what we are trying to achieve. However, when I thought about it a little more, I realized that the whole notion of consensus is pretty problematic. How in the world could consensus be reached? I think that this should not be the goal. Any kind of a system that requires consensus on such a huge and socially constructed notion as the purposes of literacy education is fundamentally flawed. And yet, I wouldn’t want the system to move forward without defining some purposes. I agree with Peggy that EFF standards are a good stand in for now.

Sorry to be long winded, but the last thing I want to say has to do with the role that practitioners play in this model. They essentially are posed as receivers and then revisors of new knowledge generated through basic research. I would like to nominate teachers as posers of questions and initiators of research as well (not just reactors). Their findings could be “tested” by other practitioners and researchers. We have to keep in mind that teachers often pose different kinds of questions from those that researchers might be interested in, and go about answering them in very different ways. They have access to information in very different ways than traditional researchers. I think the arrows on the diagram need to move back and forth! Alisa Belzer

Oct-27-2003     5:43 PM Message 57 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: mschwag

Reply to: AlisaBelzer    [   Message 56  ]

I’ve enjoyed reading through the posting so far and although late to adding to the discussion would like to comment on a couple points. First, I agree with Ajita and Sandy that selecting ASE students (those whose reading skills test at grade 8 and above) as a group to begin developing program models and conducting experimental research with may be problematic.

Although there exist standardized assessments (GED) that can be used as outcome measures many of these students look at the GED as a guidepost on route to a high school diploma. Awarding a high school diploma is a district function and districts must comply with the state ed code which (at least in California) includes passing a high school exit exam. Thus, adult students need to show mastery of K-12 content standards tested on a statewide test to be able to receive a high school diploma. At least 18 states have similar requirements.

This raises many questions for me. If California were to adopt EFF for these students, how do we know they will be sufficiently prepared to pass the high school exit exam? If students are successful on the GED, can we assume they will be fully prepared to pass an exit exam? The K-12 content standards focus on application, analysis, and synthesis within each of the academic areas (English-Language Arts, Math, Science, Social Studies). Is the GED of sufficient rigor to ensure that students successful on this measure are likely to be successful on a test that assesses mastery of in-depth knowledge and critical thinking within these content areas? If we look at acquiring a high school diploma without the state policy context how legitimate are our findings?

Another question relates to the more threaded discussion about standardization, which provides for the benefits of reliability and establishing validity in assessment. Someone brought up the notion of using observational tools and more classroom-based types of assessment. Without the rigor of structured tools and common ways of scoring it would be difficult to use these types of measures in an evaluation study. As research sites are selected and begin implementing program models it will be essential that common tools are used to assess the variety of outcomes that define student success. How else can we combine data to gather the large number of students necessary to make valid statements about student achievement?

Mahna Schwager, WestEd

Oct-28-2003     10:07 AM Message 58 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: AWilder106

Reply to: mschwag    [   Message 57  ]

Anone--info needed:

Is there a web site that lists research studies that meet the crriteria for evidence-based? Perhaps experimental and quasi-experimental models in adult literacy? I realize this could mean two things: 1) studies outside program models, and 2) studies of program models themselves. Thanks.

Oct-29-2003     9:35 AM Message 59 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: abe102

Reply to: AWilder106    [   Message 58  ]

To AWilder. There is a study, "Linking Research and Evaluation to Policy and Pratice in Adult Education: Final Report done by ABT Associates Inc. I think you can find their web site thru Yahoo.

Oct-30-2003     3:33 PM Message 60 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: mariecora

Reply to: abe102    [   Message 59  ]

I think the notion of what would distinguish a program with a 'model' from one that does not have a 'model' is important. When I ran literacy programs in RI, we had a variety of "models" that we used (if that's in fact what they were!) - I guess only a couple of our programs were models - the other programs contained characteristics that set them apart from each other. So for example, the volunteer programs had a particular model for training and for delivering instruction (education combined with discussion combined with new teachers team-teaching). Characteristics that set them apart were for example, that our ESOL programs were run in small groups, and our GED programs were run as one-to-one programs. Is that a model? I'm not sure. But this one is: I also ran a Deaf Literacy Program in which a very complex structure of bilingual education (English/ASL) was developed by the staff (all Deaf). I would say this is a model because it was much more focused on specific content (Deaf literacy), and a very specific structure of teaching/learning was developed in order to try and reach those specific goals. Also, it could be articulated in a package to others who inquired about how we set up that program. Actually, so was our volunteer training model (a "model" I would argue I guess).

But this brings me back to an original thought of mine: that defining some of these pieces seems crucial before other thinking can move forward. When I think of all the vastly different programs I work with in Massachusetts, I feel like we must necessarily come up with base-line components, rather than base-line models.

Marie

Oct-30-2003     6:38 PM Message 61 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: DJRosen

Reply to: mariecora    [   Message 60  ]

Marie, you distinguish -- at the end of your post -- between models and baseline components. Am I correct in understanding "baseline components" as a foundation of program features, for example: sufficient intensity of instruction, reasonable staffing stability, basic support services such as counseling and access to computers, and perhaps others?

And are you agreeing that before serious and costly research is undertaken, the program(s) where the research is done must have such foundation program features in place?

Are you then saying, in addition, that a program to be studied should have a clearly defined model (not yet defined) which shows promise, and is likely to be worth the effort to study it?

Nov-6-2003     3:09 PM Message 62 of 62 Go to >

Subject:

RE: Discussion Part One: How to Improve the Process for Developing the System

From: mariecora

Reply to: DJRosen    [   Message 61  ]

Hi David and eveyrone, sorry for such a late reply....regarding your question to me around having 'baseline components" - yes, what you describe is what I mean: appropriate staff, levels of instruction, appropriate materials, support services. And a further yes: I do not think it's wise to study just anything, I feel like we should identify first WHAT is worth studying. I think we should identify what the basic components of any successful program must have before we then devote precious time, energy, resources to examining it in-depth. Again, the issue of defining "baseline components" rears its head, but I feel that this area is much more manageable to develop than what program model might be worth studying. I think this is already happening on the list with folks discussing content-oriented programming - what are the pieces of the content-based programs that are common? That seem to be successful? Let's identify those pieces.

Go on to Part 2 at http://www.alri.org/Rosen/ebae2.htm or return to Synthesis at http://www.alri.org/Rosen/ebae.htm