“Man is a game playing animal and a computer is another way to play games” -Scott Adams
Markeda Newell (2010) in his article entitled “Exploring the Use of Computer Simulation to Evaluate the Implementation of Problem-Solving Consultation” brings up the concern that problem solving consultation research has proliferated over the last 20 years but there is a severe lack of evidence in the use of this research by front line school psychologists.
Kratochwill, Sladeczek and Plunge (1995) in their article entitled, “The Evolution of Behavior Consultation” state that there are “more advances in research and theory than in actual implementation of consultation in practice.
One is left to wonder what questions are being used in consultation and even more importantly if the questions are the right ones” (p. 146).
Newell (2010) shares that three themes emerged from data analysis and interviews from the school psychologists that participated in this qualitative case study:
- use of behavioral approach to problem conceptualization
- individualized approach to intervention design
- minimal attention to plan evaluation.
Newell found the latter three points disturbing and shares that the divide between current research data and practice techniques used in the field may be to blame.
Gutkin (2002) shares that there is an inability to make comparative or summative analyses about consultation outcomes because different people are doing different things, which fosters uncertainty about whether consultants implement the process effectively.
Newell (2010) shares Gutkin’s premise and believes answers may come from inventive new ways to collect data (computer simulation) and that “qualitative inquiry into the consultation process can lead to better understanding of the implementation and outcomes of problem solving consultation” (p. 229).
Newell (2010) states that one of the major challenges in conducting a qualitative consultation research study in current practice settings is not being able to get a picture of the whole process that school psychologists use in creating an evaluation and implementing a support plan for their clients.
Simulation to Newell represents a midpoint between “the decontextualized artificiality of the laboratory setting and the sometimes intractable and inaccessible real world setting” (Robson, 2002, p. 363).
The beauty to Newell (2010) of using computer simulation is the ability to identify how well consultants “adhered to the problem-solving model, how they conceptualized the problem, and how they selected and evaluated the intervention” (p. 231).
The methods used in this study consisted of:
- recruitment practices used
- research design style
- procedures used
- how data analysis was collected and analyzed
Criterion sampling was used to recruit participants for the study. Patton (2002) in his book Qualitative research & evaluation methods (3rd ed)” states that criterion sampling is the process of selecting cases that meet specific criteria that can illuminate the focus of the case.
With the latter in mind, recruitment consisted of three important criteria chosen school psychologists need to have to be selected:
- work in an elementary school (case simulations were set up for elementary clients)
- trained in problem solving consultation (and really used it!)
- had access to a personal computer (Newell, 2010).
Research design style was a case study. What has been stated earlier was the need for a “whole picture” snapshot of the variables that affected choices taken by the school psychologists for their clients.
Research style such as case studies specialize focus on being able to “gather comprehensive, systematic, and in depth information about each case of interest” (Patton, 2002, p. 447). Through the use of case study methodology coupled with computer simulation, stage by stage analysis could be conducted to provide data in detail to properly research the problem-solving consultation process used by the four school psychologists chosen for this study (Newell, 2010).
Procedures used in the simulation consisted of: (1) training (simulation assistant guided training on CCD-ROM); and (2) engaging in consultation (simulated school).
Engaging in consultation consisted of three steps with the first being the Problem Identification Interview (interviewed simulated teacher in their classroom). Participants created a list of questions for the teacher about the student and responses not all ready pre-programmed would be received in about 24 hours.
The second step for engaging consultation was Problem Analysis Interview which involved having a second interview with teacher to generate hypotheses about why the behavior was occurring (comparing notes so to speak).
The simulation program also allowed for simulated parents and/or students to be interviewed if requested by school psychologists.
Lastly, the third step consisted of Plan Implementation and Evaluation in which the school psychologist provided intervention recommendations as well as plan for evaluating the intervention (Newell, 2010).
Data collection consisted of: (1) problem solving portfolios (all interviews, data requests, intervention recommendation and reflections collected and stored); (2) interviews with consultants (once completed three referral cases exit interview given); and (3) problem-solving coding protocol (measure of expected norm score verse actual score of each consultant in completing consultation model).
During the analysis process of the data, it was found that consultants identified target behaviors (I would hope so) and all of them designed intervention plans (whew) but what became painfully obvious was none of the consultants established goals for consultation process, developed goals for behavior and many had to be prompted to even develop a plan for monitoring the intervention (Newell, 2010).
The three themes discussed in the summary of this paper:
- use of behavioral approach to problem conceptualization
- individualized approach to intervention design
- minimal attention to plan evaluation will be the basis from which the insights of this research project will be discussed in regards to the use of simulated consultation.
The use of behavioral approach to problem conceptualization is not surprising. The consultants focused on when the behavior occurred, consequences of the behavior and descriptions of the behavior.
When the consultants requested data this also reflected a behavioral conceptualization of the problem. Newell (2010) states that many of the interview questions from the consultants were aimed at “operationalizing the behavior, identifying the conditions under which the behavior occurred, identifying reinforcers or rewards, and identifying strategies that have been tried to address the behavior” (p. 246).
The consultants in this study were allowed to approach the referral questions any way they wanted, any data they desired and could approach the there referral concerns in any way they like. All four of the consultants took the behavioral approach to their referrals.
Newell (2010) premises that the behavioral approach narrows the focus on the behavior itself while leaving out examination of other variables such as cultural differences, ecology of the classroom and interactions the students were having with home and school.
What may have been missing in the training of the current consultants is the idea that “a behavioral orientation is too limiting given advancements in the knowledge base about the importance of other contextual variables” (p. 250).
Though the point is well taken in updating research information missing in the background of the consultants in the study, two points must be remembered that were not stated in the study.
Point number one being the lack of time a consultant has for each student under their jurisdiction and care. A consultant may have only time to change the immediate behavior versus trying to find some deep underlying cause.
Point number two is why is there an incessant need in many cases to find causation versus relief of the immediate problem? Do we need to spend months analyzing?
Martens (1993) in his article, “A Behavioral Approach to Consultation,” shares that current problem-solving consultation is founded upon behavioral principles of human behavior and that there is significant amount of research supporting that positive results of using behavioral theory problem solving works.
Newell (2010) disagrees with Martens to a degree and premises that: “although embracing a behavioral approach is not problematic, in and of itself, this theoretical approach to problem solving raises concern about the degree to which the broader context of the problem can be acknowledged within this framework” (p. 250).
The second theme revolves around the individualized approach to intervention design the consultants created. Generally the consultants identified the same target behavior for the same cases; they varied in regards to interventions to address the behaviors.
Newell (2010) shares that problem-solving approach to consultation in of themselves should allow the consultant freedom in making decisions which will guide help the consultee/client to deal with issue at hand.
What concerned Newell was the rationale used in many of the specific cases but admits that it is difficult to evaluate right/wrong decisions.
Newell felt that many of the consultants had used what had worked for them in the past (stated in interviews) and in some instance, supported the work all ready being done by the teacher.
Kratochwill and Shernoff (2004) in their article entitled, “Evidence-Based practice: Promoting Evidence Based Interventions in School Psychology,” state: “school psychologists often rely more on their judgment when selecting interventions instead of research” (p. 35).
Again, time may have been a significant factor missing in this research project. Yes, it would be nice to be all to take the time to thoroughly research all variables in current literature studies on each particular issue but in reality, there are only so many hours in a day.
One must, as a consultant, be open to new ideas and keeping skills fresh, open-minded to change but one does not have the time to collect all research and implement for each case.
Newell (2010) concludes, “it is important to note that the lack of research behind the interventions selected combined with the limited attention to evaluation among these consultants is concerning because it’s unclear what effect, if any, the interventions will have on the target behavior” (p. 251).
Lastly, theme three discussed the minimal attention to plan evaluation by the consultants. Very little monitoring of the intervention was suggested by the consultants (disturbing) and left it up in many cases for teacher to monitor behavior creating their own plan to do so.
None of the consultants created goals for the consultation process on the targeted behaviors. Newell (2010) concludes that their findings in this study on the lack of goal setting and monitoring behavior options “is consistent with previous research indicating that evaluation is an often overlooked component of the consultation process and is one reason it is difficult to know whether or not the consultative intervention has been effective” (p. 251).
Newell (2010) shares many limitations to the use of computer simulation studies:
- use of computer simulation can be difficult to be comfortable for some than others
- computer simulation has some expense
- wait time for responses on questions not all ready preprogrammed
- consultation involves rapport building as well as responding to nonverbal cues.
Though the simulation showed potential in pointing out key misgivings by the consultants, the use of this must be further explored through a more wide range of participants (more than four) and further expanse in technology.
One could not argue to some degree what Newell (2010) concludes: “More in-depth inquiry about why and how consultation decisions are made in real-world contexts can be very useful for moving beyond whether school psychologists know the problem-solving model to assessing how school psychologists interpret the implementation of the problem-solving consultation process” (p. 253).
Gutkin, T. B. (2002). Training school-based consultants: Some thoughts on grains of sand and building anthills. Journal of Educational and Psychological Consultation, 13, 133–146.
Kratochwill, T. R., Elliott, S. N., & Busse, R. T. (1995). Behavior consultation: A five year evaluation of consultant and client outcomes. School Psychology Quarterly, 10, 87–117.
Kratochwill, T. R., & Shernoff, E. (2004). Evidence-based practice: Promoting evidence based interventions in school psychology. School Psychology Review, 33, 34– 48.
Martens, B. K. (1993). A behavioral approach to consultation. In J. E. Zins, T. R. Kratochwill, & S. N. Elliott (Eds.), Handbook of consultation services for children (pp. 65–86). San Francisco: Jossey-Bass. Newell, Markeda. (2010).
Exploring the use of computer simulation to evaluate the implementation of problem-solving consultation. Journal of Educational and Psychological Consultation, 20(3), 228-255. Robson, C. (2002). Real world research. Malden, MA: Blackwell. Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed). Thousand Oaks, CA: Sage.