With every assessment that is given, there needs to be a specific set of goals behind that assessment to make the results become meaningful and useful. Without those standards and recognized criteria, an assessment cannot be an accurate measurement of the abilities or skills possessed by the learner. While many instructors and students will spend most of their time focusing on the results, we as potential instructors would need to recognize the methods used to develop the standards by which results are measured.
The need of Standards
As previously stated, standards are required to make an accurate picture of the skill level of a learner through the results of an assessment. The actual assessment method is not necessarily important, as long as it can accurately show the performance of an individual with regards to a specific skill set.
The first order would be to define the standards that are to be identified by the assessment, and how performance indicators for these standards should be adapted to the target student population (Browder, 2003). The criteria being set need to be standard across the board, so that accurate results can be measured. Once set, various assessment methods can be applied to measure those particular performance requirements.
The example given by Browder would be methods of assessing the performance and skill level of disabled students. In this situation, passing out milk to other classmates in the morning can address standards in listening, speaking, number operations, and problem solving (Browder, 2003). This same can be said with a learning team within the University of Phoenix. Team behavior can be used to assess organization skills, team-building abilities, leadership qualities, and teamwork skills. The assessments in both examples are not standard written assessments, but yet have the same qualitative properties, if the criteria being measured are taken in context of the demonstrated skills in each activity.
With the understanding that standards for several criteria are being set and need to be reached with each assessment, it becomes necessary to define the criteria to the student. Otherwise the student will tend to become unaware of the standards they are required to reach, and thereby left to imagine their own requirements, right or wrong (Hinett, 1997). This can lead to misunderstandings that inhibit the student’s ability to perform under ideal conditions for proper assessment.
This is where rubrics become important to students. They define the standards that are required, and outline what criteria are assessed in the learning environment. This initial rubric can take the form of a complete course outline with grade expectations and assessment points that will be looked at, or it can be a simple set of instructions and rules to follow during the assessment. At each level certain standards are required identified and presented to the learners for clarification and guidance to what is expected of them.
Developing Standards
Now that we understand why standards are important and how they are implemented in an evaluation environment, it is now necessary to understand how such standards are developed. Black and Duhon (2003) identify a clear way to develop standard requirements and grant validity to assessment findings. They identify valid criteria as results to the extent which scores on the test are correlated with other variables that the instructing institution expects for associated test performance (Black, 2003). This method is generally developed as the results of previous experience of the educational institution with similar student reactions. Once the school has identified the standard they wish to set through their experience, they can then compare their findings with those of other schools with similar demographics. This presents an industry standard that is expected for all schools to reach. But suppose a new concept, technology, skill, or process is developed? How is one to identify a correct method of measurement that has the potential of standardizing criteria being assessed?
The method can easily be identified by first identifying the criteria of the assessment itself. Is there a skill that should be identified, and if so, how can it be measured? Once that concept is identified, similar methods can be used as control comparisons. The example that Black and Duhon use relates to the performance of business majors on the new Educational Service’s Test (ETS) Major Field Test in Business. The goal was to see how accurate common methods of assessing achievement would measure up to the ETS.
The criteria
The students being tested were being organized with the following criteria in mind:
1. GPA (both in Business specific courses, and overall)
2. ACT/SAT scores (Both accumulative and English/Math only)
3. Age difference
4. Gender
5. Major emphasis
The results
Once the material was gathered the following results were gathered:
1. For each Business GPA point increase, the average ETS score was 7.49 points higher.
2. For each Accumulative ACT score point increase, an average of 1.51 points increase was found for the ETS.
3. For every year increase in age, an increase of 0.71 points was average on the ETS.
4. As for Gender, males tended to score 3.79 points over women.
5. In respect to a major emphasis, those majoring in Management tended to score 3.57 points lower than all other majors, once all other criteria had been controlled.
Once the statistics have been gathered, it is now important to understand how they are significant. If there is a high correlation (+/- 0.70) between any pair of independent individuals, it indicates a statistical mean, with it’s corresponding distortions (Black, 2003). Once a mean or “collinearity” has been reached, it represents a valid, measurement that can be used as a standard towards additional results. It identifies the statistical predictions of where students will generally score based on previous experience, skill exposure, and educational background. Once that standard can be reliably measured, assessments become equally reliable.
Conclusion
So, through identifying statistical trends in scoring results, as well as the criteria that should be measured, evaluations and assessments can be used as a reliable tool for instructors to see what requirements need to be met in order to produce the best results in education. Students are also able to realize those requirements by following defining tools such as rubrics that are presented to guide them through their educational aims. It keeps them mindful of the standards required by the educational institution, and thereby keeps them focused on the skills that the course is supposed to teach them.
Resources
Hinett, Karen Review Symposium: Enhancing Learning through Self-assessment, Assessment in Education, Vol. 4, No. 2, 1997, p. 321
Browder, Diane; Spooner, Fred; Algozzine, Robert; Ahlgrim-Delzell, Lynn; Flowers, Claudia; Karvonen, Meagan What We Know And Need to Know About Alternative Assessment, Exceptional Children, Fall 2003, Vol. 70, Iss. 1, p. 45
Black, H. Tyrone; Duhon, David L. Evaluating and Improving Student Achievement in Business Programs: The Effective Use of Standardized Assessment Tests, Journal of Education for Business, Washington, Nov/Dec 2003. Vol. 79, Iss. 2, pg. 90
Showing posts with label standards. Show all posts
Showing posts with label standards. Show all posts
Tuesday, June 19, 2007
Thursday, June 07, 2007
The Quest for Standardized Certificates
The computer industry is truly a wonder. No other industry has so much innovation, grows so quickly, or comes up with such a diverse method of doing things. As such, the industry seems to lack specifics regarding how to best learn about the technology. Most people within the industry then learn by experimenting, after given some basic information from the developing company.
But what if the skills desired are not necessarily applicable to a single company, but range through an entire industry and has diverse applications? This becomes problematic, as the development of any training material or reference material can become outdated quickly, and yet the need for such material is overwhelming.
On such example is Search Engine Optimization. Currently, there are several programs out there that claim to offer certifications based on their own issuance, but there is no guarantee that the certificate is recognized as an industry standard. Because of this, Search Engine Optimization is treated like an art form, rather than a set of skills that can be applied in a given situation.
The Lack of Standardization
Standardization only comes from an overwhelming acceptance from the industry at large. This can either be done unilaterally (i.e., Microsoft Certification, Sun Certification, Red Hat Certification), through a cooperative that organizes themselves from the industry's base to form a set of skill standards (i.e., Linux Professional Institute), or from a set of recognized experts that determine the standard within their fields (i.e., standard college educational standards).
In the world of industrial training, the needs assessment is based on the skills required to do the job. Most companies conduct their own internal needs assessment, which will result in a job description. Some may even look at what other companies assign, and try to duplicate those requirements without truly understanding what the requirements mean. Either way, the needs assessment is completed based off of an individual company's needs.
How to Build Solidarity
Solidarity within an industry comes from an industry-wide recognition of the requirements for a specific position or skill. Once solidarity has been reached (i.e., network administrator tasks are identified), they can be customized based on the company's unique requirements and are more easily met by those looking to participate within that industry.
In order to gain that industry-wide recognition, it would be necessary to focus on building ties between the industry leaders, find the similarities, and focus on the skills that can bring about success as defined within the industry. This means conducting a massive task analysis across multiple companies in order to identify the standard skills that will benefit all.
The Daunting Task Simplified
While this may seem like an impossible dream, it can be achieved if identified in these generic terms:
1. What is the expected result?
2. What skills are necessary to achieve that result?
3. Which results and skills are unique to the company, and which are universal?
4. Which should be unique, and which should be universal?
5. How can the skills be best presented, so that those learning can apply them?
Once these questions are answered, a consensus has been reached to the point that training can be given, and that training has recognized industry approval. At that point, any certificate that comes with it can be seen as an industry standard. That is, provided that there is an evaluation system that can certify the skills can be practiced.
The Evaluation
All standard certificates come with some sort of standardized evaluation system. Whether it is the old reliable multiple choice test, or a more hands-on evaluation process, the results are the same. That person is now recognized by the industry to have the skills required to perform the job he was trained to do. Whether that be SEO Specialist, Linux System Administrator, or Underwater Basket Weaver, the evaluation system is necessary to identify what is being absorbed.
But that's not all! It's also an excellent way to evaluate how effective the training material is, or the reference material can be. It all comes down to the Bell Curve, that hated curve what was always incorrectly applied in High School. The Bell Curve is actually the shape of a graph showing the normal distribution where the mean, median and mode are all identical.
If the materials achieve a success rate that at least meets that normal distribution, then the material can be called a success. If it weights heavily below the acceptable level then there is something wrong with the materials or the evaluation method, and therefore needs review and revision. This is very much unlike the High School bell curve grading system, where the bell curve was applied after grading to assign grades. If that method were applied in professional training, it would do injustice to the skills required, as it does not accurately reflect learning.
But what if the skills desired are not necessarily applicable to a single company, but range through an entire industry and has diverse applications? This becomes problematic, as the development of any training material or reference material can become outdated quickly, and yet the need for such material is overwhelming.
On such example is Search Engine Optimization. Currently, there are several programs out there that claim to offer certifications based on their own issuance, but there is no guarantee that the certificate is recognized as an industry standard. Because of this, Search Engine Optimization is treated like an art form, rather than a set of skills that can be applied in a given situation.
The Lack of Standardization
Standardization only comes from an overwhelming acceptance from the industry at large. This can either be done unilaterally (i.e., Microsoft Certification, Sun Certification, Red Hat Certification), through a cooperative that organizes themselves from the industry's base to form a set of skill standards (i.e., Linux Professional Institute), or from a set of recognized experts that determine the standard within their fields (i.e., standard college educational standards).
In the world of industrial training, the needs assessment is based on the skills required to do the job. Most companies conduct their own internal needs assessment, which will result in a job description. Some may even look at what other companies assign, and try to duplicate those requirements without truly understanding what the requirements mean. Either way, the needs assessment is completed based off of an individual company's needs.
How to Build Solidarity
Solidarity within an industry comes from an industry-wide recognition of the requirements for a specific position or skill. Once solidarity has been reached (i.e., network administrator tasks are identified), they can be customized based on the company's unique requirements and are more easily met by those looking to participate within that industry.
In order to gain that industry-wide recognition, it would be necessary to focus on building ties between the industry leaders, find the similarities, and focus on the skills that can bring about success as defined within the industry. This means conducting a massive task analysis across multiple companies in order to identify the standard skills that will benefit all.
The Daunting Task Simplified
While this may seem like an impossible dream, it can be achieved if identified in these generic terms:
1. What is the expected result?
2. What skills are necessary to achieve that result?
3. Which results and skills are unique to the company, and which are universal?
4. Which should be unique, and which should be universal?
5. How can the skills be best presented, so that those learning can apply them?
Once these questions are answered, a consensus has been reached to the point that training can be given, and that training has recognized industry approval. At that point, any certificate that comes with it can be seen as an industry standard. That is, provided that there is an evaluation system that can certify the skills can be practiced.
The Evaluation
All standard certificates come with some sort of standardized evaluation system. Whether it is the old reliable multiple choice test, or a more hands-on evaluation process, the results are the same. That person is now recognized by the industry to have the skills required to perform the job he was trained to do. Whether that be SEO Specialist, Linux System Administrator, or Underwater Basket Weaver, the evaluation system is necessary to identify what is being absorbed.
But that's not all! It's also an excellent way to evaluate how effective the training material is, or the reference material can be. It all comes down to the Bell Curve, that hated curve what was always incorrectly applied in High School. The Bell Curve is actually the shape of a graph showing the normal distribution where the mean, median and mode are all identical.
If the materials achieve a success rate that at least meets that normal distribution, then the material can be called a success. If it weights heavily below the acceptable level then there is something wrong with the materials or the evaluation method, and therefore needs review and revision. This is very much unlike the High School bell curve grading system, where the bell curve was applied after grading to assign grades. If that method were applied in professional training, it would do injustice to the skills required, as it does not accurately reflect learning.
Subscribe to:
Posts (Atom)