Confused about validation and moderation? You’re not alone…
Written by Shashi Hodge, Managing Director of RTO Advice Group
If you’re confused about what validation means these days, no-one could blame you. Assessment validation has had a variety of meanings in VET over the years and the release of the Standards for RTOs by ASQA in 2015 brought a new dimension of meaning for RTOs to get their heads around.
Unlike in the past where many RTOs conducted validation as a process where mostly the tools and processes were checked for validity against the Training Package and unit requirements, today your validation must be a comprehensive process that considers not only your tools and processes but also whether the judgements that your assessors have been making are consistent and valid.
From talking to many of our clients and people who have recently completed the Certificate IV in Training and Assessment, confusion about the two terms, the purpose of each, and the requirements an RTO must meet today in regards to each, is rife.
Comparison between the standards and frameworks
AQTF 2007
In the AQTF 2007, the definition of validation stated that validation was a process ‘for ensuring that the way a unit of competency or group of units is assessed, and the evidence collected through these assessments, is consistent with the requirements of the unit or group of units, of competency and of industry…’ The definition also stated that validation can be ‘undertaken before, during and after the actual assessment activity occurs.’
In this early version of the AQTF, there was no specific standard or condition that stated you must conduct validation, however validation along with moderation was outlined in the User’s Guide as an approach to demonstrating that your assessment system was robust under Element 1.5, and validation at least, was certainly an expectation come audit time.
The standard industry practice for conducting validation during this period, was to concentrate on whether the assessment tools and systems covered all the unit of competency requirements. It was standard practice to compare the assessment tools and documents and map them against the unit of competency requirement, thus ensuring the tools were valid against the unit. It also considered the principles of assessment and rules of evidence, but rarely, back then, did validation include consideration of assessments completed by students. To moderate during this time meant a group of assessors met and compared their assessment decisions for the same unit, to ensure consistent decisions were made. The outcomes would bring about improvements for future assessment systems, tools and processes.
AQTF 2010
In the AQTF 2010, validation was introduced as an auditable requirement under Element 1.5 and the definition of validation was changed to:
‘Validation is a quality review process. It involves checking that the assessment tool produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course have been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes.’
In this User’s Guide, moderation was used alongside validation and the guide made it seem as though it was optional as to whether moderation was conducted. Moderation in the AQTF 2010 was defined as:
‘The process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements.’
The notable difference here is that moderation was now clarified to be an active process that was used to bring about changes to the assessor judgement, meaning that the previous way of conducting moderation would no longer work, as usually moderation was conducted long after an assessment decision had been made and the student had long-been issued their qualification or statement.
Standards for Nationally Registered Training Organisations 2012
In the SNRs 2012, no definition of either validation or moderation was provided however, like the AQTF 2010, only validation was required.
Standards for RTOs 2015
With the release of the Standards for RTOs 2015, the user guide to the Standards and ASQA’s Fact Sheet that followed, validation had a level of clarity added to its meaning, word of which does not seem to be totally understood in industry as yet.
The written definition of validation today, as per the Standards, is much the same as the AQTF 2010 however, ASQA’s user guide states that validation is a ‘review of assessment judgements made by your RTO’ and that ‘assessment validation is generally conducted after assessment is completed’. As outlined above, this is different from the definition in the AQTF 2007 which stated that validation could occur before, during, or after assessment. As per ASQA’s Fact Sheet on Conducting Validation, the reason to conduct validation after assessment is so that an RTO can consider the validity of both assessment practices, and assessment judgements.
Confusingly for many, ASQAs Fact Sheet released last year states that moderation is a ‘quality control process aimed at bringing assessment judgements into alignment’ which seems to be a hybrid of the definitions of moderation and validation from the AQTF 2010.
To provide some clarity, an RTO must conduct validation and now there are rules around how many assessments must be validated and when. Back in the day, the amount and timing was left largely up to the RTO to decide. Today, an RTO must ensure they validate a statistically valid sample of the total number of decisions made for a unit. Many may be surprised at the high volume of assessments that must now be validated using ASQA’s sample calculator.
Tips for conducting validation
- Assessment validation, to be effective, must include:
- A review of the assessment tasks, tools and documents against the unit of competency requirements to ensure that they cover all unit requirements and will allow the principles of assessment and rules of evidence to be met.
- A review of a statistically valid sample size of assessments that have already been marked by the assessor. This review is to ensure that the assessor has assessed according to the requirements of the tasks and unit/s, has documented decisions appropriately, there is sufficient evidence to support the decision, and judgements have been consistent from one student to the next.
- Ensure you have a validation tool which guides the validation process, prompts discussion where there are groups, and also demonstrates you have validated a sufficient amount of decisions. Ensure your validation tool prompts the checking of the assessment tasks and tools against the unit, the principles of assessment and rules of evidence, and requires that a range of evidence from a number of assessment decisions are compared and contrasted to determine future improvements. Make sure your tool also requires a summary of the recommended improvements to arise out of the validation findings and that there is a place for management sign off to ensure the improvements have occurred.
- To make sure your sample size is statistically valid, use ASQAs calculator to work it out, available here: http://www.asqa.gov.au/media-and-publications/conducting-validation1.html
- Make sure you use a random selection approach to choose the particular student assessments that are validated. The ASQA Fact Sheet suggests some good ways to do this
- Consider whether incorporating validation into your standard assessment process – so that every assessment is validated – may be worthwhile. This may be of benefit for those who are new to the industry or have a new course on scope – to ensure the new assessment system is working and to get more timely feedback, or it might be a good process to introduce for high-risk units. It may also be a more efficient approach to validation where you have low enrolment numbers as the sample size needs to be quite a large proportion of your assessments when you have a low number of decisions being made for a unit.
- Collect feedback from participants about the process so you can make sure you continuously improve your assessment validation processes.
- Most importantly, make sure you act on the outcomes of your validation. Otherwise the whole exercise is pointless.
Moderation today
If an RTO wants to conduct moderation today, it can, but it doesn’t have to. Moderation is a process which is to be conducted before an outcome for a unit is advised to the student. It can add to the quality control mechanisms of an RTO where it is seen to add value. Moderation can be useful where a judgment is in question, or if an assessor is unsure about their decision and needs to confer with others.
It may also be a useful strategy to use for high risk qualifications and units, such as qualifications and units from the TAE training package, as an example.
Need help?
If you need help implementing an effective validation system, need help with a validation tool, or even help with leading or conducting validation with your staff, get in touch to see how we can help you.