ASQA, AI and the RTO Drone Mentality: This Isn't Self-Assurance

It's been a busy day and I've just gotten around to reading the latest edition of ASQA IQ, and I had another one of my face-palm moments (actually, I had a few).

ASQA has released five principles for the responsible use of AI in VET. There's nothing particularly surprising in them: providers should have appropriate governance around AI, humans remain accountable for decisions, information needs to be managed securely, AI use should support equity and inclusion, and its use should be appropriate for the training product, industry and learner cohort.

All perfectly reasonable.

Then I got to the section describing what ASQA is actually seeing from providers.

Apparently, ASQA is receiving applications and responses containing generic AI-generated wording that doesn't reflect the provider's actual operations. Some submissions make claims that aren't supported by the accompanying evidence. Some providers appear to be using AI to draft corrective actions that don't genuinely reflect what they've done to address non-compliance. Oh, and, just in case you're wondering how much human review some of these submissions have received, ASQA says it has even received material containing visible AI prompts such as "Would you like me to draft a formalised report on this?"

WTF?

I don't use that expression lightly in an article about compliance, but I'm not sure there's a more sophisticated response.

This isn't an argument against AI

I use AI to help me run my business. It can be extremely useful for interrogating ideas, drafting, organising information and improving efficiency. I don't believe professional competence is demonstrated by stubbornly refusing to use technology that can help us work better.

ASQA isn't telling providers not to use it either. In fact, its guidance specifically acknowledges its usefulness for efficiency and drafting. The concern is the apparent absence of meaningful human oversight in some of what ASQA is receiving.

If you use AI to help draft a response to ASQA, fine. But surely, SURELY, someone within the RTO reads it before it is submitted? Surely they check that the processes being described actually exist? Surely they confirm that the evidence supports the claims being made? Surely the corrective actions described in a response to non-compliance are actions the organisation has actually taken? SURELY?????

If we're at the point where the national regulator needs to remind RTOs of this, I think AI is the least interesting part of the story.

The drone mentality

I've spoken before about something I think has held parts of the VET sector back for a long time: the tendency to wait for someone else to tell us what to do.

What does ASQA want?

What wording should we put in the policy?

What template should we use?

What evidence will the auditor expect?

Just tell us what we need to do and we'll do it!

To be clear, this isn't every RTO. I work with plenty of providers and leaders who think critically about their systems, challenge assumptions, understand their risks and make considered decisions about what quality looks like in their context.

But the drone mentality is far too prominent in our sector (observe the chat box in an ASQA briefing before you argue with me on this).

Years of prescriptive compliance behaviour haven't necessarily made RTOs good at following instructions. What they have done is condition parts of the sector to search relentlessly for something they can point to later and say, “But ASQA said...”

Now we have the 2025 Standards asking providers to operate within a much stronger culture of self-assurance, and that requires something vastly different.

It requires RTOs to THINK.

To understand their own operations well enough to make decisions without needing the regulator to make those decisions for them. To look at the requirements, consider their context, interrogate the evidence, understand the risks and decide what makes sense.

And then to own that decision.

Not because ASQA told them to do it that way, or because a consultant gave them a template, or because an AI generated something that sounded convincing. Because the RTO can explain why it made the decision, demonstrate the thinking behind it and show that what it has put in place actually works.

That's the capability self-assurance demands. Self-assurance isn't waiting for ASQA to provide the answer. Self-assurance means understanding your own organisation well enough to know where your risks are, whether your systems are working and where improvement is needed.

That can't happen if the starting point for every decision is "What does ASQA want us to do?"

ASQA should establish regulatory expectations. It should provide guidance where clarification is genuinely needed. But the regulator cannot design every provider's systems, determine every appropriate control or prescribe a response to every emerging technology.

Nor should we want it to!

If every new development is followed by RTOs waiting for ASQA to publish a fact sheet telling them exactly what to do, we're not building self-assured organisations. We're reproducing the same compliance dependency under a different set of Standards.

AI provides a very good test of this. We didn't need five AI principles to know that confidential information should be protected. We didn't need ASQA to tell us that technology used in assessment needs to produce valid and sufficient evidence. We shouldn't need regulatory guidance to establish that humans remain accountable for decisions made within an RTO.

These are existing professional and organisational responsibilities being applied to a new tool. ASQA itself makes that clear: the principles don't introduce new regulatory requirements.

What concerns me more than the stray ChatGPT prompt

Leaving "Would you like me to draft a formalised report on this?" in a submission is embarrassing. But it's not the part I'd be losing sleep over.

ASQA says some AI-generated corrective actions may not genuinely reflect the provider's actual practices, improvements or evidence of implementation. That should concern RTO leaders far more than an obvious copy-and-paste mistake.

Think about what a response to non-compliance should represent: something went wrong, the RTO investigated it, people within the organisation understood why it happened and considered whether the problem extended elsewhere, appropriate action was taken, the organisation collected evidence that the action had been implemented, and considered whether it was effective.

The written response is simply the explanation of that work.

If AI is being asked to generate the corrective action before that organisational thinking has occurred, we've reversed the process. We've created the appearance of a response and then presented it as evidence that the organisation has responded.

That's compliance theatre. AI might make the theatre more convincing, but it didn't create the underlying problem.

There is some irony here

At the same time, RTOs across Australia are trying to work out what to do about learners submitting AI-generated assessment responses.

We're worried that learners might submit beautifully written answers they don't understand. We're concerned about authenticity. We want to know whether the evidence actually represents the learner's capability rather than the capability of the AI that generated it.

Fair enough.

Now apply exactly the same logic to an RTO submission.

If AI produces an impressive description of your governance arrangements, but the people responsible for those arrangements can't explain them, does that submission demonstrate organisational capability?

If it writes a corrective action that nobody has actually implemented, what evidence do we have that anything has been corrected?

If it describes processes that sound excellent but don't exist in practice, whose capability is ASQA actually assessing?

There's something more than a little uncomfortable about demanding that learners understand and stand behind the work they submit while some providers apparently aren't managing to meet that same standard themselves.

ASQA's case study shows what good judgement looks like

There's another part of ASQA's AI guidance that I think deserves attention. The case study describes an RTO exploring whether AI avatars could be used in the assessment of BSBPEF502 Develop and use emotional intelligence.

First of all: EMOTIONAL INTELLIGENCE. Just sit with that for a second.

Of all the units where you might consider replacing interaction with a real person with an AI avatar, this is certainly an interesting choice.

It also raises a question that ASQA's case study doesn't really explore the way a regulator should: why was the RTO looking for an alternative to human interaction in the first place?

ASQA tells us why the RTO explored it. Learners had previously completed role plays with trainers or volunteers acting as workplace colleagues, and these were challenging to organise. The provider decided to explore whether AI avatars could solve that problem.

And this is where my next WTF moment kicks in.

The problem wasn't that the assessment required interaction with other people. The problem was that interaction with other people was difficult to organise.

Those are two very different things, and it immediately reminded me of something I've encountered repeatedly when developing assessment tools for RTOs.

I've lost count of the number of times I've designed a simulation or other interactive assessment activity because the unit requires the learner to demonstrate interaction skills, only to receive feedback along the lines of:

"Can we change this to a case study? We have 50 students to one assessor and they don't have time to assess all of those simulations."

Again: WTF?

If the skills we're required to assess involve interacting with another human being, the fact that assessing those skills properly takes time isn't a design flaw we need to engineer out of the assessment. It's the cost of assessing the skill.

And this is where I worry about some of the conversations we're having about AI and assessment efficiency. If an RTO is looking to technology because it doesn't have sufficient trainer and assessor time to observe learners demonstrating the skills the training product actually requires, AI isn't solving the underlying problem. It's potentially helping us accommodate a delivery model that wasn't adequately resourced for valid assessment in the first place.

A case study isn't automatically an acceptable substitute for interaction because it's easier to mark. An AI avatar isn't automatically an acceptable substitute for a person because one assessor has 50 learners. The assessment method still has to allow the learner to demonstrate what the training product requires and produce evidence capable of supporting the assessment decision.

To its credit, that's exactly where the RTO in ASQA's case study eventually landed. They tested the technology rather than simply assuming that using AI made the assessment innovative or efficient, and, unsurprisingly, they found problems.

But I have to admit, if I were still a senior leader in an RTO and a team came to me after spending significant time investigating whether AI avatars could be used to assess Develop and use emotional intelligence, you've guessed it: another WTF moment.

I'd be asking why we spent so much time investigating it in the first place. I'm all for innovation, but innovation isn't about finding increasingly "creative" places to insert AI. Sometimes the title of the unit alone should be enough to make us question whether a particular technology is a sensible fit.

There's a cost to experimentation too. Staff time is finite, and I'd much rather see that capability directed towards solving genuine problems in training and assessment than trying to make a new technology work somewhere it was never particularly well suited to begin with.

Being innovative also means knowing when NOT to innovate.

The review team ultimately concluded that relying solely on the avatar wouldn't provide sufficient confidence in the assessment evidence. They didn't abandon the technology altogether; they worked out what it was actually useful for. The avatar remained a training and practice tool, while assessment included other methods involving human interaction.

They got to a defensible decision in the end, but I don't think that's the lesson we should take from this case study.

Professional judgement shouldn't only appear at the end of an experiment, once we've gathered enough evidence to confirm what the requirements of the unit should probably have told us at the beginning.

And that's where this example connects back to the wider problem I have with RTOs constantly waiting to be told what they can and can't do. We shouldn't need ASQA to publish a rule saying AI avatars are permitted for X but prohibited for Y. Nor should we need to trial every possible application of a new technology before deciding whether it makes educational sense.

We should be able to look at what we're trying to assess, apply our knowledge of assessment and the training product, and say: no, that's not an appropriate assessment method.

That's not resistance to innovation. That's having enough professional capability to know where innovation is actually useful.

And I think there is another question RTO leaders should take from this case study:

Are we using technology to improve assessment, or are we using technology to make an under-resourced assessment model possible?

If the requirements of a unit demand time-consuming observation, interaction or professional judgement, sometimes the answer isn't to find a more efficient assessment method. It's to allocate enough resources to assess it properly.

That's where the self-assurance conversation needs to go.

RTOs need to get comfortable making decisions

The 2025 Standards create an opportunity for the sector to move away from the idea that compliance means waiting for the regulator to tell us exactly what good practice looks like and then replicating it. However, that only works if RTOs are willing to take responsibility for thinking.

Use AI. Don't use AI. Use it differently in different parts of the organisation. Those decisions should depend on what you're trying to achieve, the risks involved, the people affected and the evidence you have about whether it works.

ASQA's principles are useful prompts for those conversations. They shouldn't become another checklist that removes the need to have them.

The same applies well beyond AI.

A self-assuring RTO should be able to explain why its systems work the way they do. Its leaders should understand the evidence on which they're basing their decisions. Its staff should be capable of identifying problems and thinking through appropriate responses rather than waiting for a consultant, resource provider, regulator or AI tool to supply the answer.

That doesn't mean never seeking advice. Good organisations seek expertise all the time. It means retaining ownership of the thinking.

If parts of the sector have become so accustomed to being told what to do that we're now willing to let generative AI tell us what our own organisations supposedly do, we have a much bigger problem than ChatGPT.

It won't be solved by another ASQA fact sheet.