Something is happening in the world of public policy and it ought to be happening within learning and development. Earlier this year, the government’s “Behavioural Insights Team” opened their report with something that should be learnt from and used to transform the way in which we evaluate training:
‘Randomised controlled trials (RCTs) are the best way of determining whether a policy is working…[but] RCTs are not routinely used to test the effectiveness of public policy interventions in the UK. We think that they should be.’
I’d like to second this and add that randomised control trials should be used to test the effectiveness of training interventions – whether online, in the classroom, virtual or otherwise. In learning and development, resources are tight and people need to know what works.
More and more we’re seeing clients, industry professionals and industry bodies like the Learning and Performance Institute requesting that vendors provide rigorous and demonstrable proof of the effectiveness of training. Quite rightly, there isn’t one category in either the E-learning Age or the LPI’s learning awards that doesn’t ask entrants to provide some measure of demonstrable performance improvement or return on investment.
Yet we also know how challenging measuring return on investment can be. Randomised trials are the best way to find out whether training will work.
What are randomised control trials and how can they help the L&D industry?
What makes RCTs different from other methods of evaluation is that they involve randomly assigned control groups – groups of people who aren’t given the training – which allows you to compare the effectiveness of a new intervention against what would have happened if you’d done nothing. In drug trials, the control group of people are often given a placebo or ‘sugar pill.’
So, imagine that you’ve introduced a new performance management scheme for people who are underperforming. How will you know whether those receiving the extra support might not have improved anyway? And how can you know that it wasn’t something else that improved their performance?
For example, one member of staff might be underperforming at work because they’ve been suffering from undiagnosed stress. Perhaps you introduce your performance management system at around the same time that they go to the doctor to get treatment for their stress. Their performance improves. How do you know that it was the new performance management system that caused the improvement in their performance? After all, it could have just been the help they got from the doctor.
In a randomised control trial, you would control for all the other factors which will undoubtedly also be affecting the performance of your staff.
In the fictitious example in the diagram above, we can see that those staff members who participated in the performance management intervention were much more likely to improve in terms of their individual performance than those who didn’t (in this case, twice as likely, which is pretty good). Because we have a control group (a group of people who aren’t entered into the performance management intervention), we know that it’s the intervention that’s improving the performance of staff and not something else (like the fact that a poor manager was just replaced with a good one or that they just sought treatment for an underlying medical condition like stress or depression).
In short, randomised control trials are the best way to find out if your training, rather than something else, made the difference.
Why does it need to be random? And how easy is it to run one?
The answer to the first question will be covered in my Advance article next week. The answer to the second is: randomised control trials can be much cheaper and simpler to run than you might think, partly because of the reasons given above. It’s especially important in times like these when L&D budgets are being cut. I’ll elaborate on this as well as provide a three step plan for how to go about running your own randomised control trial in an Advance article next week.
For now, it would be great if people want to comment below. Let us know:
- Have you or your organisation run a randomised control trial to test or evaluate the success of any of your training programmes?
- If so, what were the results? Did you think it was worthwhile?
- Randomised control trials have been used for about 60 years to test the effectiveness of new medicines and are being used by the UK government to test policy interventions. Does anybody have any reservations about using this approach in the training industry?
‘It’s about making a difference’ says Saffron Interactive designer shortlisted for Learning Awards 2013
Moira Nicolson of Saffron Interactive has been shortlisted for ‘Instructional designer of the year’ at next year’s Learning Awards for designing a course on mental resilience with Transport for London (TfL). The award recognises the instructional designer behind the most ‘innovative learning intervention’ which delivers a ‘demonstrable performance improvement.’
‘There are so many people in Britain today suffering from a mental illness who are too intimidated to seek help, says Moira. ‘TfL’s Occupational Health team is already doing so much to make sure that its staff are supported and I’m really pleased to have been able to create a course which has helped them to reduce their absenteeism rates by an average of one day per worker.’
The overall cost of mental illness to UK business is £26bn a year, and Labour leader Ed Miliband recently called mental illness ‘the biggest unaddressed health challenge of our age.’
The Learning Awards are organised each year by the Learning and Performance Institute (formerly the Institute of IT Training), a not-for-profit organisation which accredits leading suppliers of learning technologies. Judges include prominent figures in learning such as Nigel Paine, Brian Sutton and Alan Bellinger.
In 2010 the same award was won by Saffron Interactive instructional designer Stephanie Dedhar. Stephanie is now a regular speaker at industry conferences and a learning and performance consultant for BP. In total, Saffron Interactive has received six industry awards in the past five years.
‘We’re all very proud of Moira and the dedication she put into the Transport for London project,’ says Noorie Sazen, CEO at Saffron Interactive. ‘Doubtless she has a meteoric rise in the world of learning ahead of her, and being shortlisted for this award is a wonderful recognition of the immense value she adds to Saffron and to our customers.’
The Learning Awards ceremony takes place on 7 February 2013 at The Dorchester in London.
As U.S. political speeches go, many of us will agree that George Bush was never destined to grace us with the same public speaking prowess that seemed to come so naturally to individuals like Martin Luther King Jr. and Steve Jobs. Bush once proclaimed “You teach a child to read, and he or her will be able to pass a literacy test”.
In this rather amusing example, I shouldn’t really need to point out that he could have perhaps chosen his words a bit more wisely. However, it does raise an important point: when communicating with your audience either verbally or in writing, carefully craft your words around the information that you are trying to deliver.
An excellent article that I was reading recently whilst on my way to work illustrates my point. At the time of publishing, it was particularly relevant as Barack Obama and Mitt Romney were just about to put on their political boxing gloves and step inside the ring for their second televised debate. It analysed the different approaches that each had adopted for their campaigns; Obama, having being elected on the previous success of slogans such as ‘Hope’ and ‘Change’ decided to stick with a similar strategy communicating his vision for the future with ‘Forward.’, whilst Romney decided on a seemingly more traditional and patriotic ‘Believe in America’.
Which you prefer is up to you, but what I found interesting, and relevant to designing e-learning, is the ability that slogans have to resonate with people and communicate a much wider point without the need for any lengthy explanation. Obama selected ‘Forward.’ because it allows the everyday American to get a taste of where their potential leader wants to take the country without getting into the complications of political policies. Somehow, I don’t think that ‘I will do a better job this time round, moving America forward by investing an additional $98 billion in education‘ has quite the same effect…
My point is that learners don’t want to be bogged down by paragraph after paragraph of information when it is not necessary. We’ve found that individuals normally lose concentration within the first 10-20 minutes of an e-learning module. Make best use of this time by ensuring that everything they read or hear is delivered in a well-written and concise way that contributes to their development. And don’t be afraid of introducing creative visuals either.
In the same article that I was reading a few weeks back, I was also drawn to an infographic that quite creatively explained the more intricate details of the American election. Everywhere I looked I was rewarded with bursts of facts or figures that helped me learn. Take a look at the example above for instance (click on the image to see an enlarged version). Do you think that you would still want to read about how mobile technology has impacted on the U.S. presidential campaigns if it was neatly formatted in Times New Roman 12 on Microsoft Word?
I’m not saying that we should look to utilise infographics in all of our e-learning; some of you may even be against the idea completely. However, I want to highlight the power that a blend of creative imagery and well-thought content can have on a learner. For example, after typing in ‘infographic’ on Google, I found myself learning about all sorts of obscure topics – from how many chickens I would need to fill an asteroid a mile wide and deep to the company I should fly with if I wanted to experience my ‘happiest’ journey – purely because I was drawn in by this combination.
Whatever the subject, one of the main challenges in e-learning (and one that we experience all too often here at Saffron) is explaining the concept that more content does not necessarily mean that more will be learnt. Training shouldn’t just be about trying to overload individuals with as much subject matter as possible; it should aim to employ a range of methods in order to build engagement with the course, changing behaviours for the long-term.
Why not get a head start with your business communications or e-learning by checking out our top 10 tips:
In 1970 the futurologist Alvin Toffler wrote his best-selling book, Future Shock, predicting a society under siege from mass ‘stress and disorientation’ caused by information overload and (broadly) too much change in too short a period of time…
As it turns out, our ‘super-industrial’ society has proved surprisingly resilient to the ever-accelerating pace of technological change. If you don’t believe me, just look at this Youtube video of a nine-month-old baby who is at perfectly at ease with an iPad.
But the probably unstoppable rise of the ‘Bring your own device’ (BYOD) phenomenon is causing a little future-shock. The main victims are IT teams – and learning professionals – in the business arena. Whilst 64% of firms are expanding ‘mobility support’ according to Forrester, and many tech companies are embracing the change on a strategic level (Cisco, for example, has seen 13,000 personal iPads enter its BYOD programme so far), others are suffering.
Smartphones and tablets logged into company email servers and private networks can very easily be lost, hacked or simply stolen. And how about troubleshooting a business critical application across an eco-system consisting of dozens of different handsets and three or four operating systems? You get the picture.
But the benefits are there too, especially for e-learning. Steve Wheeler wrote in March about how the BYOD debate in education was shaping up and pointed to ‘teachers who believe that allowing students to bring their own devices to school will liberate learning’. Nowadays very few are arguing that e-learning will become less effective once learners are able to access it anywhere and anytime.
The facts on the ground are that, over the past 18 months, mobile learning has evolved from buzzword into an actually existing requirement. Important decision-makers now want assessed and interactive e-learning courses on mobile devices, especially for programmes targeting top-tier employees and graduates. Tablet learning, in short, is becoming an important factor in the war for talent. But can e-learning professionals deliver?
I’ve pinned down three areas where I think the e-learning community has some work to do if viable BYOD e-learning can become a reality.
- Is your e-learning actually mobile responsive? Really? Not just the videos? At Saffron we’re aiming to make all new courses multi-device using HTML5 technology. How you do that is a different matter. (I’ll leave the ‘to scroll or not to scroll’ debate for another post…)
- Is your learning management system mobile optimised? This one could be very tricky for some, so what’s the work-around? By the way, yesterday Udemy released an iPad app.
- If your e-learning courses live in an internal server, how will that private user information fare in the big, wide, mobile world? Secure sign-on is a problem, but a dedicated native app might be the answer. Cisco has already created an enterprise App store, for example.
Rather than solving the problem, these three pointers are intended to help frame the debate. What do you think? Is a BYOD future-shock for e-learning avoidable?