What do we mean by ‘evidence’ in the context of our programme-level evaluation.
What we mean by ‘evidence’ is anything that helps to confirm or illustrate or exemplify a message. A message may be arrived at through formal evaluation processes – when it is likely to be supported by data collected and analysed for that purpose – or through personal reflection, or as part of the emerging collective ‘story’ of the project.
During the pilot phase we used the generic framework to define the problem space in which we expected messages to emerge. We know that unanticipated outcomes, messages and stories can be equally valuable to those we set out to collect – but these too require evidence to support and illustrate them. This framework has been developed throughout the programme and adapted to reflect phase 2 – Phase2 synthesis framework and phase 3 – phase 3 evaluation framework Both of these versions of the framework also linked to evidence gathered during those phases.
We don’t want to tell you what evidence you ‘ought’ to bring forward to support your key messages, only that the better evidence we have as a programme, the more powerfully we can communicate those messages. Creative forms of evidence such as multimedia, talking heads, screenshots and images have an important role to play in communicating project outcomes. However, the type of evidence needs to match the type of message or the claim being made. For example, if we are claiming that OERs can save on overall development costs, we need fairly robust, quantitative data to back up that message. If we are saying that a number of issues seem to motivate staff to release open resources, we need a different kind and quality of evidence: quotes from staff – written or spoken – are appropriate to illustrate the issues involved.
Evidence generated by complex and innovative processes such as OER release is often itself complex, context-specific and difficult to generalise. This is fine, providing the complexity and the limits of generalisability are acknowledged. Remember that what might be ‘just anecdotal’ evidence in the context of one project, if it is repeated across 20 or 30 other projects, becomes clearer, stronger and more credible evidence at programme level.
Examples of ‘messages’ and ‘evidence’ are summarised in the table below. There are many more, and we will be adding to this list of suggestions.
Notice that any of these project level messages, if repeated across enough projects, lend themselves to being expressed in a more general way, e.g. ‘There is evidence that involvement in OER projects builds staff skills’ ‘There is evidence that senior management involvement can overcome barriers to OER release’ etc.
Example evidence for OER projects |
|
Message |
Example evidence |
Being involved in this project has helped staff to develop their skills |
|
Senior management support has helped to overcome barriers to open release |
|
This project has saved on overall development costs of learning resources |
|
Releasing OERs is enhancing the reputation of the course/dept/institution |
|
Personal recognition has been a strong motivator for individual staff |
|
The OERs released are of at least as good quality as materials produced for ‘closed’ teaching contexts |
|