Quality issues

Approaches to Quality Assurance and Enhancement

Releasing OERs exposes individuals and institutions in a new way and may feel threatening. Concerns about quality were significant for the pilot projects, particularly the institutional ones, and often emerged as a barrier that projects had to address. Quality can be applied in both a technical and pedagogical sense and release of OERs at an institutional level provided an opportunity for existing quality measures to be reconsidered/evaluated and opened up useful dialogue across the institution.

Quality issues were interpreted as a question for the pilot programme, not a solution. Hence the question ‘What quality approaches are most supportive of open release?’ It was agreed that educational quality is the critical factor to be assessed, and all projects were concerned to know whether this has been preserved, or enhanced, by the process of open release – the concensus answer is that the process had triggered extensive reflection on the materials that did, indeed, enhance educational quality. Academic quality checks are largely provided by existing institutional or subject community processes, and some projects have argued explicitly that if OER release is to become a part of mainstream curriculum development, then the same quality processes need to be used.

it seemed counter-productive to try to introduce new QA measures into the course simply because the resources were being made public. In our view, the spirit of OERs is about sharing the work we do, rather than introducing exceptions into our practices.(ChemistryFM final report)

However, in many institutions, ‘e-learning’ or ‘online learning’ quality processes are still immature, and even in an e-learning context there may be little specialist understanding of what makes resources repurposable or reusable in other contexts. Iinstitutional QA measures generally relate to whole modules, of which the released resources were only one aspect, and within which the resources would have been used in pedagogically specific ways. So some projects have argued explicitly that it could not be assumed that the materials themselves would ‘carry’ the quality assurance into a more open context, in which they might be accessed by a wide range of different users with different requirements. This argument is more apparent in subject strand projects, and in institutional strand projects where institutions saw OER release as showcase or ‘reputation enhancement’ mechanisms and where the individual authors might feel very exposed to the high expectations of their institutions. It was less explicit in individual strand projects, where individuals are very close to their materials and may be trying to enhance their personal reputations but have fewer external expectations to meet. The response, in many cases, across all strands, has been to provide a second line of peer review within the consortium in subject-based projects, or through stakeholder and user review and beta testing for institutional and individual projects.

‘CORRE process includes: internal validation by OTTER team (proof-reading, testing links etc.), validation by contributing authors, reality check by students and feedback data gathered from users who fill in questionnaires.’ OTTER Project final report

Peer and User Review

Projects were provided with resources to help them decide appropriate quality criteria, but all in practice used their own criteria and/or allowed individual reviewers to use their professional judgement. Peer review has a number of advantages, as recorded by the projects, but it is time consuming and not all reviewers are equally experienced in designing for re-use. It may therefore be considered advisable to develop a general quality checklist for open educational resources, drawing on the work which some projects have done already, and focusing on the ‘add-on’ issues of legality, accessibility, technical interoperability, repurposability, metadata/discoverabillity, and accompanying information (incl pedagogical).

Trust has been an important issue in quality processes, with tensions apparent over the extent to which consortia or institutions are prepared to rely on the professionalism of academics either as developers or reviewers of resources.

In the end you have to rely on the professionalism of colleagues; mutual trust is a delicate commodity and it will not be developed through an emphasis on standards, unnecessary quality processes etc.(OCEP final report)

Interestingly, this issue is less apparent among individual projects, even where they are relying on the public for review.

OER quality processes need to be robust and sustainable over the longer term as open content has no natural ‘review’ cycle, unlike institutional content. Some projects have put additional review points in place in recognition of the ‘special’ status of resources for open release, while others chose to treat the content as institutional content that does go through this review cycle and this is one reason that projects were so concerned with version control and having one primary place of deposit. Some projects in the individual strand, working closely with user communities, are, instead, relying on user feedback for ongoing review, formalised by the ChemistryFM project as “public peer review”. On this model quality control and enhancement becomes an issue of tracking and responding to usage and feedback.

These issues are discussed in more detail in the strand pages:
Individual Strand – Quality Issues
Subject Strand – Quality Issues
Institutional Strand – Quality Issues

Back to Technical and hosting issues
Forward to Pedagogy and end use issues