At this stage of the inquiry, about the impact of Otago Polytechnic's initial developments in open educational practices, the strongest indication of the benefit is expressed in the return on investment (ROI). This is calculated by gains in branding awareness, educational resource quality, and savings in IT infrastructure costs. These benefits have led to the recommendation that further investment, with more organisation-wide coordination and commitment, be made. Especially in support service units, so that the impact that open education has, on learning outcomes and teacher job satisfaction across the board, can be more accurately evaluated in comparison to current standard practices.

It was difficult to establish the impact that initial developments in open educational practices had on learning outcomes without the commitment of the services across the organisation that help promote understanding, establish expectations, and support teaching and learning. It was also difficult to determine how open educational practices impact staffs' sense of job satisfaction and motivation, without the commitment needed by the organisation to recognise and reward such new practices by staff.

Gains, savings and costs

Return on investment after training and resourcing an Otago Polytechnic teacher how to use social media for open education

It is estimated to cost $4 000 to train one person how to use social media for open educational practices. At Otago Polytechnic, such a person goes on to return over $8 000 worth of savings and gains to their organisation in the first year. ($3 511.16 per year worth of brand awareness, $1 030.77 per year in quality gains by sampling and reusing free content, and up to $3 461.54 per year worth of savings in IT infrastructure, and support costs by using readily available and popular social media platforms). This study sampled the work of 13 staff, collectively returning $104 045.03 worth of savings and gains to the organisation, from an investment in training made by the organisation worth $52 000.

This return is determined by putting a dollar value to over-all gains, adding them to an estimate of over-all savings, and finally subtracting the costs, leaving just the return.

That return is divided by the number of people in the convenience sample (13), to determine a per person contribution.

ROI = (Gains + Savings - Costs) / Number of practitioners

Gains

Gains are measured for both resource quality improvements and brand awareness marketing.

Brand awareness

To determine the brand awareness marketing gains of open educational practices, the return on the existing campaigns of billboards and newspapers are calculated, and a direct comparison is made as follows:

Newspaper

Take the cost of a 1/4 page ad in the main local newspaper in Dunedin (not including design): $629.
Divide it by the distribution of 47 000 copies and assume the ad in each paper gets seen by one person.
The worth of the brand awareness campaign in the local news paper is 1c per view.

Billboard

Take the cost of one billboard: $3 490 (including banner but not including design).
Divide by the stated 19 000 cars passing in a 1 week campaign and assume each car is worth one view.
The worth of a billboard is 2c per view.

Open educational resource on popular social media

The cost of producing an open educational resource is included in the normal teaching practice after training. The cost of training is subtracted at the end of the calculation of return in savings and gains.
Assume that an educational resource is worth the combined worth of a newspaper ad and a billboard, based on it being more informative and educational.
Combine the worth of a newspaper view and a billboard view to calculate the worth of an open educational resource on popular social media.
The worth of an open educational resource is 3c per view.

Quality gains

Where an educational resource uses free media within it, this is assumed as a quality gain based on the assumption that the resource did not use media before hand due to a barrier of cost. The monetary worth of this quality gain is determined by the equivalent commercial worth of the free media sampled.

An image is worth $20 based on a low royalty fee rate.
An audio track is worth $50 based on a low royalty fee rate.
A page of text is worth $50 based on it taking one hour to produce.
A five minute video or 10 slide presentation is worth $150 based on it taking three hours to produce.

Savings

Advice from other institutions running equivalent technical infrastructure composed of wikis, blogs and Youtube (as an example of media sharing) is used to establish an estimate of savings in running costs through the use of existing social media platforms. These are true fixed costs per year for setting up and running each system (wiki, media and blog), and do not account for the worth of opportunities implicit in the community of practice built into the open platforms.

Wiki platform ~NZ$15 000 - based on Wikieducator pricing.
Media sharing site ~NZ$20 000 - based on estimates made at University of Canterbury's UCTV.
Blogging system ~NZ$10 000 - based on equivalent utility service provider charges.

Costs

A review of efforts by the Educational Development Centre to informally train staff in the use of social media for open educational use is used to establish a cost of training per person. For example:

10 hours of 1:1 intensive training, including compensation for the trainee's time: $1 000;
10 hours of 1:1 support, including compensation for the trainee's time: $1 000;
Equipment: $2 000.
Total training costs: $4 000 for each staff member.

Findings

Wikieducator savings and gains

Five staff published over 500 educational resources on Wikieducator. These resources have been viewed over 970 620 times, totaling $29 119 worth of brand awareness. By using freely licensed images and graphics, these five staff have improved the quality of their resources to the worth of $10 400. Additionally, all the staff using the free services provided by Wikieducator have collectively saved the Polytechnic around $15 000 per year in costs of setting up, running and support services for its own wiki platform.

Sample size (number of staff): 5.
Number of resources: 500.
Total data size: 60 MB.
Number of views: 970 620.
Worth of brand awareness gains: $29 118.60.
Number of free images reused: 520.
Worth of quality gains by reusing free images: $10 400.
Total worth of gains: $39 518.60 or $7 903.72 each staff member.
Worth of savings in using the Wikieducator platform: $15 000.

Youtube savings and gains

Three staff published over 95 educational resources on Youtube. These resources have been viewed over 355 600 times, totaling $10 668 worth of brand awareness. Twelve of these videos use freely licensed audio tracks worth a total of $600. Additionally, all the staff using the free services of Youtube have collectively saved the Polytechnic $10 000 per year in costs of providing that service internally (1/2 of the cost of a total multimedia hosting service when divided between Slideshare and Youtube).

Sample size (number of staff): 3.
Number of videos: 95.
Total data size: ~950 MB.
Number of views: 355 600.
Worth of brand awareness gains: $10668.
Number of free audio tracks: 12.
Worth of quality gains by using free audio tracks: $600.
Total worth of gains: $11 268 or $3 756 each staff member.
Worth of savings in using the Youtube platform: $10 000 (half of $20 000, split with the Slideshare platform).

Slideshare savings and gains

Two staff published over 94 educational resources on Slideshare. These resources have been viewed over 14 000 times, totaling $420 worth of brand awareness. By using freely licensed images and graphics, these five staff have improved the quality of their resources to the worth of $1 400. Additionally, all the staff using the free services of Slideshare have collectively saved the Polytechnic $10 000 per year in costs of providing that service internally (1/2 of the cost of a total multimedia hosting service when divided between Slideshare and Youtube).

Sample size (number of staff): 2.
Number of presentations: 94.
Total data size: ~180 MB.
Number of views: ~14 000.
Worth of brand awareness gains: $420.
Number of free images used: ~70.
Worth of quality gains by using free images: $1 400.
Total worth of gains: $1 820 or $910 each staff member.
Worth of savings in using the Slideshare platform: $10 000 (half of $20 000, split with the Youtube platform).

Blogger savings and gains

Three staff published over 102 blog posts relating to their work with the Polytechnic. These resources have been viewed over 181 281 times, totaling $5 438 worth of brand awareness. By using freely licensed images and graphics, these three staff have improved the quality of their resources to the worth of $1 000. Additionally, all staff using free blogging services have collectively saved the Polytechnic around $10 000 per year in costs of providing that service internally (1/2 of the cost of a total multimedia hosting service when divided between Slideshare and Youtube).

Sample size (number of staff): 3.
Number of posts: 102.
Total data size: ~50 MB.
Number of views: 181 281.
Worth of brand awareness gains: $5 438.43.
Number of free images used: ~50.
Worth of quality gains by using free images: $1 000.
Total worth of gains: $6 438.43 or $2 146.14 each staff member.
Worth of savings in using the Blogger platform: $10 000.

Over all savings and gains

Sample size (number of staff): 13.
Worth of branding gains: $45 645.03 or $3 511.16 each staff member.
Worth of quality gains: $13 400 or $1 030.77 each staff member.
Worth of savings: $45 000 or $3 461.54 each staff member (decreasing amount as numbers of staff increase, but cost of platform remains the same).
Total savings and gains: $104 045.03 or $8 003.46 each staff member.
Total costs: $52 000 or $4 000 each staff member.
The return on investment for developing open educational practices is 50%.

Educational outcomes

The reason it is difficult to determine meaningfully the impact that open education practices have on educational outcomes is because it is not sufficient to simply compare statistics of participation, completion and enrollment rates between closed and open courses. The wider Polytechnic services for supporting students are currently heavily biased towards the standard closed educational practices. Therefore, a comparison would only be meaningful once open educational practices become normalised within the supporting services such as: administration; student services; library; and marketing, with all services communicating a consistent message about learning in open educational contexts. Otherwise, the factors related to student support (participation, completion and enrollment rates) which are known to impact on educational outcomes may be unfairly compromised in courses using open educational practices.

There are some statistics already available about open courses. In the instance of Facilitating Online, there was a significant rise in public and international interest in the course in 2008 after it was developed into an open course. This statistic would compare to a normal marketing response rate for a course, but this was the first time international interest was shown. The drop in numbers in 2009, after initial interest and participation from the global community might be attributed to a number of factors. For example, insufficient support services and policy to assist informal learners to progress to enroll as formal learners, or costs of enrollment for international students. The subsequent rise of participation in 2010 might be an indication that the course redevelopment and administration is stabilising, and that there is more understanding of the open model in the organisation. Once an adequate level of support and policy is achieved, it is hoped that enrollment and completion rates will rise to meet the levels of interest. To achieve this, the fee to international students needs to be reviewed, and options of recognition of prior learning assessment need to be implemented.

Digital literacy in Art

Project report

In 2007, the Art School undertook some analysis of the software skills of the current students enrolled in the Bachelor of Fine Arts (Photography).

There were 95% under 30 years old, and 84% of those were between 20 and 25 years old (i.e. most students come straight from school).
Computer skills: all had basic skills, and 70% had intermediate skills.
Image-editing skills: all had basic image-editing skills, but only 10% had intermediate skills.

In 2009, the skills of the students enrolled in the senior courses in photography, who participated in the digital literacy course were at a higher level than the 2007 class.

Computer skills: 100% all had intermediate skills, and 70% had advanced skills.
Image-editing skills: 100% had intermediate skills, and 70% had advanced skills.
Additional skills: 90% students had social networking skills that they regularly used to enhance and develop their artistic practice.
Additional skills: 50% of students used additional (to photographic equipment) digital tools regularly to enhance and develop their artistic practice.
Additional skills: 90% students had a web site/web presence to market themselves.

In 2010, several improvements in digital capability were measured.

All students (100%) in the electronic arts and photography studios submitted their workbooks online.
All students (100%) in the Art School accessed their learning resources online for art history and theory.
Half the students (50%) in other senior disciplines engaged with online resources on a daily basis.
All first year BVA and Diploma students (100% ) had and maintained a blog for their studio research.
The majority of postgraduate students (80% ) maintained an online presence.

In summary.

Prior to the introduction of digital literacy in 2007, none of the students in the School of Art submitted their studio workbooks as blogs.

In 2008, this increased to 20% of photography and electronic arts students with a smattering of students in other studios engaging in online practices.

Without any significant change to students' digital literacy there probably would have been a gradual increase in the number of students engaging with online teaching materials. With support for digital literacy there has been a substantial shift in the philosophy and practice in the School of Art and this is reflected in the above statistics.

The practices of embedding digital literacy within the core components of an Art School experience have been proven to shift learner experience and give students real choice in their engagements with the online world. This is an improved educational outcome and has made a very real difference to learner experience at the Dunedin School of Art.

Facilitating Online

Year	Participants	Enrollments	Completion	Comment
2006	13	13	10	Closed course, administered externally.
2007	15	15	9	Partly closed course, administered externally.
2008	89	9	13	Open course, administered externally.
2009	39	15	15	Open course, change in facilitators, administered externally.
2010	70	8	4 formal, 9 informal	Open course, administered by Otago Polytechnic.

Flexible Learning

This course is administered by Otago Polytechnic.

Year	Participants	Enrollments	Completion	Comment
2006	0 informal	15	12	Closed course
2007	0 informal	15	12	Partly closed course
2008	7 informal	20	15	Open course
2009	1 informal	20	14	Open course
2010	5 informal (4 since enrolled)	26	21 (3 in progress)	Open course

Permaculture Design

This wiki page is used to enhance the face to face teaching.

Year	Participants	Enrollments	Completion	Comment
2008	0 formal, 15 informal (5 online)	0 formal	10 informal	Informal input to help develop the course. Development discussions/comments can be found here and here.
2009	9	9	9	Open course with five participating from outside the Polytechnic. Four were already full time students in Horticulture.
2010	15	15	13	Open course with two participating from outside the Polytechnic. Thirteen already full time students in Horticulture.

Travel and Tourism

Year	Participants	Enrollments	Completion	Comment
2009	23	23	20	Open course
2010	24	24	23	Open course

Anatomy and Physiology of Animals

Year	Enrolled students	Informal students	Textbook sales	Comment
2008	274	4	1	Text book developed on Wikibooks, provided free to enrolled students,sale on Lulu.com for informals
2009	283	28	22
2010	305	141	115

Research study: Staff Perspectives on the Practice of Open Education

A research study was commissioned by Otago Polytechnic to a third party research team to establish how open educational practices were understood by Otago Polytechnic staff, and how they were impacting on their sense of job satisfaction, motivation, and their perceptions of teaching and learning performance (values). The research resulted in the University of Otago Report for Otago Polytechnic: Otago Polytechnic Staff Perspectives on the Practice of Open Education. Prepared by Russell Butson, Shelagh Ferguson, and Carla Thomson in May 2010.

Five participants took part in the research which used a videography methodology. A wider sample, would have provided more certainty regarding the findings, however, some valuable information was revealed. For example, there was a clear connection between participants' beliefs of Open Education and their actual practice. However, there appeared to be a level of immaturity in the participants' understanding of open educational practices, for example, sharing open resources. Participants spoke exclusively about the resources they had created and shared, and made no reference to using open resources from other providers. Three stages of practice in the use of open resources were found:

no interest in sharing resources;
an interest in sharing their resources with other; and
an interest in open reciprocal sharing of resources.

The impact of open education practices on job satisfaction, motivation, and perceptions of teaching and learning performance were not clearly ascertained. The findings indicate there is a need for strategic commitment to open education, and how it is supported, recognised and rewarded.

Introduction

This small scale ethnographic study, as part of the larger Ako Aotearea study, was designed to investigate the ways in which a convenience sample of teaching staff understood and interacted with open education resources. The research sample did not seek to be representative of the Polytechnic’s teaching staff, but represented informants who could give particular insight and depth to the research. The researchers aimed to determine staffs' views on the use of open education resources, and to gain insights into how staff understood and valued their work in open education. To this end, a number of staff members took part in videotaped interviews where they discussed and demonstrated their open education practice.

It is important to note that the research was undertaken within the backdrop of the Otago Polytechnic’s open resource intellectual property policy which uses a default Creative Commons by attribution copyright. The staff interviewed were clearly aware of the policy and felt encouraged and supported in their quest to explore the open sharing of resources.

Rationale for Videography

The researchers wanted the data to be determined by the beliefs and experiences of the participants, and therefore, did not wish to employ a method that imposed an external theoretical framework to define or compartmentalize the data they were seeking. Videography is not premised on the existence of an underlying conceptual scheme, or deep structure for explaining behaviour, rather it allows researchers to capture a vast array of thoughts, ideas, beliefs and expectations as they occur in mutual exchange and negotiation. As a methodology, videography allows the capture of rich and thick data that gives insight into practice and underpinning values.

Ethical Considerations

Ethical approval (Category B) for this study was provided by Higher Education Development Centre, University of Otago. Before participants were interviewed, they were provided with ethics approved information sheets and consent forms (Appendix 5). They all gave written consent to be part of this project.

The participant interviews were video recorded and partially transcribed. The video files were only available to the researchers. The actual video records were not included in any research outputs, and the transcriptions were altered to remove any identifying information prior to use in research outputs (reports, publications, and conference papers). Those participants who requested it were given digital copies of their video interview. To maintain anonymity, the participants are represented, in this report and in any written research outputs, by anonymous markers.

Recruitment/sampling

This project employed purposeful sampling. To achieve the aim of gaining some deep understanding into staff views and experiences with open education, the researchers built a convenient sample from staff who had used open education resources. Eight faculty members who were involved in the provision of open education resources were invited via email to be interviewed. Five agreed and were interviewed.

Data Collection

Interviews were conducted in the participants’ work space (home or office). One researcher conducted the interview while another videoed it. The interviews were semi-structured, in that the researcher had a predetermined set of questions or interview schedule. However, the style of the questions was open-ended allowing for responsiveness to the lead of the interviewee, and some latitude in terms of relevance. In addition to answering interview questions, the interviewees demonstrated their practice with open education resources online.

Data Analysis

The participants’ video recordings were dealt with through an iterative process in which the researchers viewed, annotated, partially transcribed, and coded the video files according to emerging themes. These processes were facilitated by NVivo software. After gathering and working with data on the level of individual participants, the research team employed an inductive, iterative analytical approach. Under this approach, the data was subjected to multiple readings. This allowed for the identification of themes within and across the video records and interview transcripts (see appendix 1,-4). Using a constant comparative approach , the researchers moved backwards and forwards among the data, gradually moving from coding to conceptual categories and then to theory development.

Outcomes/Findings

The length of the four interviews ranged from 40 minutes to just over an hour long with 238 minutes (3.97 hours) in total of interview footage being recorded. Each semi-structured interview began with the interviewer asking about the participant’s philosophy of teaching and learning. Depending on the direction established by the response to this initial question, the participants moved on to discuss, over the course of the interview, their perceptions of open education, and their reasons for using open education resources. They also demonstrated and discussed the ways they used open education resources, and specific functionality/applications, for example, blogs, and Youtube.

These general commonalities notwithstanding, each participant had a different focus, or position in relation to open education. Consequently, the interviews each had a very different ‘flavour’ and the resulting video data were rich and diverse. Four overarching/main themes emerged from the interviews.

Rationale for Open Education.
Practice of Open Education.
Resistance to Open Education.
Beliefs about Teaching and Learning.

The majority of the interview data was categorised according to these themes. All the participants made interview comments which fell under each theme. Table 1 below shows the total number of comments made in relation to each theme, and the percentage of the total video footage which these comments covered. For example, the participants made 106 comments which were coded at the ‘rationale for open education’, covering 35% of the total interview footage.

Table 1: Interview comments and coverage according to theme

Theme	Comments	Coverage
Rationale for Open Education	106	35%
Practice of Open Education	95	29%
Resistance to Open Education	54	18%
Beliefs about teaching and learning	41	11%

As can be seen in Table 1, rationale for, and practice of open education were the focus of a greater percentage of interviews compared to resistance to open education and beliefs about teaching and learning.

Evidently having a sound rationale of open education is likely to result in behaviours that are clearly aligned with open education practice. While this result is only built on the experience of five people, the researchers believe the self reports of these participants, as they expressed their beliefs and outlined their practice, revealed a sound coupling between having a rationale and actual practice. Their conclusion on this point is that for Open Education to work, the organisation needs to signal a clear message (e.g., through policy) that encourages and supports the practice. Secondly the organisation needs to engage its population in promoting a clear understanding of the rationale and value of this approach. Finally, the adoption by the organisation of an ‘open’ schema would result in a change in practice.

The participants in this study, while having a clear rationale, appear to be a minority within the institution. It is possible that the lack of champions (staff with a clear rationale for Open Education) may be the reason Otago Polytechnic, regardless of its institutional support for open resources, appears to have poor uptake regarding actual open education practice. However, the research sample was small and further research is needed to validate this observation.

It is interesting to note that the participants spoke exclusively about the resources they had created and shared. There was no reference to using open resources from other providers. The impression the researchers were left with was that staff interviewed had all moved from a ‘no interest’ state to one of ‘interest’ (Figure-2). With only one of the participants showing any interest or understanding of resource sharing as a collaborative reciprocal process, the researchers concluded that the four other staff members were at what is defined as stage-2: a rather egocentric state where the focus was exclusively on their resource. One other participant had moved away from this position, and was clearly more aware of the benefits when open resources were both shared and sourced.

Figure‐2: Stages of practice in the use of open resources.

While the above schema (Figure-2) emerged from analysis of the data, the researchers cannot say for sure, without further research, that the three states are necessarily part of a continuum. They foresee that it is likely that staff could move along this continuum with the right type of support and development aimed at facilitating a change in attitude and hence practice. It is worth mentioning that the discussions in this study were around open resources, and the acknowledgment of authorship associated with these. The researchers did not discuss concealed use of resources that may occur at stage-1, and which are often repackaged and do not include acknowledgment to the original author/creator.

The findings suggest two essential elements are important if an institution wishes to promote the open development and sharing of educational resources. Firstly, institutional support (ref-OP policy of OE) for open distribution of educational resources is required, and secondly shared understanding of, and rationale for the free distribution of educational resources is necessary. It is not unusual in studies looking at beliefs and practice to find a disconnect between these aspects, and one that is often comfortably maintained. However, what is interesting in this study is the clear connection between the participants' beliefs of open education and their practice.

To finish a key issue is raised which the researchers struggled with: what actually is Open Education? Firstly, the participants did not raise this as an issue. There appeared to be a general understanding about the meaning of Open Education which allowed all the participants to talk from a similar perspective. It was paramount for the researchers to 'dig a bit deeper' regarding the scope of Open Education in order to develop an understanding prior to any analysis. It became clear to them very early on that Open Education was a parent idea that incorporated such practice as resource sharing (Open Education Resources). While the participants used Open Education on many occasions – the researchers believe they were actually referring to Open Education Resources. In fact all of the interview material was essentially focusing on the open sharing of resources. This raised the issue of whether to define education as resources or by programme. The question arose, were the resources valuable on their own, or did they only make sense when embedded within a programme?

Answering this question was not part of this research, but was something that surfaced as a result. The researchers hope that further research into Open Education can explore this issue more fully.