thinking thoughts academic practice – sam barber

ARP | Action Research Project Refinement of the topic and scope

Why does this matter to me / higher education?

“Organisations that are trying to block people, or who say they are not ready are going to find their staff are doing it anyway – but without the governance, ethics, security and intellectual property controls they might have had if they’d “allowed” them“
(Jordan 2023)

the challenge of feedback and marking meets the opportunities and challenges of generative Ai

[Academics] talk … very positively about the actual work, even just in the critique of the work, ‘it was okay, but this needs … ’ and it’s full of passion and drive. That all disappears when it comes to sitting at the computer and
writing up the report. And then it becomes, you know … for want of a better phrase, ‘these bloody reports’!
(Senior Leader Interview, Arts)

Naomi E. Winstone & David Boud (2020): The need to disentangle
assessment and feedback in higher education, Studies in Higher Education, DOI:
10.1080/03075079.2020.1779687

my personal starting point

As I reached for another coffee before marking assignment number 15 of 20, my partner’s parting comment to me as he left the house for a dog walk was ”Why don’t you get ChatGPT to do the feedback for you’?

“yeah right…I don’t think it works like that” I eye-rolled into the empty space

“does it?’

maybe it does – I copied and pasted a block of text and asked a question (I now know is a called a prompt and there’s a whole emerging career path in being a chatGPT ‘prompt engineer’ )

ME to AI: can you mark this assignment……???

MY three starting points.

Something to do with AI

Something to do with assessment / marking

Something that connects to social justice

WHY?

A collision of the launch of ChatGPT, the marking strikes by university staff, and the impact of the inclusive practices unit on my own thinking, led me to reflect on how AI might impact how we assess and/or mark student work in the future and whether it would have the potential to bring a more equitable approach to marking student work.

Could AI bring rigour, fairness, and a more efficient method of providing feedback on student work, ironing out inconsistencies sometimes brought about by workload pressures or personal biases of staff.

I also felt the predominate narrative from universities was on the dangers of AI, something to be ‘managed’ to avoid problems. This was sometimes balanced by acknowledging the opportunities it brought but with an undercurrent of slight panic from institutions at the opportunity for ‘cheating’ from students. There appeared to be less discussion on how staff may or may not engage with it and here I thought there might be an opportunity to learn something to inform our discussions around the use of AI.

At the same time that I was pondering on all this I coincidentally (or probably not based on my internet search history) received an invitation to a symposium on the future of AI in Higher Education which seemed somewhat serendipitous…. or was it AI in action.

Social Justice context

I was slightly unclear how the social justice aspects of the work we have been doing on the course connected to my topic, but a conversation in one of our tutorials helped with clarity on this, combined with some reading around the topic of consistency, bias, and fairness in assessment and marking.

I had to be cautious to not disappear down a rabbit hole on the ethics of AI as there is so much to be learned and understood, however, that was not the scope of this ARP. Nevertheless, I felt it was important to understand some of the arguments and the following resources were incredibly helpful in giving me direction and inspiration:

Ethics of AI

An article by M. Healy was fascinating for me where he discusses the fundamental issues around the original design approach for Generative AI and argues that the use of labour from developing countries such as india and Kenya amounts to a form of ‘digital colonialism’

“Writing in The Guardian, Niamh Rowe (2023) notes that through a subcontractor, OpenAI paid Kenyan moderators $1.46 to $3.74 an hour to review texts and images “depicting graphic scenes of violence, self-harm, murder, rape, necrophilia, child abuse, bestiality and incest” leaving them with “serious trauma.” The colonial
implications and comparisons of a San Francisco tech company offshoring damaging work to low-paid African moderators are striking“

This created some conflict for me in terms of exploring this topic. Reflecting on the Inclusive Practices unit, I questioned whether we should be utilising ChatGPT, how will Higher Education take a position not only on the inherent challenges with this technology but also address the issues around its original design and development?

/Ellie, E. (2022) The potential of artificial intelligence in assessment feedback, Times Higher Education (THE). Available at: https://www.timeshighereducation.com/campus/potential-artificial-intelligence-assessment-feedback (Accessed: 19 November 2023).

Healy, M. (2023) ‘Approaches to generative artificial intelligence, a social justice perspective’, SSRN Electronic Journal [Preprint]. doi:10.2139/ssrn.4544617.

The Student Context

According to Tapper in the article ‘A Pedagogy of Social Justice Education: social identity, theory and intersectionality’ he discussed that ‘students’ identities need to be considered in all educational settings’, this led me to consider what role Generative AI may have in bringing fairer assessment processes.

The Times Educational Supplement article discusses the potential AI has to improve the relationship students have with assessment feedback.

“This is especially pertinent for students from widening participation backgrounds, or mature students entering higher education for the first time. Many non-traditional students have not just had complicated relationships with education; they may also have had complicated relationships with educators”

It also discusses whether the workload issues around feedback generation are ones of capacity and capability for tutors that may be ‘fixed’ by training or recruitment or whether they are issues of pedagogy, necessitating a move away from summative assessments which require heavy workload and time-consuming marking processes

Hahn Tapper, A.J. (2013). A Pedagogy of Social Justice Education: Social Identity Theory, Intersectionality, and Empowerment. Conflict Resolution Quarterly, 30(4), pp.411–445. doi:https://doi.org/10.1002/crq.21072.

more context

AI, ChatGPT, Mid-journey – we can’t go anywhere at the moment without the topic of AI and its role in our world, media, decision-making, and crucially the world of education. Some institutions are tackling it, some have their metaphorical fingers in their ears and keeping it in the too hard box.

The recent industrial action and marking boycott, the issues of fairness and inclusivity from the IP unit plus hours of marking text-based assignments over the last few months of the academic year led me to muse on the role of AI in teaching design and delivery including the topic of assessment

To manage the scope for this ARP I’ve decided to focus on marking and specifically the issue of the use of ChatGPT for feedback and informing marks for text-based assignments

I’m interested in the functional use case but also the more emotional aspects of engaging with AI

Research Questions

Starting point
How might ChatGPT be used for feedback on text-based assignments on the MADM an MAAD courses

Revised question
How might ChatGPT (LLM) work as a tool to support in the generation of feedback for text-based assignments for Masters level students at UAL?

Taking Stock

Topic is chosen

The scope is a little unclear ….needs more refinement

Half identified participants – 5-6 teaching staff across two MA courses. I’m working to keep the scope manageable in the context of my new teaching role and also marking final major projects in December and over the break.

The students are the primary stakeholders in terms of the impact on their studying however I feel focusing on the feelings and attitudes of teaching staff would be an appropriate starting point. Marking has been a significant issue in terms of recent industrial action

I’m going to do an observation and interview – comparing previously marked work with chatGPT’s output – put their own prompt in, then give them a prompt to use. Discussion during and after – semi-structured discussion

Radar chart to identify associated words – cheating, easier, relief, time ??

What do I need to do next

Write a brief for participants

Confirm participants

Identify work / prompts

Ethics consideration: remarking of work for both students and teaching staff

AI Conversations: A series of discussions surrounding the use and future of AI in Higher Education

As part of my exploratory work around the topic of AI, i attended a series of discussions organised by UAL to discuss and debate various issues around the topic of AI. I created mindmap notes on each discussion and highlighted areas of how this might either (1) add to my knowledge of AI or (2) inform the design of my research

Conversation 1
How are university students using AI?
Sue Atwell from JISC

Summary: An outline of student research that explored 5 questions with students around their use of AI

Application to my ARP: highlighted in yellow

Could the questions they used be helpful for me and transferable if they are doing peer-reviewed work?

The outcomes indicated a need to shift to teaching staff the use of AI and capability with tools and understanding the use of tools.

Inclusive Practices: Artefact

Inclusive Practices: Blog Link

Sam Barber Blog | Inclusive Practice

Inclusive Practices PG Cert | Artefact Reflection

This reflection is centered around a teaching intervention for the Collaborative Unit which is in the second term of the MA in Design Management (MADM) at LCC.

My approach to designing the artefact was to:

reflect on which elements of the Inclusive Practices (IP) unit have resonated with me most in terms of direct relevance to my teaching practice
identify an inflection point on the course where an artefact could bring a real immediate benefit to students’ learning
connect to my own personal areas of interest and connection, to bring authenticity behind why I have developed the artefact

ARTEFACT DESIGN STARTING POINTS

Reflect on which elements of the Inclusive Practices (IP) unit have resonated with me most in terms of direct relevance to my teaching practice

Engaging with Freire’s position in Pedagogy of the Oppressed (2006) gave me a starting point for the artefact design. The artefact is primarily situated in the key areas highlighted below on the adapted image of Pedagogy of Social Justice.

According to Tapper in the article ‘A Pedagogy of social justice education: social identity, theory and intersectionality’ he discussed that ‘students’ identities need to be considered in all educational settings’, this led me to consider an artefact that could give students the opportunity to share and reflect on their own individual identities, how this may impact their creative learning and their group identity.

I have placed the artefact in a unit on the course which is about collaborative working skills in design so a reflection on their group identity would be valuable. During a conversation with Sara Ekenger, Programme Lead for the MADM it was confirmed that there are 15 different nationalities, and a total of 22 languages spoken on the unit this year meaning a huge range of diversity and individual identities

A diagram of a diagram

Description automatically generated

Secondly, I wanted to explore an artefact that incorporates storytelling, we have seen many examples of this through the IP unit, for example text-based stories in Shades of Noir, Film by Christine Kum in the unit on disability and there is significant evidence that storytelling can promote inclusivity and be an enabler of effective working practice, it also underpins so many creative endeavours that it felt an appropriate area to explore.

“When people hear stories that feel representative, it creates a vehicle for nuanced conversations, which are what truly drive change. Stories invite perspective-taking: the concept of standing in someone else’s shoes and imagining what it’s like to be them”

Identify an inflection point on the course where an artefact could bring a real immediate benefit to students’ learning

I have placed the artefact in the second term of the course, students know each other better and have a more established relationship with the staff members on the course. As a result, this may encourage more open and authentic conversations about key inclusivity issues.

Additionally, the students have had one term of living in a culture that is not their own and observing themselves differently in a new city, with new peers and friendships, and an unfamiliar cultural context. Their cognitive overload may have lessened, and they may be open to sharing to encourage better collaborative working.

connect to my own personal areas of interest and connection, to bring authenticity behind why I have developed the artefact

I have spent eight years of my adult life living and working in Southeast Asia, specifically Bangkok and Singapore, including undertaking postgraduate study whilst in Singapore. On the course I was the only non-Singaporean student and yet I was probably the most comfortable speaking up and engaging with tutors. I was very conscious of my own voice and my own positionality in that group, although I would not have had those specific words for it then. It was a situation where I was both in a minority and yet there was a slightly confusing power dynamic. Singapore systems operate within a strong colonial context and influence and as a white British woman I feel this conferred some type of power within that educational context.

I have had experiences whilst working in these countries where specific, intentional discussions around identity have led to strong working relationships and experiences where that has been overlooked and feel that is led to less connectivity and creative outcomes. In my experience where people are given a framework and process through which to communicate it can bring equity to share of voice and lead to more inclusive discussions.

I need to be mindful that whilst my experience has influenced my choice of artefact I cannot assume that because I have experienced working and living in another culture that their experience is the same but in reverse. It is not directly transferable. I am also aware that my experience was as a working professional, moving with my family and therefore had a completely different support system to many of the students on my course.

THE FINAL ARTEFACT

EVOLUTION OF THE ARTEFACT

Due to the structure of the MADM course and the limitations of my teaching schedule it was not possible for me to explore using this artefact with students, however I am viewing the artefact as a method through which to reflect on issues of inclusion and diversity woven throughout the Inclusive Practices unit of the PGCert.

I secured feedback from the following people to help me with designing it further:

Shani-Louise Osei tutorials
Conversation with Sara Ekenger regarding the cohort
Feedback from Romero Bryant (Unit Lead Collaborative Unit) on the initial artefact idea
Peers on the Inclusive Practices Unit PG Cert in our sharing session

Shani suggested thinking more about which areas of diversity and inclusion the artefact would be focused on, how the intervention genuinely addresses inequity in staff student hierarchy and how to create a safe space, and finally how would I measure the outcome.

Sara was able to give me insight into the cohort demographics

Romero recognised the opportunity for this intervention, he strongly supported the concept of storytelling and the proposed timings around this. He felt the framework idea was strong and he suggested:

“Perhaps we could tie this into the collaborative unit as part of contextualising their orientation into the UK and how they may have to adapt from former ways of doing things, before even diving into the heavy theories on this unit?”

Romero Bryan

These were all helpful and constructive challenges. Whilst I may not yet have addressed all of them, the process of consultation and challenge has without doubt enabled me to go back to the literature and reapply some different concepts to my practice in terms of designing this artefact.

THE REDESIGN

I considered the guidance around ‘assumptive acts of behaviour’ from Shades of Noir (2018) curriculum design and specifically created guidance in the artefact on how to engage with one another during this activity in response to the conversation with Shani on safe spaces.

I have encouraged the practice of active listening skills, however I do recognise this may take different forms for people with unseen disabilities, for example, according to the website Autism Speaks eye contact can be challenging and stating that eye contact is a sign that you are listening maybe exclusionary for people with autism or ADHD. I have personal experience of this from a previous workshop in a professional context where I outlined eye contact as an indicator of listening and was, quite rightly, gently challenged by a participant who had mild autism. A key learning moment for me!

I had originally designed a more directed outcome for the students which was a visual representation of their group identity. However, on reflection I have changed that to be more open (see slides above). This was in response to the reading on Retention & Attainment in Art & Design (2016) and specifically the concept of the Pedagogy of Ambiguity (which is included in my blog) recognising that there is ‘neither one correct end result nor one way to get there’. This connects into the staff/student power dynamic in the classroom, my role in this exercise is to support and create a safe space for discussion, by overly directing the outcome I could be inadvertently excluding particular students and hopefully enabling a more inclusive space for creativity.

Additionally, the article ‘The power of language or the language of power’ in Shades of Noir (2018) made me stop and think about the considerations around the language that might be used around these stories. We talk of having a cohort of all international students, which, in itself, is a term loaded with complexity and huge diversity. There could be significant differences in their proficiency in English, how comfortable they feel speaking publicly, they could be from a culture where they have been marginalised , have an unseen disability or are less comfortable with a culture of public sharing of backgrounds and personal perspectives.

It would be my role as a facilitator of the session to be mindful of this and take action to ensure students do not feel marginalised and excluded compared to students for who maybe more proficient in expressing themselves in English or more confident sharing in a public space. It is important that people are not forced into sharing personal details with which they are uncomfortable, and that the storytelling does not become performative. Setting clear boundaries and expectations is important, and to be clear that the ‘’purpose of storytelling is to strengthen and improve relationships, as opposed to proving how entertaining or vulnerable you can be’’

A key consideration from the peer sharing session was how to protect the students in their sharing, particularly independently of the classroom where I would be there to facilitate. This is an excellent point and again the setting of boundaries is essential, with mutually agreed boundaries put in place by the students.

CONCLUSION
This process has been an iterative one, starting with a clear structure around what I was hoping to achieve gave me direction, and the feedback process constructive and positive. One reflection is that the Inclusive Practices unit has added a layer of nuance, inclusivity and insight which I would not have had previously.

Reference list

Autism Speaks. (2015). Why is it so hard for someone with autism to make eye contact? | Autism Speaks. [online] Available at: https://www.autismspeaks.org/expert-opinion/why-it-so-hard-someone-autism-make-eye-contact [Accessed 28 Jul. 10AD].

Ekenger, S. (2023). CONVERSATION.

Finnigan, T. (2016). Retention and attainment in the disciplines: Art and Design. [online] Available at: https://s3.eu-west-2.amazonaws.com/assets.creode.advancehe-document-manager/documents/hea/private/ug_retention_and_attainment_in_art_and_design2_1568037344.pdf [Accessed 25 Jun. 2023].

Hiscock, C. (2018). Discourse: The Power of Language and Communication. [online] Shades Of Noir. Available at: https://shadesofnoir.org.uk/journals/discourse-the-power-of-language-and-communication/ [Accessed 4 Jul. 2023].

Rezvani, S. and Gordon, S.A. (2021). How Sharing Our Stories Builds Inclusion. [online] Harvard Business Review. Available at: https://hbr.org/2021/11/how-sharing-our-stories-builds-inclusion [Accessed 1 Jul. 2023].

Shades Of Noir. (2018). Curriculum Design. [online] Available at: https://shadesofnoir.org.uk/programmes/curriculum-design/ [Accessed 14 Jul. 2023].

THE Campus Learn, Share, Connect. (2023). Managing cognitive load for EAL – and all – students. [online] Available at: https://www.timeshighereducation.com/campus/managing-cognitive-load-eal-and-all-students.

www.practice-space.org. (2020). Resource 2: Making Storytelling Inclusive and Equitable | The Practice Space. [online] Available at: https://www.practice-space.org/storytelling_guide/resource-2-making-storytelling-inclusive-and-equitable/.

ARP | Positionality

I am a lecturer on the MA Design Management course and have taught on that course as an AL and then permanent staff since September 2023. Outside of my teaching I own my own consultancy practice and have witnessed the growth in the use of AI since 2022 in the organisations I work with, and have reflected back into the University context and realised we are not exploring or challenging ourselves on our course to understand how to address this issue.

I bring industry and commercial context to my view of this technology and am aware that the ethical considerations that UAL may put around the use and adoption of AI may be more rigorous than that of industry so have tried to be aware of that, however, feel that as our students will be using these tools in the future we have an obligation to understand how we, as teachers, will use and adopt them in our practice.

The issues of fairness in marking and ensuring that feedback is personalised to all students is critical however I am personally also aware of the time that this takes and the pressures it can place on the teaching staff. I have, this week, completed marking of 15 final major projects, all of which take at least 3-4 hours to read, mark, moderate, and wordsmith constructive and positive feedback. The appeal of a ‘third marker’ who could support me, who doesn’t get tired or grouchy and is always consistent is high. Therefore this may lead me to be more sympathetic to the role that AI could play for teaching staff.

I believe in the role that digital technologies may bring to supporting students with neurodiverse conditions and the same applies to teaching staff. I have a daughter with ADHD and dyslexia and have seen first hand the frustrations and lack of confidence that can emerge from not ‘fitting’ within the educational system.

ARP | Musings

First hiccup

So first interview didn’t go as planned as confusion between myself and the participant on whether it was in person or online! Fortunately, this didn’t cause too much delay as we were due to meet the following week so able to shift things back a week!

Following the tutorial on ethics, I decided to rethink whether to survey the wider school staff, which feels too extensive for this piece of research and I also feel like I may be putting my ‘head above the parapet’ on this topic I feel if I extend out to the school. Not sure why I feel like this, and it’s not really like me to shy away from rationale conversations, I guess because I’m new to the School and feeling my way with the ‘way things are done around here’…

I will undertake the 4 tutor interviews and then consider whether to run an online session with students on the PGCert to explore the topic from a student point of view. I have discussed with four members of the course who are willing to give their time but I’m going to make a call on it after I’ve done the comparative marking exercise.

Blog Tutorial – online session

Useful session thinking more widely around blog topics – in particular how we work or can create focus – in particular for me was interesting on how to get started, how to get focused, a discussion I frequently have with students and it’s good to reflect on how I am doing this myself.

Today I have brought myself down to the co-working space of which I am a member, it’s Sunday morning, totally quiet, no-one is here and I’ve given myself three focused hours to:

update the blog
prepare for next week’s research sessions with tutors
read two articles on AI and teaching practice

There are a lot of A’s in Samantha

We also did some activities around Mnemonics, again it’s a technique I’m familiar with to support students and my own children in their exam revision but it was new to think about it as a keyword tool or a way of triggering ideas for blog posts…

S – social justice
A – artificial intelligence
M – marking
A – articulate
N – nuance
T – tutors
H – hold-up
A – academic

We discussed using these as starting point words from which to develop keywords or indeed to start blog posts – I’m going to extend them out to see where each one can be developed and then see where I can identify where to explore more.

We discussed how this might help us identify where gaps exist – I could have done with fewer A’s and an E for ethics – Samentha/Semantha!

In thinking about this gap it made me reconsider if exploring the ethical considerations could be more explicity addressed in the next interviews…. whilst I’ve always known that it was a core issue I don’t feel that I specifically explored it enough in my first interview.

KEY REFLECTION – what it has made me consider is whether to add some specific questions into my comparative research exercise around the issues of ethics – exploring how much tutors know about the problematic ethical issues of using AI or ChatGPT – most cite the issue of referencing and incorrect information however there are deeper issues around how AI has been developed, of particular interest is the paper by Myke Healy, Approaches to Generative Artificial Intelligence, A Social Justice Perspective where he explores the concept of digital colonialism.

I’m also experimenting with listening to Binaural Beats on Spotify to see if this helps with focus and concentration!

Update: I’ve moved from Binaural beats which seemed to just make me sleepy to a Focus Mix which is working brilliantly!

ARP | References/Bibliography

Bibliography

Adams, T.E., Ellis, C. and Jones, S.H. (2017) ‘Autoethnography’, The International Encyclopedia of Communication Research Methods, pp. 1–11. doi:10.1002/9781118901731.iecrm0011.

Barnett, S. (2023) ChatGPT is making universities rethink plagiarism, Wired. Available at: https://www.wired.com/story/chatgpt-college-university-plagiarism/ (Accessed: 19 November 2023).

Beyond Chatgpt (2023) Centre for Teaching and Learning. Available at: https://www.ctl.ox.ac.uk/beyond-chatgpt (Accessed: 18 January 2024).

Compton, M. (2023) Evolving AI Literacy A Shared Journey, https://mcompton.uk/2023/11/29/evolving-ai-literacy-a-shared-journey/. Available at: https://mcompton.uk/2023/11/29/evolving-ai-literacy-a-shared-journey/.

Ellie, E. (2022) The potential of artificial intelligence in assessment feedback, Times Higher Education (THE). Available at: https://www.timeshighereducation.com/campus/potential-artificial-intelligence-assessment-feedback (Accessed: 19 November 2023).

Gerritson, J. (2023) ‘Using AI to mark pupils’ work can be unfair and discriminatory – education ministry’, RNZ News, 31 May. Available at: https://www.rnz.co.nz/news/national/491047/using-ai-to-mark-pupils-work-can-be-unfair-and-discriminatory-education-ministry#:~:text=31%20May%202023-,Using%20AI%20to%20mark%20pupils’%20work%20can,unfair%20and%20discriminatory%20%2D%20education%20ministry&text=The%20Education%20Ministry%20has%20warned,discriminatory%20or%20just%20plain%20wrong. (Accessed: 15 November 2023).

Gray, C. and Malins, J. (2004) ‘Chapter 5: Intepreting the map: methods of evaluation and analysis’, in Visualizing research: A guide to the research process in art and Design. Burlington: Ashgate.

Healy, M. (2023) ‘Approaches to generative artificial intelligence, a social justice perspective’, SSRN Electronic Journal [Preprint]. doi:10.2139/ssrn.4544617.

How are university students using AI (no date). UAL. Available at: https://ual.cloud.panopto.eu/Panopto/Pages/Embed.aspx?id=60c8c1cd-955e-477f-a8a4-b09a00d9a1c2 (Accessed: 14 January 2024).

Innovative Digital Assessments (no date) Centre for Innovation in Education – University of Liverpool. Available at: https://www.liverpool.ac.uk/centre-for-innovation-in-education (Accessed: 19 November 2023).

Kumar, R. (2023) ‘Faculty members’ use of artificial intelligence to grade student papers: A case of implications’, International Journal for Educational Integrity, 19(1). doi:10.1007/s40979-023-00130-7.

Mathers, N.J., Fox, N.J. and Hunn, A. (1998) Using interviews in a research project. NHS Executive, Trent.

Shahrokh, Z.D. (2019) A short introduction to comparative research , ResearchGate. Available at: https://www.researchgate.net/ (Accessed: 19 November 2023).

Silver, H.F. (2010) Compare & Contrast: Teaching comparative thinking to strengthen student learning. Alexandria, VA: Association for Supervision and Curriculum Development.

What makes good feedback (no date) What Makes Good Feedback | Learning and Teaching @ Newcastle | Newcastle University. Available at: https://www.ncl.ac.uk/learning-and-teaching/effective-practice/assessment/good-feedback/ (Accessed: 14 January 2024).

‘AI Conversations’ (no date) in. London: UAL.

‘International Symposium on ChatGPT and the Future of Learning: Exploring the Opportunities and Challenges in Higher Education’ (2023) in. Bahrain: Gulf University.

ARP | Action Plan & Tools

Initial list

Ethics form
Draft activity plan/brief
Run a pilot with a partner
Get feedback
Run the first activity with a tutor
Feedback from a tutor on how the session was run
Refine activity
Run subsequent interviews and activity
Capture outputs
Reflect on sessions
Collate feedback
Plan next steps
Presentation slides

TOOLS used
miro |voice memo | word transcription | excel project planning

I used MIRO as a working document to capture ideas (structured and unstructured) and as a place to try to visualise findings, processes and also resources.

Sam Barber Action Research Project | MIRO (click to view MIRO board)

Action Plan

In addition to the list above I have also been using an Excel project plan.

recording & Transcription

Have also been exploring voice-to-text transcription services and which may work most effectively for these sessions – from Simon Says to using record on Word – which is what I’ve decided to trial for my first session. I will have my ipad open by the side and will supplement this with a voice recording app on my phone to ensure I don’t lose any content or can check back with the voice recording if the text recording is unclear on Word.

UPDATE on recording & TRANSCRIPTION

Transcription proved to be slightly more problematic than I had anticipated.

After trialing Otter.ai / Recording on Word in the interviews, plus several other options I eventually used an App called Voice Memo which I could use on my phone.

I could then upload the voice recording to Word on the web and it would then transcribe those voice recordings to a Word document. This was free of charge and worked really well.

Testing the session plan!

I spent an hour this afternoon with my very patient partner, as he role-played being a UAL tutor. My main objective was to run through how the research intervention would work in practice, could I use the data the way I thought, how ‘clunky’ would it feel, would the way I had designed the flow of the session facilitate effective discussion and also the questions lead to interesting insights.

The question set is open and I have the first interview tomorrow so can learn again where the conversations take us. The draft action plan and questions I designed generally worked ok however the flow was not optimal so have reorganized the structure of the session moving questions around and creating a clearer plan for me to work from during the sessions.

draft guide for the intervention

**Session plan and flow – Action and discussion points**

ARP | Research Methods & Reflections on methods

RESEARCH METHODS

Comparative Research
I decided to proceed with a comparative research approach with a semi-structured interview with four tutors from the MA Design Management Course.
I read ‘Teaching Comparative Thinking to Strengthen Student Learning’ (Silver, 2010) which outlines that comparative thinking is ‘one of our first and most natural modes of thought’ and therefore I felt would work effectively alongside a semi-structured interview which is flexible and exploratory and enables movement within a topic.

Most of the reading I could find on comparative methods tended to focus on social sciences and communication science which are not directly related to my topic however the principles of prompting conversations and observations and exploring perspectives through the mode of comparison was interesting to me.

Silver, H.F. (2010) Compare & Contrast: Teaching comparative thinking to strengthen student learning. Alexandria, VA: Association for Supervision and Curriculum Development.

The comparative method was as follows:

Use ChatGPT to generate feedback on a previously marked assignment by the tutor
Input a student assignment and also the assessment criteria
Ask the tutor to read the ChatGPT-generated feedback and discuss
Ask the tutor to then compare the ChatGPT-generated feedback to their original feedback
Semi-structured interview to discuss this comparison

Self Ethnography
I also conducted a comparative marking exercise on my own assignment for the Inclusive Practices unit for the PGCert. This would give me the opportunity to reflect on how I, as a student, felt about the comments from ChatGPT on my own work.

This type of research enables the reseracher to ‘engage in rigorous self-reflection—typically referred to as “reflexivity”—to identify and interrogate
the intersections between the self and social life’

Adams, T.E., Ellis, C. and Jones, S.H. (2017) ‘Autoethnography’, The International Encyclopedia of Communication Research Methods, pp. 1–11. doi:10.1002/9781118901731.iecrm0011.

Why semi-structured interviews?

As outlined by Mathers et al (2000) “With semi-structured interviewing, the open-ended nature of the question defines the topic under investigation, but also provides opportunities for the interviewer and interviewee to discuss some topics in more detail“
This approach would also be defined as Interactive Rationalism where the researcher “embraces ‘soft’ and flexible technical measures in dealing with the problem of how to maximize reliable responses. Techniques include continually sharing any emerging interpretations and insights with those studied, conducting interviews of various kinds and in various places” (Alvesson)
The interviewees were all people I have existing work relationships with, I know them relatively well and could adopt an informal conversational style of interview. The potential downside of this is they may also feel less comfortable being open knowing that they are
Secondary Research & Other sources
My references and bibliography are situated in a different post (ARP | references) and my secondary research predominately focused on:

(1) understanding the current landscape of AI in Higher Education and the arguments and positions around that.
(2) the principles of the research methods
(3) issues around the fairness of marking and feedback, again this is a vast topic, and therefore tried to limit my reading to

I attended several Webinars from inside and outside UAL including a Symposium on the use of AI in Higher Education

I have approached Digital Learning team within UAL on what projects or workstreams are exploring around the use of AI for teaching staff. I have found the University focus tends to be on the use of AI for students and it is less clear what work or research is being done for teaching.

Limitations and reflections:

Size of sample: I recognise this was a relatively small sample of tutors to work with, however, each interview was an hour in length (if not a little more), and due to time considerations I felt this was achievable. The analysis phase was going to take place at the same time as I had a significant Final Major Project marking to undertake for my MA course and it was important for me to manage my workload over the December period.

In terms of Action Research, I felt that this intervention was sufficient to explore initial thoughts and ideas from the teaching staff and due to the size and scope of this area of study (AI in Higher Education) I could see that there were many ways it could spiral and keeping the study small and compact felt like the most appropriate starting point. This was reinforced in our PG Cert Workshops which reassured me.

I would have loved to have included a student perspective and shared the feedback from ChatGPT and tutors with the students whose work I was using as the sample but again, they were in the middle of completing their FMPs and it would have been unreasonable to request their time for this study.

However, I do feel there is a huge scope to move this project to another phase, and may look to do this in the future if time and resources allow.

I had some set questions for the semi-structured interviews which worked well however it quickly became evident that tutors were keen to do more exploration around the capability of ChatGPT and explore within those topics to ‘test’ the technology as shown below, In general this worked effectively and allowed us to explore and let the tutor interest and concerns lead the discussion

Additional explorations that emerged through the process

ARP | Ethics

Ethical Considerations

AI is ‘awash’ with ethical issues ( see link for more commentary https://samb22.myblog.arts.ac.uk/2023/10/27/action-research-project-refinement/)

There is no single webinar, paper, or online discussion that is not prefaced or has its core content an acknowledgment of the deeply troubling and yet hugely liberating opportunities of AI. The discussions I have had with participants and with peers in education quickly get to a brain-twisting debate of the ethics and how poorly equipped many of us feel to explore these.

I was curious to find out what happens to any data I upload to ChatGPT in the form of student assignments or comments from tutors so I asked ChatGPT what it does with this data

“As of my last knowledge update in January 2022, documents or information uploaded to ChatGPT are not used to train the model. The training process for models like ChatGPT involves large-scale datasets collected beforehand, and user interactions with the model during conversations are not stored or used for training.

It’s important to note that the model doesn’t have the capability to remember past interactions or store personal data shared during conversations. Privacy and confidentiality are prioritized in the design and use of language models like ChatGPT. However, it’s always a good practice to avoid sharing sensitive or personally identifiable information in online conversations.” Chat GPT

This gave me some confidence that any information I used on ChatGPT as part of this ARP would remain confidential, however, there is still some conflicting advice on this when you perform a search. As a result, I felt it was important to request permission to use students’ assignments which were part of the comparative marking exercise

ethical enquiry form (use link)

https://artslondon-my.sharepoint.com/:w:/g/personal/s_barber_lcc_arts_ac_uk/EdGUcYO_avJKhzkz-HXhKFcBEcJjdBKx6HzsL0bEY10sDg?e=tE4eZg

Ethics Form key considerations

In terms of my ethics form there were several considerations:

The comparative exercise meant that I had to use previous students’ assignments so I wrote to all of them individually with a brief outline of the ARP and requested permission to use their work
Tutors are concerned that the comparison between ChatGPT feedback and their own may in some way raise issues of competence or professional integrity. However, this did not emerge as an issue. there were some passing comments about ‘is this going to take my job’ but as outlined in the findings, none of the tutors I worked with felt threatened but saw the possibility of supporting their roles with AI even if not with Chat GPT
Confidentiality of those tutors taking part in the ARP

I found the completion of the ethics form a useful exercise to explore and clarify in my own mind both the process and the priority ethical concerns.

Ethics Form Feedback

In terms of feedback, Lindsay provided useful thought prompts…

Hi Sam – this is looking good. You just need to have a brief think about Q9.

I had simply missed completing Q9 so updated this

I’m excited to see how these comparative exercises go. On a research design note (unrelated to ethics), I’d encourage some flexibility in the comparative exercises. Don’t feel that you have to just try it once and that’s it. Remember when we did it in the tutorial and we tried a couple of different things to see what difference it made? It’s totally ok to have a play around with it and try different things; just document as you go.

I had started to consider which parts of my approach needed to be structured and which could evolve as I went through the process.

consent form and participant information form

Participant Information Form – distributed at point of invitation to the research

ARP | Presentations

In general, I did not develop presentations for work in class but used notes and my MIRO board such as the images below. I like to work in a very iterative and visual way and MIRO works very well in this way.

Thoughts on research questions for discussion in Workshop

Final Presentation Slides

https://artslondon-my.sharepoint.com/:p:/g/personal/s_barber_lcc_arts_ac_uk /EbtPebpsOfBBqhKzmPbg070BHvBFQr-VO2TAZLzGX5hfUA?e=XGBLTX

ARP-Presentation_180224_final-1 Download

ARP | Project Findings & Analysis

secondary research
REadings
Ai in higher education symposium
AI conversations: UAL webinars

There were a number of readings which gave me direction and context for my work

I attended several Webinars and one online Conference, many of these were focused on some functional elements of the use of AI, sharing how educators and institutions were using it, however underpinning all of them was a sentiment that AI is here and we’re not leaning into it enough to understand how to embrace and use it ethically.

There are discrepancies across higher education in terms of the willingness and capability to embed it and really understanding how and who should be leading this work.

The paper by Myke Healey on Approaches to Generative Artificial Intelligence A Social Justice Perspective gave me further insight into the challenge around the design and development of some LLM tools and the idea of ‘digital colonialism’ leading to me to reflect on the compelxity for universities such as UAL where social justice is one of our pillars and yet this technology is being adopted and used by students and teachers already. This was supported by ‘AI’s English Problem’ which explored the Western bias of LLM

“ChatGPT can also make up words, struggle with syntax and generate gibberish in many underrepresented languages, Andrew Deck writes for Rest of World”,

However despite all this, surely we need to lean into exploring capability and have informed discussions on the role of AI in educations. Oxford University’s paper ‘Beyond ChatGPT’ is a really helpful document but even there the conclusions are around the engagement of AI with students and less on how teaching staff should / could / would want to connect with it.

The Russell Group issues five principles following consultation and engagement with the sector

Universities will support students and staff to become AI-literate.
Staff should be equipped to support students to use generative AI tools
effectively and appropriately in their learning experience.
Universities will adapt teaching and assessment to incorporate the
ethical use of generative AI and support equal access.
Universities will ensure academic rigour and integrity is upheld.
Universities will work collaboratively to share best practice as the
technology and its application in education evolves.

Beyond Chatgpt (2023) Centre for Teaching and Learning. Available at: https://www.ctl.ox.ac.uk/beyond-chatgpt (Accessed: 18 January 2024).

Point 1 – that universities will support students and staff to become AI literate led me to consider whether interventions such as this research could be part of such literacy training.

For thoughts on how secondary data informed by approach please see https://samb22.myblog.arts.ac.uk/2023/10/27/arp-research-methods/(opens in a new tab)

research analysis
Thematic clustering

Whilst the sample size for my research was not significant there was quite a lot of data from the conversations with tutors which needed to be segmented and themed. Braun and Clark outline that…

“Thematic analysis is a qualitative method for uncovering a collection of themes, ‘some level of patterned response or meaning’ (Braun & Clarke, Citation2006, p. 82) within a data set. It goes beyond word or phrase counting to analyses involving ‘identifying and describing both implicit and explicit ideas’ (Guest, MacQueen, & Namey, Citation2012, p. 10).

They go on to outline that it can be used for 2 – 400 participants so my sample size of 4 fits within this criteria. As this is action research we are not exploring statistically robust samples and want to understand what impact our intervention has had.

I used a method that Lindsay had outlined in the PG Cert Workshops of downloading MP3 files to Word for transcription. I then printed out the interviews and highlighted interesting comments or discussion points, using a different colour for each tutor. I then cut out those comments and clustered thematically looking for insight or topic areas.

I then used MIRO to mindmap the themes and additionally look for any commonalities or observations beyond the themes such as where differing opinions may have arisen, and where the conversations led beyond AI.

research findings

I mapped the key themes and then used a mind-mapping technique on MIRO to capture these visually

The research created positive and encouraging conversations with my colleagues on the role that AI may play in supporting teaching. It is far from simple and loaded with ethical conversations however there may be ways in which we can use it as a constructive tool to support.

The concept of a third marker in the room, a tool to test assessment criteria looking for gaps and providing ideas on how to structure are all interesting. None of these require student work to be uploaded to an AI. For tutors with neurodiversity such as dyslexia then the early indications are that this could prove to be immensely useful.

outcome presentation

I decided to do a story from a tutor’s point of view of the research conversations that would capture the key findings. I started writing a story but decided to shorten it as it felt too long and not effective enough in communicating the key findings. I wanted something I could read out in the class presentations so shortened to a poem/story.

However I feel the analysis I’ve done is also clear enough that if I wanted to do any follow on research that I would know which areas to dig into further

A tutors story

https://artslondon-my.sharepoint.com/:w:/g/personal/s_barber_lcc_arts_ac_uk/ETDJHdZwwwhKiDC1UYVl48oBm0ZjW31wO5ZF-4hxBS4VWA?e=hf6f2H

reflections on what next

I would be interested in exploring this topic further and engaging with digital learning team on what pilots or projects are being developed around the role of AI in supporting teaching. One of my colleagues on the course has also explored AI as her PGCert and tested an intervention in one of my teaching sessions. It was great to work together on that and participating in her project has continued to prompt thoughts for myself on this issue