ChatGPT Ate My Homework

Strange reflections

This was my second “through a scanner darkly” moment with LLM’s. The first being when it wrote a working website from a description.

On the day the capability to build a custom GPT was launched by OpenAI I decided as a test case to load up my MA Thesis as background information into GPT4 and use it to interrogate the thesis.

Background

With a digital education conference (DEC23) fast approaching and intrigued by the launch of custom GPTs at Open AI Dev dayI though of using my own earlier work as a test case. Elements of my MA thesis has long been annoying me as they languish unpublished and a source of academic guilt and a useful test case I though was to see if the new custom GPT could help me extract value from the thesis and spur on the development of a publication or article.

I also wanted to see what was the capability of the new models while being knowledgable about the base material to the point where hallucinations are easier to spot.

The overall goal I suppose is to explore what it will look like if the students and researchers of today really lean into the capabilities of these emerging AI technology. It’s useful I think to explore not only what its impact on assessment might be but also how it can expand and enhance our ability to learn and research. Ultimately AI will disrupt our understanding of knowledge and learning and shift I think our values around education and learning, but that’s for another post.

Process

Feed and train your new creation.

I created a new custom AI with ChatGPT and in a workflow that I think will begin to be common place the AI guides you through the process of creating the new GPT with natural language input. It asked me for a name and used its image generator to make a cheesy logo.

Wanting it to have an academic tone and be careful with its references I set it the following criteria:

’Scholarly Mentor’ adopts a formal tone, framing its responses as though it is an academic collaborator. This GPT engages with users in a manner akin to a scholarly peer, offering insights, critiques, and guidance with the depth and rigor expected in academic discourse. It leverages the works of John Creswell and David Silverman as key references, ensuring its feedback on qualitative research theses is grounded in established academic principles. ‘Scholarly Mentor’ emphasizes accuracy and scholarly referencing in its responses, employing the past passive tense for examples and suggestions. It aims to enhance the research design, methodology, data interpretation, and narrative structure of academic theses to meet high scholarly standards.- Load the background context

With tone established I then loaded up my MA thesis as a PDF as its “Knowledge” and set web browsing and code interpreter to be available.

I like your manifesto let’s put it to the testo.

With all that in place I then asked it to provide a summary of the work. It did. It did it rather well in fact.

This synopsis is very good and it is strange to see it reflected back like this. There are obviously parts of a thesis that already provide a summary put this appears original and shown it can read the document. Next I put it to work.

Great minds think alike …….

One of the take away items I had from my research was that a sense of identity as a researcher was a prime motivating factor for postgraduates and that understand developed out of the thesis, it wasn’t something I looked for going in to the work. It’s something I want to write up from the work.

So I asked my newly built GPT to analyse the thesis and draw out of it topics which might be valuable to white up for peer reviewed publication.

• Erosion of Learning Outcomes in Postgraduate Research Projects:
• Misalignment Between Research Outputs and Assessments in Arts and Humanities:
• Identity Formation Among Postgraduate Researchers:
• Impact of Research Output Types on Postgraduate Learning:
• Challenges and Opportunities in Qualitative and Mixed Methods Research Recognition:
• Professionalization of PhD Programs and Its Impact on Research Quality:
• Assessment Systems and Their Influence on Research Outputs in STEM and Humanities

There it is at number 3 on the list.

I then go on to thrash out the bones of a paper in this conversation in a manner not dissimilar to a conversation with a well informed academic colleague.

The custom GPT is available here if you want to explore further. I think a paid version of ChatGPT may be required.

Implications, challenges, opportunity.

Since doing this work back in November 2023 the world has already moved on with context windows of up to 10 million tokens and really mind-blowing stuff on AI generated video the pace of change is extraordinary. Serious grown up people are talking about AGI before the end of the decade at 50:50. How then might the world look for a PhD graduate emerging. Other than “oh crap, we may rethink all our assessments” what might be the near term academic implications of these things. I say near term because in even the medium term all bets are off. All of these postulates have HUGE academic and ethical issues but I feel are useful as “What if?” positions.

AI assistant to the researcher

Self critique of writing, argument and structure? Will we expect the researcher or student to have use an AI to critique the work prior to presenting drafts for others to read. Will the AI become your first second option? I know this approach is already becoming part of my own workflow.

AI as triangulation tool for Qual analysis? Qual analysis is a difficult and lonely task for many researchers where your own judgment is constantly under scrutiny. Could a AI assistant provide a second view on the data to assist with validation of the analysis. I have elsewhere got an AI to auto code interviews and it’s starting to get very good.

AI making previously undiscovered connections? Following on from checking human codes one of the things we are finding with AI (not necessarily LLMs) is that they can be very good at making connections that we can not in applications such as antibiotic discovery and protein folding.

AI second supervisor

Let’s bring the AI further into the project. Could a custom GPT provide an impartial critique of work? In times when emotions are getting a bit frayed in a project and the supervisor relationship getting a bit strained could an impartial AI view on the work provide a much needed firebreak or even mediate between two positions. Also much like the self critique could the GPT provide a first pass reading of drafts to reduce supervision burden.

Could it even be possible that we employ a custom GPT in a viva situation to explore the researchers understanding of the thesis and to assist our often imperfect recall of how the research is reported.

GPT as thesis

With the ability already here to make the trained GPT available to the wider world can the GPT itself replace the thesis.

Making a custom GPT trained over the entire arc of the project available could become a primary means of dissemination. As we stand we could already build a research poster that speaks for itself (note to self, next project). Such a GPT would allow other researcher to query your thesis directly through the shared GPT while presenting the authors position in terms of argument. This could help control the tone and interpretation of your work.

Who’s line is it anyway

How much further do we need to go before the custom GPT acts effectively as a co-author of your papers and would need to be credited as such. We are

building increasingly sophisticated custom GPT that could function effectively as a co-author on papers derived from your work. Where does the line of ownership get drawn, who’s research is it in the end.

Spam in a can

These positions only 2 years ago would have looked very science fiction but the pace at which they could become everyday fact is very rapid indeed. The Mercury astronauts used to be maligned as being demoted from pilot to spam in a can. We have to think hard about our relationship and use of AI in education and research because nobody wants to be the spam.