Turns Out We Cannot Tell If a Poem is AI Written or Not

Most people cannot distinguish AI written from human-written poems. Chances are you are part of this group of people. Yes. You who read poetry as much as you eat. You for whom poetry is your cup of tea. You who probably write poems whenever the muse passes by your door.

As AI continues to gain prominence, especially in the literary space, we the literati cannot afford not to raise eyebrows. An emerging discourse is its implications on the value of art, ethical concerns over its use, and shifting perceptions of art and creativity. We are gradually conceding to the idea of AI being a co-creator of knowledge and art. Amidst this discourse, other people argue that AI will serve artists the same way a camera serves a photographer. The mere ownership of a camera does not guarantee the quality of its output. Rather, an individual’s education and social and cultural context, as well as experience, determine decision-making and modifications that go into producing the art. Similarly, even with AI, human beings still dictate the purpose and direction of art. Despite this shelter of wishful thinking, concerns still persist on ownership, authenticity, and public perception.

Recently, we have seen a number of mainstream newspapers publishing articles written by AI under someone’s name. What struck us as interesting is that the public only outrightly knew it was AI written because of the obvious AI footprints left on them. On seeing such, as concerned lovers of literature, we had several questions: How did all these errors surpass the eye of the supposed writer of the article? And what about the editor? Can the average reader tell an AI article without its footprints? And do readers really care if a piece of writing is human-written or not? In finding the answers to these questions, we embarked on literary research.

Our first episode of this series of research focuses on poetry as a literary genre. Poetry is arguably the earliest genre of literature. It is a genre that heavily relies on emotional resonance, something AI arguably struggles with. It is also one of the rare forms of writing whose work is intimately tied to its creator. As one thread from a not-so-famous Reddit user says, “If a poem is written well, it was written with the poet’s voice and for a voice.” For this and other reasons, poetry was really the first love for many of us who savour the art of storytelling. So it was easy to decide on where to start.

We sought to find out if lovers of poetry can differentiate AI written poems from human-written poems. We also wanted to know if poetry readers care if a poem is AI written or human-written. The findings were very intriguing and shocking. We shared survey forms with the public and got 33 responses. All of our respondents stated that they read poetry. 69.7% stated they read poetry often. 30.3% stated that they read poetry occasionally. A whopping 84.8% claimed to be concerned over the idea of an AI-written poem, expressing preference for human-written poems. 9.1% said they do not mind whether a poem is AI written or not. 6.5% are not sure where they stand in terms of preference on whether a poem is AI Written or not.

The survey had 4 poems. 2 were AI written while 2 were human-written. To be precise, I (Oyamo) wrote two poems and William Ogutu wrote two other poems. We then selected two of the best for this research from the four human-written poems. Each poem had multiple choices for responses: Human, AI or Not sure.

Overall, 53% of the respondents deciphered the poems correctly. 40% did not and 7% were not sure of their judgement. Of the human-written poems, 59 % were able to decipher correctly, while 32% thought they were AI written. 9% were not sure whether the human-written poems were AI written or human-written. Of the AI written poems, 47% were deciphered correctly, while 47% were thought to be human-written. 6% were not sure whether it was AI or handwritten.

The survey also requested the respondents to rank the poems if they are: Excellent, good, average, poor or very poor. Overall, the best-ranked poem was AI written. However, a human-written poem got the highest number of ‘excellent’ ratings, which was 8, while an AI written poem had 6. Only one poem was rated very poor and it was an AI written poem.

We find it interesting that 40% of the respondents, who are actually readers of poetry, could not correctly tell an AI written poem from a human-written poem. This, compared to the fact that a whopping 84.8% expressed their preference for human-written poems, does it mean then that readers are being conned into reading AI written poems? Should we have regulations to protect these kinds of readers? If yes, what would these regulations look and sound like?

To the poets of our time, I would like to pose some questions: do you use AI to compose your poems? To what extent in your use of AI does the poem cease being your creation? Does it bother you that 40% cannot decipher if a poem is AI written or Human written? Do you feel threatened by the usage of AI to compose poems? Should we have regulations to protect poets? And what would we be protecting poets from? How would those regulations look and sound?

In conclusion, AI is here with us, and it is here to stay. It has transformed creative expression, blurring the lines between human- and machine-generated art. In poetry, AI models are increasingly capable of producing verses that resemble human creativity. Is it here to make the life of a poet better or worse? It is a question we need to collectively mould an answer to. We will only do that by guiding AI on how to co-exist with us, because clearly, fighting it is futile. We need to know what moral yardstick we are going to use if we are to come up with any regulations.