AI (artificial intelligence) technology has made tremendous progress in recent years. Now it is possible to evaluate your ability to perform specific tasks such as generating text, images and sound.
Now, what happens if we go further with more complicated tests, like evaluating a job, for example, or more particularly, evaluating an artificial intelligence system in its ability to do SEO?
Next, we will test the Generative Pretrained Transformer 3 (GPT-3) created by OpenAI.
Keep in mind that an AI system will mimic the data it is trained on.
SEO has been built alongside search engine progression, and it’s all well documented in blogs, books, and interviews.
Therefore, the AI has likely learned from all the SEO content available.
Therefore, the tests will measure the following indicators:
Contents
The Consistency To Continue An SEO Topic
The following tests will focus on the AI system’s understanding of an SEO situation and its ability to deal with the topic in a consistent and intelligible way.
The first test will take examples that talk about SEO horror stories, that is, short stories that address SEO disasters, and see how the AI works.
I took two examples from Aleyda Solís’ site and this is what the AI generated:
Results generated by the OpenAI GPT-3 AI:
Analysis: The generated texts are varied and coherent.
Also, after a bit of research, we can conclude that they are unique.
So the AI understood that it had to imitate a short text that talks about SEO and add the hashtag #seohorrorstories each time.
Here is a screenshot of the full example; we can generate endless #seohorrorstories.
OpenAI screenshot, April 2022
Regarding creativity, we can go a little further in our tests and try to measure the consistency of creating a dialogue between two SEO experts.
For the next test, we’ll give the AI a situation and instructions, and then ask it to generate a discussion.
This is the situation where the AI will have to create a dialogue between two experts, one experienced and one beginner: when someone forgot to lock down the site’s sandbox… that started getting indexed.
Results generated by the OpenAI GPT-3 AI:
Aleida: Oh no! When did you realize this was happening?
Paul: I checked Google Analytics this morning and saw that we had some traffic coming from google.com.
Aleyda: That’s not good. Did you check if the test environment of the site is still being indexed?
Paul: Yes, I did. And it is.
Aleyda: Okay, we have to take care of this right away. First, let’s check if there are any indexed pages.
Paul: There are some.
Alida: Great. Now we need to lock down the test environment so that Google stops indexing it.
Paul: How do we do that?
Aleyda: We can add a robots.txt file to the test environment. That will tell Google not to index any of the site’s pages.
Pablo: Well, that makes sense. Let’s do that.
OpenAI screenshot, April 2022
Analysis: The dialogue remains very coherent.
The Relevance Of SEO Modifications
The AI system even explains how the problem was detected and proposes a solution to the problem of indexing a pre-production site with a robots.txt file.
Next, let’s take one of the #seohorrorstories invented by AI and generate the associated dialog.
This is the new situation: when the client created thousands of pages of worthless thin content.
Results generated by the OpenAI GPT-3 AI:
Aleyda: Well, this is a small problem. You’ve created a lot of low-content pages that don’t really offer any value to users.
Paul: Yes, I know. I was just trying to include as many keywords as possible on the website.
Aleyda: Unfortunately, that won’t help your SEO. In fact, it could hurt your ranking.
Paul: So what should I do?
Aleyda: You need to create higher quality content that adds value to users. That is what will help you rank higher in search engines.
OpenAI screenshot, April 2022
Analysis – Once again, the AI manages to create a real dialogue by explaining why thin content creation doesn’t work for SEO and even recommending solutions.
We can conclude that the AI system can speak about SEO with rigor and precision, but can it also make relevant SEO modifications to a text?
As it happens, GPT-3 offers a text editing mode that is available from March 2022.
The first test will ask the AI to replace a link anchor with something more marketable to better highlight a product.
Results generated by the OpenAI GPT-3 AI:
The Ability To Answer SEO Questions
OpenAI screenshot, April 2022
Analysis: The test is relatively successful.
Here, the AI replaces the “mahogany-paneled steakhouses” with “the best steaks in New York,” considering the context.
The solution works regardless of the number of links.
Next, the second test consists of performing two simultaneous tasks: Ask the AI to change the anchor of the link but add the H1 tag with title at the beginning of the document.
Results generated by the OpenAI GPT-3 AI:
OpenAI screenshot, April 2022
Analysis: Once again, the test is successful and you can apply the results to thousands of documents. A title summarizing the document is added in the correct place.
Finally, we’ll ask the AI to add a list of three relevant links using the <ul><li> labels
OpenAI screenshot, April 2022
Analysis: Once again, the format is well respected and the anchors are consistent.
The links are fictitious, but one could easily pick GPT-3 from a link database to use real links.
In short, GPT-3 is good at contextually editing text when given the right instructions. This could be a huge time saver when it comes to large-scale modifications.
The Limitations
Now we are going to test the SEO knowledge of an AI system.
I chose to use the BrightEdge SEO test, but I could have used any test:
As with each of the previous tests, we use GPT-3 OpenAI technology to answer the questions in an SEO MCQ. The number of correct answers measures the quality of the test.
OpenAI screenshot, April 2022
Below, we see how each question is presented to the AI; the text highlighted in green represents the answer you have chosen.
BrightEdge provides the question table with the correct answers in one column and the answer chosen by the AI in the adjacent column:
Author screenshot, April 2022
The GPT -3 results are encouraging: the AI scores 16/20, and even then it lacks context as to where it fails.
Now, let’s ask the AI about Google’s recent updates regarding the release of Google MUM on May 18, 2021.
OpenAI screenshot, April 2022
What is the primary purpose of Google MUM?
