ChatGPT Can't Solve Questions: Here's How It Let Me Down

Two months ago, I encountered a great tweet on Twitter that focused on a course on ChatGPT Prompt Engineering for Developers.

https://twitter.com/rowancheung/status/1653422146809483264?s=20

There is an endless world of AIs, it's a great place to lose yourself. It also has a lot of growth potential. Nevertheless, instead of taking the course and starting with creating AI-Based Apps, I did nothing.

After that, I enrolled in the course

Two days ago, I rediscovered this website. It focuses on AIs, such as machine learning, deep learning or more straightforward stuff like how to build apps using ChatGPT API. Upon registering on the website, I promptly enrolled in the course I had seen on Twitter

This course is conducted by Isa Fulford from OpenAI And Andrew Ng from Stanford. I did learn a lot and gained insight.

I thought it was time to create a paid ChatGPT-based application. However, I soon realized that it was a mistake.

Creating a disappointment

It is a Telegram bot coded in Python designed to solve students' problems and provide answers. Users can send their questions or problems as images to the Telegram bot. The images are first processed by OCR.space, a free Optical Character Recognition (OCR) API. Once the images are converted into text, they are sent to OpenAI to retrieve the answers. The process typically requires 500-700 tokens, equating to an average cost of $0.001 per question. Solving five hundred questions costs half a dollar, making it an affordable option with the potential for significant profits. I could sell 500 question credits for five dollars, resulting in a 900% profit margin.

I quickly coded a demo product to test the system. At first, it was seen that it had a pretty high accuracy rate, like solving 9 out of 10 accurately. The problem started here. I only focused on the positive side while testing it. I deviated from the important aspects of this project.

In fact, ChatGPT is not able to solve problems. It often produces low-quality, inaccurate, and biased answers, which leads to poor performance. Deep down, I had read all of them previously, but I rejected them.

It took me ten hours to understand the quality and accuracy problem. Subsequently, I made the decision to establish a repository on GitHub for anyone interested.

Here are the main two functions:

#!/usr/bin/env python
import openai,requests

def create(text):
    response = openai.ChatCompletion.create(
      model="gpt-3.5-turbo",
      messages=[

              {"role": "user", "content": f"""
  Act like the best teacher in the world. Check the information well before you give it to them. Find the answer to the question given below and explain the solution in a few sentences. Also, the question is never wrong. It always has an answer: {text} """},
          ]
  )

    result = ''
    for choice in response.choices:
        result += choice.message.content
        print(choice.message.content)
    return result


# Define the OCR function
def ocr_space_file(filename, overlay=False, language='tur'):
    payload = {'isOverlayRequired': overlay,
               'apikey': API_KEY,
               'language': language
               }
    with open(filename, 'rb') as f:
        r = requests.post('https://api.ocr.space/parse/image',
                          files={filename: f},
                          data=payload)
    return r.json()

To see the all project, visit the (demo) project page.