Of the last 7 tests, conducted daily, this test has passed 71.0% of the time.
-
Today's request cost $0.009
-
-
-
-
-
Method
-
We send a image of a handwritten note to determine if it can correctly read the text. If it correctly gets the text, it gets a 100%. Otherwise, it gets a 0%.
-
Prompt
-
- Read the text in the image. Return only the text, with punctuation.
-
-
Image
-
-
Result
-
The words of songs on the album have been echoing in my head all week. "Fades into the grey of my day oh tea."
Of the last 7 tests, conducted daily, this test has passed 0% of the time.
-
Today's request cost $0.02
+
Today's request cost $0.021
@@ -417,43 +363,56 @@
Prompt
Image
Result
-
To determine if there are any missing annotations, I'll count the visible cars in the image and compare that with the number of red bounding boxes.
+
The image provided contains visible cars annotated with red bounding boxes. I'll manually count the cars and the annotations to find any missing annotations.
+
+1. Cars in the image:
+ - 1 car (left oncoming lane, closest)
+ - 2 cars (next in line after the front car in the oncoming lane)
+ - 3 cars (close proximity in the far center lane)
+ - 1 car on the rightmost lane (closest to the camera)
-In the provided image:
+Total cars: 7
-1. Visible cars (including partial and fully visible):
- - Front white car (on the right).
- - Black SUV to the left.
- - Several distant cars in the middle lane.
- - Additional cars that may be hard to distinguish clearly.
+2. Annotations:
+ - 1 car (left oncoming lane, closest)
+ - 1 car (next in line after the front car, on the left)
+ - 3 cars (in the far center lane)
+ - 1 car on the rightmost lane (closest to the camera)
+
+Total annotations: 6
-2. Red bounding boxes present:
- - There are 8 red bounding boxes.
+The car closest to the camera on the right lane isn't annotated.
-Here, the number of cars in the image appears to be accurately annotated with red boxes. Therefore, there are no missing annotations.
+Therefore, there is 1 missing annotation.
-JSON output:
```json
{
- "missing": 0
+ "missing": 1
}
-```
-
-This indicates that there are no missing annotations in the provided image.
Can GPT-4V classify an image without being trained on that particular use case?
-
-
Fail
+
+
Pass
@@ -462,62 +421,48 @@
Measurement Test
Last 7-Day Performance
-
+
-
+
-
+
-
+
-
+
-
+
-
Of the last 7 tests, conducted daily, this test has passed 14.0% of the time.
-
Today's request cost $0.009
+
Of the last 7 tests, conducted daily, this test has passed 100% of the time.
+
Today's request cost $0.005
Method
-
This test gives GPT-4 a image of a square sticker with a ruler on one side and asks it to provide a number for the length and width. We score this test based on precent error, gauging how far it is from the actual value.
+
We check to see if the model can correctly identify the vehicle. If it can, it recieves a 100%, if it is incorrect, it recieves a 0%.
Prompt
- What is the length and width of this square sticker, based on the ruler? Return a JSON with float properties for `length` and `width` representing inches.
+ What is in the image? Return the class of the object in the image. Here are the classes: Toyota Camry, Tesla Model 3. You can only return one class from that list.
Can GPT-4V classify an image without being trained on that particular use case?
+
Document OCR
+
Can GPT-4V read a document and return the exact characters in the text?
@@ -547,31 +492,31 @@
Zero Shot Classification
Of the last 7 tests, conducted daily, this test has passed 100% of the time.
-
Today's request cost $0.005
+
Today's request cost $0.009
Method
-
We check to see if the model can correctly identify the vehicle. If it can, it recieves a 100%, if it is incorrect, it recieves a 0%.
+
We send a screenshot of a typed document to determine if it can correctly read the text. If it correctly gets the text, it gets a 100%. Otherwise, it gets a 0%.
Prompt
- What is in the image? Return the class of the object in the image. Here are the classes: Toyota Camry, Tesla Model 3. You can only return one class from that list.
+ Read the text in the image. Return only the text, with punctuation.
Image
-
+
Result
-
Toyota Camry
+
I was thinking earlier today that I have gone through, to use the lingo, eras of listening to each of Swift's Eras. Meta indeed. I started listening to Ms. Swift's music after hearing the Midnights album. A few weeks after hearing the album for the first time, I found myself playing various songs on repeat. I listened to the album in order multiple times.
Can GPT-4V read a document and return the exact characters in the text?
+
Handwriting OCR
+
Can GPT-4V read handwriting?
@@ -586,7 +531,7 @@
Document OCR
-
+
@@ -596,31 +541,31 @@
Document OCR
-
+
-
Of the last 7 tests, conducted daily, this test has passed 100% of the time.
+
Of the last 7 tests, conducted daily, this test has passed 71.0% of the time.
Today's request cost $0.009
Method
-
We send a screenshot of a typed document to determine if it can correctly read the text. If it correctly gets the text, it gets a 100%. Otherwise, it gets a 0%.
+
We send a image of a handwritten note to determine if it can correctly read the text. If it correctly gets the text, it gets a 100%. Otherwise, it gets a 0%.
Prompt
Read the text in the image. Return only the text, with punctuation.
Image
-
+
Result
-
I was thinking earlier today that I have gone through, to use the lingo, eras of listening to each of Swift's Eras. Meta indeed. I started listening to Ms. Swift's music after hearing the Midnights album. A few weeks after hearing the album for the first time, I found myself playing various songs on repeat. I listened to the album in order multiple times.
+
The words of songs on the album have been echoing in my head all week. "Fades into the grey of my day old tea."
Of the last 7 tests, conducted daily, this test has passed 100% of the time.
-
Today's request cost $0.016
+
Today's request cost $0.015
@@ -728,7 +673,66 @@
Result
-
+
+
+
+
+
Measurement Test
+
Can GPT-4 Measure Items Using a Reference?
+
+
+
+
Pass
+
+
+
+
+
+ Last 7-Day Performance
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Of the last 7 tests, conducted daily, this test has passed 14.0% of the time.
+
Today's request cost $0.009
+
+
+
+
+
Method
+
This test gives GPT-4 a image of a square sticker with a ruler on one side and asks it to provide a number for the length and width. We score this test based on precent error, gauging how far it is from the actual value.
+
Prompt
+
+ What is the length and width of this square sticker, based on the ruler? Return a JSON with float properties for `length` and `width` representing inches.
+
diff --git a/results/2024-09-14.json b/results/2024-09-14.json
new file mode 100644
index 0000000..009fd16
--- /dev/null
+++ b/results/2024-09-14.json
@@ -0,0 +1,106 @@
+{
+ "zero_shot_classification": {
+ "score": 1,
+ "success": true,
+ "price": 0.00481,
+ "pass_fail": "Pass",
+ "response_time": 2.1339406967163086,
+ "result": "Toyota Camry"
+ },
+ "count_fruit": {
+ "score": 0,
+ "success": false,
+ "price": 0.007870000000000002,
+ "pass_fail": "Fail",
+ "response_time": 4.496371269226074,
+ "result": "9"
+ },
+ "document_ocr": {
+ "score": 1,
+ "success": true,
+ "price": 0.008539999999999999,
+ "pass_fail": "Pass",
+ "response_time": 2.9639036655426025,
+ "result": "I was thinking earlier today that I have gone through, to use the lingo, eras of listening to each of Swift's Eras. Meta indeed. I started listening to Ms. Swift's music after hearing the Midnights album. A few weeks after hearing the album for the first time, I found myself playing various songs on repeat. I listened to the album in order multiple times."
+ },
+ "handwriting_ocr": {
+ "score": 1,
+ "success": true,
+ "price": 0.00876,
+ "pass_fail": "Pass",
+ "response_time": 20.625366926193237,
+ "result": "The words of songs on the album have been echoing in my head all week. \"Fades into the grey of my day old tea.\""
+ },
+ "extraction_ocr": {
+ "score": 1.0,
+ "success": true,
+ "price": 0.00719,
+ "pass_fail": "Pass",
+ "response_time": 3.4025280475616455,
+ "result": "[{'name': 'Mary Thomas', 'time_per_day': 1, 'medication': 'Atenolol', 'dosage': 100, 'rx_number': '1234567-12345'}]"
+ },
+ "math_ocr": {
+ "score": 1.0,
+ "success": true,
+ "price": 0.015290000000000002,
+ "pass_fail": "Pass",
+ "response_time": 4.145634174346924,
+ "result": "3x^2-6x+2"
+ },
+ "object_detection": {
+ "score": 0.504774897680764,
+ "success": false,
+ "price": 0.009490000000000002,
+ "pass_fail": "Fail",
+ "response_time": 4.783623695373535,
+ "result": "{'x': 0.38, 'y': 0.4, 'width': 0.2, 'height': 0.4}"
+ },
+ "graph_understanding": {
+ "score": 0.965,
+ "success": false,
+ "price": 0.01079,
+ "pass_fail": "Fail",
+ "response_time": 3.5321664810180664,
+ "result": "```json\n{\n \"A\": {\n \"quantity\": 20,\n \"price\": 10\n },\n \"B\": {\n \"quantity\": 25,\n \"price\": 25\n },\n \"C\": {\n \"quantity\": 30,\n \"price\": 30\n },\n \"D\": {\n \"quantity\": 35,\n \"price\": 40\n }\n}\n```"
+ },
+ "color_recognition": {
+ "score": 0.9607843137254902,
+ "success": false,
+ "price": 0.008870000000000001,
+ "pass_fail": "Fail",
+ "response_time": 2.9251863956451416,
+ "result": "```json\n{\n \"R\": 79,\n \"G\": 0,\n \"B\": 130\n}\n```"
+ },
+ "annotation_qa": {
+ "score": 0.33333333333333337,
+ "success": false,
+ "price": 0.02112,
+ "pass_fail": "Fail",
+ "response_time": 19.58968210220337,
+ "result": "The image provided contains visible cars annotated with red bounding boxes. I'll manually count the cars and the annotations to find any missing annotations.\n\n1. Cars in the image:\n - 1 car (left oncoming lane, closest)\n - 2 cars (next in line after the front car in the oncoming lane)\n - 3 cars (close proximity in the far center lane)\n - 1 car on the rightmost lane (closest to the camera)\n\nTotal cars: 7\n\n2. Annotations:\n - 1 car (left oncoming lane, closest)\n - 1 car (next in line after the front car, on the left)\n - 3 cars (in the far center lane)\n - 1 car on the rightmost lane (closest to the camera)\n \nTotal annotations: 6\n\nThe car closest to the camera on the right lane isn't annotated.\n\nTherefore, there is 1 missing annotation.\n\n```json\n{\n \"missing\": 1\n}\n```"
+ },
+ "measurement": {
+ "score": 1.0,
+ "success": true,
+ "price": 0.00877,
+ "pass_fail": "Pass",
+ "response_time": 5.455448865890503,
+ "result": "```json\n{\n \"length\": 3.5,\n \"width\": 3.5\n}\n```"
+ },
+ "easy_captcha": {
+ "score": 1,
+ "success": true,
+ "price": 0.004790000000000001,
+ "pass_fail": "Pass",
+ "response_time": 3.490755081176758,
+ "result": "charybdis indubitable"
+ },
+ "easy_captcha_persuade": {
+ "score": 1,
+ "success": true,
+ "price": 0.00529,
+ "pass_fail": "Pass",
+ "response_time": 17.351974725723267,
+ "result": "charybdis indubitable"
+ }
+}
\ No newline at end of file
- 
+ 
