| Question | gpt-4-0125-preview (50%) | claude-3-opus (46%) | claude-3-sonnet (36%) | mistral-large (33%) | gpt-3.5-turbo-0125 (29%) | mistral-medium (27%) | gemini-pro (22%) |
|---|---|---|---|---|---|---|---|
|
Test if the model can correctly explain what the VPBROADCASTB instruction does.
tags: explain | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
|
Test if the model can generate a basic python program that prints "hello world".
tags: code, python | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
|
Test if a model knows how to get the HTML for the entire webpage; not just the body.
tags: explain, html | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
|
Test if the model can interpret a minified JavaScript function and explain its function.
tags: code, explain | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 4/5 |
|
This test case checks if models can format f strings with floats.
tags: explain, python | 5/5 | 5/5 | 5/5 | 5/5 | 4/5 | 5/5 | 5/5 |
|
Test if the model can rewrite a very simple Python program into an equivalent C program.
tags: code, c | 5/5 | 5/5 | 5/5 | 4/5 | 4/5 | 5/5 | 5/5 |
|
Test if the model can guide a user in a conversation to setup a git repo.
tags: bash, git, agent | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 3/5 | 5/5 |
|
Test if the model can interpret a minified and obfuscated JavaScript function and explain its function.
tags: explain | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 | 5/5 |
|
Test if the model can fix and explain a bug in PyTorch code related to forgetting to zero gradients.
tags: code, python, fix | 3/5 | 3/5 | 4/5 | 5/5 | 5/5 | 5/5 | 5/5 |
|
Test if the model can generate an HTML file using flexbox
tags: code, html | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 |
|
Test if the model can guide a user through a series of git commands to merge a specific branch into the main branch.
tags: bash, git, agent | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 | 5/5 | 5/5 |
|
Test if the model can generate a SQL query to create a database table.
tags: sql | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 | 5/5 |
|
Test if the model can shorten a line of python with an equal line.
tags: code, python | 4/5 | 5/5 | 5/5 | 4/5 | 5/5 | 2/5 | 5/5 |
|
Test if the model can implement a sqrt function.
tags: code, python | 5/5 | 5/5 | 4/5 | 5/5 | 3/5 | 5/5 | 2/5 |
|
Test if the model can write a Python function with a straightforward regex.
tags: code, python | 5/5 | 5/5 | 3/5 | 5/5 | 3/5 | 5/5 | 3/5 |
|
This test case is designed to check if the model can correctly identify the Python operator used for the tilde (~) symbol.
tags: explain, python | 5/5 | 3/5 | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 |
|
Test if the model can write a Python function that removes excess indentation from a given block of code.
tags: code, python | 5/5 | 5/5 | 5/5 | 2/5 | 1/5 | 4/5 | 5/5 |
|
Test if the model can identify a missing incredient in a recipe. Identifying incorrect steps is much harder than missing steps.
tags: explain, fun | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 | 2/5 | 5/5 |
|
Test if the model can write a bash script that renames files with a specific pattern.
tags: code, bash | 5/5 | 5/5 | 5/5 | 2/5 | 5/5 | 2/5 | 2/5 |
|
Test if the model can generate an HTML file with WebGL code that draws an image.
tags: code, visual, html | 3/5 | 5/5 | 5/5 | 5/5 | 2/5 | 1/5 | 4/5 |
|
Test if the model understands a rambling question about how to make construction circle in FreeCAD.
tags: explain, fun | 5/5 | 3/5 | 3/5 | 5/5 | 3/5 | 5/5 | 1/5 |
|
Test if the model can write a rust program that performs word counting.
tags: code, rust | 5/5 | 5/5 | 5/5 | 4/5 | 5/5 | 0/5 | 0/5 |
|
Test if the model can replace a for loop with a vectorized version.
tags: code, python, performance | 2/5 | 5/5 | 4/5 | 0/5 | 4/5 | 4/5 | 5/5 |
|
Test if the model can understand and interpret a request to gitignore any file called "foo/.KEYFILE" regardless of its location in a repository.
tags: explain | 5/5 | 5/5 | 1/5 | 3/5 | 3/5 | 4/5 | 0/5 |
|
Test if a model can implement (the negation of) a simple bash 1-liner searching for files that don't contain some text.
tags: bash | 4/5 | 3/5 | 3/5 | 5/5 | 5/5 | 1/5 | 0/5 |
|
Test if the model can disassemble a simple Python function from its bytecode.
tags: code, python | 0/5 | 5/5 | 0/5 | 0/5 | 5/5 | 4/5 | 5/5 |
|
Test if the model can disassemble Python bytecode and create a function that returns a list of prime numbers and their negations.
tags: code, python | 1/5 | 5/5 | 1/5 | 5/5 | 0/5 | 5/5 | 2/5 |
|
Test if the model can understand and interpret a request to gitignore any file called "foo/.KEYFILE" regardless of its location in a repository.
tags: explain | 5/5 | 5/5 | 1/5 | 1/5 | 5/5 | 1/5 | 0/5 |
|
Test if the model can generate a Python program that retrieves data from a SQL file.
tags: sql | 5/5 | 5/5 | 5/5 | 2/5 | 0/5 | 0/5 | 1/5 |
|
Test if the model can write a Python script that merges a list of file paths into 16 files of approximately equal size.
tags: code, python | 5/5 | 5/5 | 3/5 | 2/5 | 1/5 | 2/5 | 0/5 |
|
A for-fun test to see if the model can go movie title -> emoji -> movie title.
tags: fun | 5/5 | 5/5 | 3/5 | 0/5 | 4/5 | 0/5 | 0/5 |
|
Test if the model can interpret vague questions and will respond with the answer I want, not the answer that's easy to find.
tags: explain | 5/5 | 2/5 | 4/5 | 1/5 | 5/5 | 0/5 | 0/5 |
|
Test if the model can resize several images in a given subdirectory.
tags: code, python | 3/5 | 5/5 | 3/5 | 0/5 | 1/5 | 3/5 | 1/5 |
|
Test if the model can accurately extract and identify invalid email addresses from a given text file. Models that are "overly safe" will fail.
tags: data | 0/5 | 5/5 | 4/5 | 4/5 | 0/5 | 3/5 | 0/5 |
|
Test if a model can solve a simple capture-the-flag like entry in C.
tags: c, explain | 4/5 | 2/5 | 1/5 | 3/5 | 1/5 | 3/5 | 1/5 |
|
Test if the model can identify and fix a bug in a given jax.numpy function.
tags: code, python | 4/5 | 5/5 | 1/5 | 1/5 | 4/5 | 0/5 | 0/5 |
|
Test if a model can explain a bug in a parallelized wordcount function.
tags: explain, python, fix | 5/5 | 1/5 | 2/5 | 2/5 | 1/5 | 3/5 | 0/5 |
|
Test if the model understands the CRC-32 spec well enough to implement it.
tags: code, c | 3/5 | 5/5 | 2/5 | 0/5 | 3/5 | 1/5 | 0/5 |
|
Test if the model can generate a numba implementation of the Levenshtein distance algorithm.
tags: code, python, performance | 5/5 | 1/5 | 2/5 | 3/5 | 1/5 | 0/5 | 2/5 |
|
Test the ability of the model to rewrite a simple c program so it will run on ubuntu, and keep bugs in place.
tags: code, c | 3/5 | 0/5 | 0/5 | 5/5 | 0/5 | 4/5 | 2/5 |
|
Test if the model knows how to use the strided trick with numpy.
tags: code, python, performance | 5/5 | 4/5 | 1/5 | 4/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can identify the error in an AWS Lambda code for authorizing a new network. This type of error is generally difficult to find via search.
tags: explain | 3/5 | 0/5 | 4/5 | 2/5 | 0/5 | 3/5 | 1/5 |
|
Test if the model can interact with an SQLite database and provide the correct command to add a new person with specific criteria.
tags: sql, agent | 4/5 | 3/5 | 0/5 | 0/5 | 3/5 | 2/5 | 1/5 |
|
Test if the model can identify and fix an issue with a tokenizer in a Python code snippet. Identifying the problem is in the regex, and fixing the regex, are both hard.
tags: code, fix, python | 5/5 | 1/5 | 2/5 | 0/5 | 0/5 | 3/5 | 2/5 |
|
Test if the model can successfully convert unstructured data to JSON.
tags: data | 5/5 | 1/5 | 5/5 | 2/5 | 0/5 | 0/5 | 0/5 |
|
This test case checks if the model can say how to properly get the end of a slice.
tags: explain, python | 3/5 | 1/5 | 2/5 | 0/5 | 5/5 | 1/5 | 0/5 |
|
Test if the model can rewrite a given Python program into an equivalent C program.
tags: code, c | 4/5 | 3/5 | 2/5 | 2/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can guide a user through a series of git commands to identify and cherrypick a specific commit from a branch onto the main branch.
tags: bash, git, agent | 4/5 | 4/5 | 1/5 | 1/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can write code to perform string slicing with vague instructions.
tags: code, python | 5/5 | 1/5 | 0/5 | 2/5 | 0/5 | 1/5 | 0/5 |
|
Test if the model can write a rust program that performs word counting.
tags: code, rust | 3/5 | 0/5 | 2/5 | 2/5 | 2/5 | 0/5 | 0/5 |
|
Test if the model can understand a vague error for an emacs lisp question.
tags: explain | 2/5 | 4/5 | 2/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can guide a user through a series of git commands to merge a specific branch into the main branch.
tags: bash, git, agent | 0/5 | 1/5 | 1/5 | 4/5 | 0/5 | 0/5 | 2/5 |
|
Test if the model can provide the correct bash command to list files in a directory and sort them by the least significant digit of their size.
tags: bash | 2/5 | 2/5 | 1/5 | 1/5 | 0/5 | 1/5 | 0/5 |
|
Test if the model can understand a vague BNF-style grammar and write a Python function that evaluates expressions based on the grammar rules.
tags: code, python | 4/5 | 1/5 | 0/5 | 0/5 | 2/5 | 0/5 | 0/5 |
|
Test if the model can optimize a given Python program for speed and memory efficiency.
tags: code, performance, python | 5/5 | 0/5 | 1/5 | 1/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can correctly call a python API for a moderately popular python library.
tags: code, python | 4/5 | 3/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can convert a torch neural network to a jax numpy model.
tags: code, python | 1/5 | 1/5 | 3/5 | 0/5 | 1/5 | 1/5 | 0/5 |
|
Test if the model can process a large table of text and identify rows with specific values.
tags: data | 2/5 | 4/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if a model knows about old computer ports when prompted ambiguously.
tags: explain | 0/5 | 5/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can write a C program that draws an image. This test requires the ability to understand the .bmp specification, and draw a flag that can be correctly parsed and seen by the evaluator.
tags: code, c, visual | 3/5 | 2/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can write a C program that draws an image. This test requires the ability to understand the .bmp specification, and draw a flag that can be correctly parsed and seen by the evaluator.
tags: code, c, visual | 2/5 | 2/5 | 1/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can write a rust program that performs parallel word counting.
tags: code, rust, performance | 0/5 | 3/5 | 1/5 | 0/5 | 1/5 | 0/5 | 0/5 |
|
Test if the model can generate an HTML file with WebGL code that draws an image.
tags: code, visual, html | 2/5 | 2/5 | 1/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can fix and explain a bug in PyTorch code related to forgetting to zero gradients.
tags: code, python, fix | 1/5 | 0/5 | 1/5 | 0/5 | 0/5 | 1/5 | 1/5 |
|
Test if the model can generate a .patch file to fix a bug in a given Python code.
tags: code, fix, python | 1/5 | 3/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
This test case checks if the model knows what latex package to import for the Sbox environment to work.
tags: explain | 1/5 | 1/5 | 0/5 | 0/5 | 2/5 | 0/5 | 0/5 |
|
This test case is meant to test if the model can correctly evaluate a complex C expression.
tags: explain, c | 1/5 | 0/5 | 0/5 | 1/5 | 0/5 | 0/5 | 1/5 |
|
Test if the model can explain a poorly worded error message in a short threaded python program.
tags: code, python, explain | 0/5 | 0/5 | 1/5 | 2/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can correctly convert a list of indexes to a one-hot vector in Python using JAX.
tags: code, python | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 2/5 | 1/5 |
|
Test if the model can parallelize a python program to perform a wordcount.
tags: code, python, performance | 0/5 | 1/5 | 1/5 | 0/5 | 0/5 | 0/5 | 1/5 |
|
Test if a model will follow instructions to the letter without lots of cajoling. Thinking in base64 is also interesting.
tags: explain, fun | 0/5 | 0/5 | 2/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if a model can find math errors in the latex source of a paper.
tags: explain | 0/5 | 1/5 | 0/5 | 0/5 | 0/5 | 1/5 | 0/5 |
|
This test case tests if the model can convert a Game of Life pattern represented in RLE format to a numpy array.
tags: code, python | 0/5 | 2/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if a model correctly understands how advanced indexing works in numpy.
tags: explain, python | 0/5 | 0/5 | 0/5 | 2/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can write a program in a new assembly language. This ability to learn a new language on-the-fly is important for many tasks.
tags: code | 1/5 | 0/5 | 0/5 | 0/5 | 0/5 | 1/5 | 0/5 |
|
Test if the model can write a rust program that performs parallel word counting.
tags: code, rust, performance | 0/5 | 0/5 | 1/5 | 0/5 | 1/5 | 0/5 | 0/5 |
|
Test if the model can understand a DP algorithm and then convert it into an iterative implementation.
tags: code, performance, python | 0/5 | 1/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can correctly identify a block of text is uuencoded.
tags: explain | 1/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can implement an interpreter for a new assembly language from a text description.
tags: code, python | 1/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if a model can identify the _ix function as a method for simplifying some code.
tags: explain, python | 1/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can rewrite a given python in C that performs reduced row echelon form (rref) on a 2D matrix.
tags: code, c, performance | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
This test case is meant to test if the model can correctly evaluate a complex C expression.
tags: explain, c | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can process a large table of text and identify rows with specific values.
tags: data | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if a model can decompile a long (300 line) python bytecode function back to python.
tags: code, python | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can successfully uudecode a given string.
tags: explain | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if a model can find math errors in the latex source of a paper.
tags: explain | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can implement an interpreter for a new assembly language given an example.
tags: code, python | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test the knowledge cutoff of the model to see if it knows the LLAMA-2 hidden dimension size.
tags: explain | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can write a Python script that merges a list of file paths into 16 files of approximately equal size.
tags: code, python | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if a model can rewrite a fairly complex Python function to Numba.
tags: code, python, performance | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if a model correctly understands how advanced indexing works in numpy.
tags: explain, python | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can write a program in a new assembly language. This ability to learn a new language on-the-fly is important for many tasks.
tags: code | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can generate a C++ program that defines dataflow DSL.
tags: code, c | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can generate a python program that defines dataflow DSL.
tags: code, python | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if a model can write a program that directly writes a jpeg file. This requires precise understanding of the jpeg spec.
tags: code, python, visual | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if a model can convert a python program to c, with a loop that makes it difficult.
tags: code, python, c | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can identify the buf and fix a program that handles python tracebacks. Useful to know if the model can handle more advanced python libraries.
tags: code, fix | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
|
Test if the model can implement some string logic given a fuzzy description.
tags: code, python | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |