Introduction

Revelo's REST API provides a powerful solution for training Large Language Models (LLMs) with a focus on code generation, utilizing human input for data labeling. With our intuitive API, you gain access to a high-quality pool of vetted tech talent, designed to significantly enhance the accuracy and performance of your LLMs.

Core concepts

Projects

A Project in Revelo's API represents a collection of tasks grouped together for easier management and organization. Projects serve as a top-level entity that allows you to efficiently oversee and coordinate the labeling process across multiple tasks.

Tasks

A Task is an individual unit of work within a project. It typically involves labeling a specific piece of information, such as a code snippet. Each task can have multiple associated questions, allowing you to gather detailed insights and labels.

Questions

Questions are the specific fields that labelers are required to answer. Revelo's API supports five distinct question categories, each tailored to gather precise and relevant data.

Responses

Responses capture the full set of answers provided for the questions within a task. When creating a task, you can specify the required number of responses, ensuring that you gather a sufficient amount of labeled data to meet your needs.

Getting started

Authentication

To authorize, use this code:

import requests
import json

url = "https://api.revelo.com/v1/projects"

payload = json.dumps({
  "name": "My project"
})
headers = {
  'Content-Type': 'application/json',
  'Authorization': '<your_api_key>'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

curl --location 'https://api.revelo.com/v1/projects' \
--header 'Content-Type: application/json' \
--header 'Authorization: your_api_key' \
--data '{
  "name": "My project"
}'

Make sure to replace your_api_key with your API key.

Revelo uses API keys to control access to its API. To obtain an API key, please contact our Sales team.

Revelo's API requires that the API key be included in all requests to the server using HTTP Basic Authentication. The API key should be provided in the Authorization header in the following format:

Authorization: <your_api_key>

You must replace your_api_key with your organization's API key.

Using your identifiers

The external_key parameters can be utilized as unique identifiers for your resources.

Responses and pagination

The API responses conform to the JSON:API specification, ensuring consistency and predictability in the data structure. All index endpoints are paginated, allowing you to efficiently manage large datasets.

Pagination Parameters

size: Controls the number of items returned per page. The default value is 20, and you can increase this up to a maximum of 100 to retrieve more items per request.
page: Specifies the page number of the results you wish to access. This allows you to navigate through the paginated data.

Navigating Pages

Within the links node of the JSON:API response, you will find URLs for navigating between pages of results. The next link provides the URL for the subsequent page, allowing for seamless data retrieval across multiple pages.

For more detailed information on how pagination is structured according to the JSON:API specification, please refer to the JSON:API Pagination section.

API Rate Limit

Revelo limits the number of REST API requests that you can make within a specific amount of time. This limit ensures that the API remains available for all users.

Revelo API's has a max api rate limit of 7,000 requests per hour.

Projects API

Retrieve all projects

Endpoint

GET https://api.revelo.com/v1/projects

Example request

import requests

url = "https://api.revelo.com/v1/projects?page=1&size=20"

payload = {}
headers = {
  'Content-Type': 'application/json',
  'Authorization': '<your_api_key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

curl --location 'https://api.revelo.com/v1/projects?page=1&size=20' \
     --header 'Authorization: your_api_key'

Example response

{
  "data": [{
    "type": "projects",
    "id": "8ca8edc4-91e6-4c21-be18-e783c7296880",
    "attributes": {
      "name": "My project",
      "status": "draft",
      "callback_url": "https://your_callback_url.domain.com",
      "created_at": "2015-05-22T14:56:29.000Z",
      "updated_at": "2015-05-22T14:56:28.000Z"
    }
  }],
  "meta": {
    "total_count": 200,
    "total_pages": 10
  },
  "links": {
    "prev": null,
    "next": "https://api.revelo.com/v1/projects?page=2&size=20",
    "self": "https://api.revelo.com/v1/projects?page=1&size=20",
    "first": "https://api.revelo.com/v1/projects?page=1&size=20",
    "last": "https://api.revelo.com/v1/projects?page=10&size=20"
  }
}

This endpoint retrieves all projects.

Query Parameters

Parameter	Default	Description
size	20	The number of rows returned per page. Adjust this to control the size of each page.
page	1	The page cursor used to navigate through paginated results.

Retrieve a project

Endpoint

GET https://api.revelo.com/v1/projects/<projectId>

Example request

import requests

url = "https://api.revelo.com/v1/projects/<projectId>"

payload = {}
headers = {
  'Content-Type': 'application/json',
  'Authorization': '<your_api_key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

curl --location 'https://api.revelo.com/v1/projects/<projectId>' \
--header 'Authorization: your_api_key'

Example response:

{
  "data": {
    "type": "projects",
    "id": "8ca8edc4-91e6-4c21-be18-e783c7296880",
    "attributes": {
      "name": "My project",
      "status": "draft",
      "callback_url": "https://your_callback_url.domain.com",
      "created_at": "2015-05-22T14:56:29.000Z",
      "updated_at": "2015-05-22T14:56:28.000Z"
    }
  }
}

This endpoint retrieves a specific project.

URL Parameters

Parameter	Description
projectId	The ID of the project to retrieve

Create project

Endpoint

POST https://api.revelo.com/v1/projects

Example request:

import requests
import json

url = "https://api.revelo.com/v1/projects"

payload = json.dumps({
  "name": "My project",
  "callback_url": "https://your_callback_url.domain.com"
})
headers = {
  'Content-Type': 'application/json',
  'Authorization': '<your_api_key>'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

curl --location 'https://api.revelo.com/v1/projects' \
--header 'Content-Type: application/json' \
--header 'Authorization: your_api_key' \
--data '{
  "name": "My project",
  "callback_url": "https://your_callback_url.domain.com"
}'

Example response:

{
  "data": {
    "type": "projects",
    "id": "8ca8edc4-91e6-4c21-be18-e783c7296880",
    "attributes": {
      "name": "My project",
      "status": "draft",
      "callback_url": "https://your_callback_url.domain.com",
      "created_at": "2015-05-22T14:56:29.000Z",
      "updated_at": "2015-05-22T14:56:28.000Z"
    }
  }
}

This endpoint creates a new project.

Accepted body parameters

Parameter	Description
name	The name of the project
callback_url	The callback URL is where Revelo's API sends notifications whenever a new response is received for a task.

Edit project

Endpoint

PUT https://api.revelo.com/v1/projects/<projectId>

Example request

import requests
import json

url = "https://api.revelo.com/v1/projects/<projectId>"

payload = json.dumps({
  "name": "My project",
  "status": "published",
  "callback_url": "https://your_callback_url.domain.com",
})
headers = {
  'Content-Type': 'application/json',
  'Authorization': '<your_api_key>'
}

response = requests.request("PUT", url, headers=headers, data=payload)

print(response.text)

curl --location --request PUT 'https://api.revelo.com/v1/projects/<projectId>' \
    --header 'Content-Type: application/json' \
    --header 'Authorization: your_api_key' \
    --data '{
      "name": "My project",
      "status": "published",
      "callback_url": "https://your_callback_url.domain.com",
    }'

Example response

{
  "data": {
    "type": "projects",
    "id": "8ca8edc4-91e6-4c21-be18-e783c7296880",
    "attributes": {
      "name": "My project",
      "status": "draft",
      "callback_url": "https://your_callback_url.domain.com",
      "created_at": "2015-05-22T14:56:29.000Z",
      "updated_at": "2015-05-22T14:56:28.000Z"
    }
  }
}

This endpoint edits a specific project.

URL Parameters

Parameter	Description
projectId	The ID of the project to be editted

Accepted body parameters

Parameter	Type	Description
name	string	The name of the project
status	string	Refer to the section Project statuses
callback_url	string	The callback URL is where Revelo's API sends notifications whenever a new response is received for a task.

Project statuses

The table below outlines the different statuses that a project can have within the API. You can use the Edit project endpoint to change a project's status as needed, enabling you to manage the project's lifecycle effectively.

Status	Description
draft	The project is in the initial stage and is not yet visible to labelers. It can be modified freely.
published	The project is active and visible to labelers, who can start working on the tasks.
paused	The project is temporarily halted. Labelers cannot work on it until it is resumed.
canceled	The project is permanently stopped and will no longer be available for labelers to work on.

Tasks API

Retrieve all tasks

This endpoint retrieves all tasks.

Endpoint

GET https://api.revelo.com/v1/projects/<projectId>/tasks

Example request

import requests

url = "https://api.revelo.com/v1/projects/<projectId>/tasks?page=1&size=20"

payload = {}
headers = {
  'Content-Type': 'application/json',
  'Authorization': '<your_api_key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

curl --location 'https://api.revelo.com/v1/projects/<projectId>/tasks?page=1&size=20' \
--header 'Authorization: your_api_key'

Example response

{
  "data": [{
    "type": "task",
    "id": "9b58544a-a850-4122-87dd-1b58da274393",
    "attributes": {
      "external_key": "your_unique_id",
      "code_snippet": "def factorial(n):\n    return 1 if n == 0 else n * factorial(n - 1)\n\nprint(factorial(5))",
      "code_language": "python",
      "required_task_responses": 5,
      "instructions": "Assess the following code",
      "project_id": "8ca8edc4-91e6-4c21-be18-e783c7296880",
      "status": "draft",
      "created_at": "2015-05-22T14:56:29.000Z",
      "updated_at": "2015-05-22T14:56:28.000Z"
    },
    "relationships": {
      "questions": {
        "data": [
          {
            "id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
            "type": "question"
          },
          {
            "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
            "type": "question"
          }
        ]
      }
    }
  }],
  "included": [
    {
      "type": "question",
      "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
      "attributes": {
        "external_key": "purpose_question_key",
        "category": "single_choice",
        "instructions": "How would you categorize the purpose of this function?",
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393"
      },
      "relationships": {
        "question_options": {
          "data": [
            {
              "id": "cb4a83f5-db90-42b7-b465-0a4fe990b9d6",
              "type": "question_option"
            },
            {
              "id": "bc847db4-c27e-4e7b-84d8-39271b377278",
              "type": "question_option"
            },
            {
              "id": "7065c7f6-9328-419a-b80d-575b4536e605",
              "type": "question_option"
            },
            {
              "id": "c2c20bf5-c40e-45d5-bb32-34aa7c96b518",
              "type": "question_option"
            }
          ]
        }
      }
    },
    {
      "type": "question",
      "id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
      "attributes": {
        "external_key": "explanation_question_key",
        "category": "free_text",
        "instructions": "If you selected 'None of the above', please explain what this code does.",
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393"
      },
      "relationships": {
        "question_options": {
            "data": []
        }
      }
    },
    {
      "type": "question_option",
      "id": "cb4a83f5-db90-42b7-b465-0a4fe990b9d6",
      "attributes": {
        "value": "0",
        "label": "Mathematical Computation"
      }
    },
    {
      "type": "question_option",
      "id": "bc847db4-c27e-4e7b-84d8-39271b377278",
      "attributes": {
        "value": "1",
        "label": "String Manipulation"
      }
    },
    {
      "type": "question_option",
      "id": "7065c7f6-9328-419a-b80d-575b4536e605",
      "attributes": {
        "value": "2",
        "label": "File I/O"
      }
    },
    {
      "type": "question_option",
      "id": "c2c20bf5-c40e-45d5-bb32-34aa7c96b518",
      "attributes": {
        "value": "3",
        "label": "None of the above"
      }
    }
  ],
  "meta": {
    "total_count": 20,
    "total_pages": 2
  },
  "links": {
    "prev": null,
    "next": "https://api.revelo.com/v1/projects/8ca8edc4-91e6-4c21-be18-e783c7296880/tasks?page=2&size=20",
    "self": "https://api.revelo.com/v1/projects/8ca8edc4-91e6-4c21-be18-e783c7296880/tasks?page=1&size=20",
    "first": "https://api.revelo.com/v1/projects/8ca8edc4-91e6-4c21-be18-e783c7296880/tasks?page=1&size=20",
    "last": "https://api.revelo.com/v1/projects/8ca8edc4-91e6-4c21-be18-e783c7296880/tasks?page=2&size=20"
  }
}

Query Parameters

Parameter	Default	Description
projectId	-	The id of the project to fetch the tasks.
size	20	The number of rows returned per page. Adjust this to control the size of each page.
page	1	The page cursor used to navigate through paginated results.

Retrieve a task

Endpoint

GET https://api.revelo.com/v1/projects/<projectId>/tasks/<taskId>

Example request

import requests

url = "https://api.revelo.com/v1/projects/<projectId>/tasks/<taskId>"

payload = {}
headers = {
  'Content-Type': 'application/json',
  'Authorization': '<your_api_key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

curl --location 'https://api.revelo.com/v1/projects/<projectId>/tasks/<taskId>' \
--header 'Authorization: your_api_key'

Example response:

{
  "data": {
    "type": "task",
    "id": "9b58544a-a850-4122-87dd-1b58da274393",
    "attributes": {
      "external_key": "your_unique_id",
      "code_snippet": "def factorial(n):\n    return 1 if n == 0 else n * factorial(n - 1)\n\nprint(factorial(5))",
      "code_language": "python",
      "required_task_responses": 5,
      "instructions": "Assess the following code",
      "project_id": "8ca8edc4-91e6-4c21-be18-e783c7296880",
      "status": "draft",
      "created_at": "2015-05-22T14:56:29.000Z",
      "updated_at": "2015-05-22T14:56:28.000Z"
    },
    "relationships": {
      "questions": {
        "data": [
          {
            "id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
            "type": "question"
          },
          {
            "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
            "type": "question"
          }
        ]
      }
    }
  },
  "included": [
    {
      "type": "question",
      "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
      "attributes": {
        "external_key": "purpose_question_key",
        "category": "single_choice",
        "instructions": "How would you categorize the purpose of this function?",
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393"
      },
      "relationships": {
        "question_options": {
          "data": [
            {
              "id": "cb4a83f5-db90-42b7-b465-0a4fe990b9d6",
              "type": "question_option"
            },
            {
              "id": "bc847db4-c27e-4e7b-84d8-39271b377278",
              "type": "question_option"
            },
            {
              "id": "7065c7f6-9328-419a-b80d-575b4536e605",
              "type": "question_option"
            },
            {
              "id": "c2c20bf5-c40e-45d5-bb32-34aa7c96b518",
              "type": "question_option"
            }
          ]
        }
      }
    },
    {
      "type": "question",
      "id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
      "attributes": {
        "external_key": "explanation_question_key",
        "category": "free_text",
        "instructions": "If you selected 'None of the above', please explain what this code does.",
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393"
      },
      "relationships": {
        "question_options": {
            "data": []
        }
      }
    },
    {
      "type": "question_option",
      "id": "cb4a83f5-db90-42b7-b465-0a4fe990b9d6",
      "attributes": {
        "value": "0",
        "label": "Mathematical Computation"
      }
    },
    {
      "type": "question_option",
      "id": "bc847db4-c27e-4e7b-84d8-39271b377278",
      "attributes": {
        "value": "1",
        "label": "String Manipulation"
      }
    },
    {
      "type": "question_option",
      "id": "7065c7f6-9328-419a-b80d-575b4536e605",
      "attributes": {
        "value": "2",
        "label": "File I/O"
      }
    },
    {
      "type": "question_option",
      "id": "c2c20bf5-c40e-45d5-bb32-34aa7c96b518",
      "attributes": {
        "value": "3",
        "label": "None of the above"
      }
    }
  ]
}

This endpoint retrieves a specific task.

URL Parameters

Parameter	Description
taskId	The ID of the task to retrieve

Create task

Endpoint

POST https://api.revelo.com/v1/projects/<projectId>/tasks

Example request

import requests
import json

url = "https://api.revelo.com/v1/projects/<projectId>/tasks"

payload = json.dumps({
  "external_key": "your_unique_id",
  "code_snippet": "def factorial(n):\n    return 1 if n == 0 else n * factorial(n - 1)\n\nprint(factorial(5))",
  "code_language": "python",
  "required_task_responses": 5,
  "instructions": "Assess the following code",
  "questions": [
    {
      "external_key": "purpose_question_key",
      "category": "single_choice",
      "instructions": "How would you categorize the purpose of this function?",
      "options": [
        {
          "value": "0",
          "label": "Mathematical Computation"
        },
        {
          "value": "1",
          "label": "String Manipulation"
        },
        {
          "value": "2",
          "label": "File I/O"
        },
        {
          "value": "3",
          "label": "None of the Above"
        }
      ]
    },
    {
      "external_key": "explanation_question_key",
      "category": "free_text",
      "instructions": "If you selected 'None of the above', please explain what this code does."
    }
  ]
})
headers = {
  'Content-Type': 'application/json'
  'Authorization': '<your_api_key>'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

curl --location 'https://api.revelo.com/v1/projects/<projectId>/tasks' \
--header 'Content-Type: application/json' \
--header 'Authorization: your_api_key' \
--data '{
  "external_key": "your_unique_id",
  "code_snippet": "def factorial(n):\n    return 1 if n == 0 else n * factorial(n - 1)\n\nprint(factorial(5))",
  "code_language": "python",
  "required_task_responses": 5,
  "instructions": "Assess the following code",
  "questions": [
    {
      "external_key": "purpose_question_key",
      "category": "single_choice",
      "instructions": "How would you categorize the purpose of this function?",
      "options": [
        {
          "value": "0",
          "label": "Mathematical Computation"
        },
        {
          "value": "1",
          "label": "String Manipulation"
        },
        {
          "value": "2",
          "label": "File I/O"
        },
        {
          "value": "3",
          "label": "None of the Above"
        }
      ]
    },
    {
      "external_key": "explanation_question_key",
      "category": "free_text",
      "instructions": "If you selected '\''None of the above'\'', please explain what this code does."
    }
  ]
}'

Example response

{
  "data": {
    "type": "task",
    "id": "9b58544a-a850-4122-87dd-1b58da274393",
    "attributes": {
      "external_key": "your_unique_id",
      "code_snippet": "def factorial(n):\n    return 1 if n == 0 else n * factorial(n - 1)\n\nprint(factorial(5))",
      "code_language": "python",
      "required_task_responses": 5,
      "instructions": "Assess the following code",
      "project_id": "8ca8edc4-91e6-4c21-be18-e783c7296880",
      "status": "draft",
      "created_at": "2015-05-22T14:56:29.000Z",
      "updated_at": "2015-05-22T14:56:28.000Z"
    },
    "relationships": {
      "questions": {
        "data": [
          {
            "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
            "type": "question"
          }
        ]
      }
    }
  },
  "included": [
    {
      "type": "question",
      "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
      "attributes": {
        "external_key": "purpose_question_key",
        "category": "single_choice",
        "instructions": "How would you categorize the purpose of this function?",
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393"
      },
      "relationships": {
        "question_options": {
          "data": [
            {
              "id": "cb4a83f5-db90-42b7-b465-0a4fe990b9d6",
              "type": "question_option"
            },
            {
              "id": "bc847db4-c27e-4e7b-84d8-39271b377278",
              "type": "question_option"
            },
            {
              "id": "7065c7f6-9328-419a-b80d-575b4536e605",
              "type": "question_option"
            },
            {
              "id": "c2c20bf5-c40e-45d5-bb32-34aa7c96b518",
              "type": "question_option"
            }
          ]
        }
      }
    },
    {
      "type": "question",
      "id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
      "attributes": {
        "external_key": "explanation_question_key",
        "category": "free_text",
        "instructions": "If you selected 'None of the above', please explain what this code does.",
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393"
      },
      "relationships": {
        "question_options": {
            "data": []
        }
      }
    },
    {
      "type": "question_option",
      "id": "cb4a83f5-db90-42b7-b465-0a4fe990b9d6",
      "attributes": {
        "value": "0",
        "label": "Mathematical Computation"
      }
    },
    {
      "type": "question_option",
      "id": "bc847db4-c27e-4e7b-84d8-39271b377278",
      "attributes": {
        "value": "1",
        "label": "String Manipulation"
      }
    },
    {
      "type": "question_option",
      "id": "7065c7f6-9328-419a-b80d-575b4536e605",
      "attributes": {
        "value": "2",
        "label": "File I/O"
      }
    },
    {
      "type": "question_option",
      "id": "c2c20bf5-c40e-45d5-bb32-34aa7c96b518",
      "attributes": {
        "value": "3",
        "label": "None of the above"
      }
    }
  ]
}

This endpoint creates a new task.

Accepted body parameters

Parameter	Type	Description
`external_key`	`string`	A unique identifier for the task. You may provide this ID to guarantee idempotency.
`instructions`	`string`	A description or instruction provided to the labeler about what to do with the task.
`code_snippet`	`string`	An optional code snippet that complements the instruction.
`code_language`	`string`	The programming language of the code snippet (e.g., `python`).
`required_task_responses`	`integer`	The number of responses you want this task to have before being considered completed.
`questions`	`array`	A list of questions associated with the task, each containing specific details (see below).
`questions.external_key`	`string`	Your key to identify the question response.
`questions.categories`	`string`	The type of question. Please refer to Question categories
`questions.instructions`	`string`	Instructions for the labeler on how to answer the question.
`questions.question_options`	`array`	An array of possible question_options for the question, applicable for choice-based question categories.
`question_options.value`	`string`	A unique value representing a specific option.
`question_options.label`	`string`	The label displayed for the option, describing the choice (e.g., `Efficient`).

Edit task

Endpoint

PUT https://api.revelo.com/v1/projects/<projectId>/tasks/<taskId>

Example request

import requests
import json

url = "https://api.revelo.com/v1/projects/<projectId>/tasks/<taskId>"

payload = json.dumps({
  "external_key": "your_unique_id",
  "code_snippet": "print('hello world')",
  "code_language": "python",
  "required_task_responses": 5,
  "status": "canceled",
  "instructions": "Assess the following code"
})
headers = {
  'Content-Type': 'application/json',
  'Authorization': '<your_api_key>'
}

response = requests.request("PUT", url, headers=headers, data=payload)

print(response.text)

curl --location --request PUT 'https://api.revelo.com/v1/projects/<projectId>/tasks/<taskId>' \
--header 'Content-Type: application/json' \
--header 'Authorization: your_api_key' \
--data '{
  "external_key": "your_unique_id",
  "code_snippet": "print('hello world')",
  "code_language": "python",
  "required_task_responses": 5,
  "status": "canceled",
  "instructions": "Assess the following code"
}'

Example response

{
  "data": {
    "type": "task",
    "id": "9b58544a-a850-4122-87dd-1b58da274393",
    "attributes": {
      "external_key": "your_unique_id",
      "code_snippet": "print('hello world')",
      "code_language": "python",
      "required_task_responses": 5,
      "instructions": "Assess the following code",
      "project_id": "8ca8edc4-91e6-4c21-be18-e783c7296880",
      "status": "canceled",
      "created_at": "2015-05-22T14:56:29.000Z",
      "updated_at": "2015-05-22T15:00:00.000Z"
    },
    "relationships": {
      "questions": {
        "data": [
          {
            "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
            "type": "question"
          },
          {
            "type": "question",
            "id": "a6caafe0-dca4-4823-a415-05147f59e4c4"
          }
        ]
      }
    }
  },
  "included": [
    {
      "type": "question",
      "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
      "attributes": {
        "external_key": "purpose_question_key",
        "category": "single_choice",
        "instructions": "How would you categorize the purpose of this function?",
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393"
      },
      "relationships": {
        "question_options": {
          "data": [
            {
              "id": "cb4a83f5-db90-42b7-b465-0a4fe990b9d6",
              "type": "question_option"
            },
            {
              "id": "bc847db4-c27e-4e7b-84d8-39271b377278",
              "type": "question_option"
            },
            {
              "type": "question_option",
              "id": "7065c7f6-9328-419a-b80d-575b4536e605"
            },
            {
              "type": "question_option",
              "id": "c2c20bf5-c40e-45d5-bb32-34aa7c96b518"
            }
          ]
        }
      }
    },
    {
      "type": "question",
      "id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
      "attributes": {
        "external_key": "explanation_question_key",
        "category": "free_text",
        "instructions": "If you selected 'None of the above', please explain what this code does.",
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393"
      },
      "relationships": {
        "question_options": {
            "data": []
        }
      }
    },
    {
      "type": "question_option",
      "id": "cb4a83f5-db90-42b7-b465-0a4fe990b9d6",
      "attributes": {
        "value": "0",
        "label": "Mathematical Computation"
      }
    },
    {
      "type": "question_option",
      "id": "bc847db4-c27e-4e7b-84d8-39271b377278",
      "attributes": {
        "value": "1",
        "label": "String Manipulation"
      }
    },
    {
      "type": "question_option",
      "id": "7065c7f6-9328-419a-b80d-575b4536e605",
      "attributes": {
        "value": "2",
        "label": "File I/O"
      }
    },
    {
      "type": "question_option",
      "id": "c2c20bf5-c40e-45d5-bb32-34aa7c96b518",
      "attributes": {
        "value": "3",
        "label": "None of the above"
      }
    }
  ]
}

This endpoint allows you to edit a specific task. Please note that it is not possible to modify existing questions or question_options through this endpoint. If you need to make changes to questions or question_options, you will need to update the task status to canceled and then create a new task with the desired modifications.

URL Parameters

Parameter	Description
taskId	The ID of the task to be editted

Accepted body parameters

Parameter	Type	Description
`external_key`	`string`	A unique identifier for the task. You may provide this ID to guarantee idempotency.
`instructions`	`string`	A description or instruction provided to the labeler about what to do with the task.
`code_snippet`	`string`	An optional code snippet that complements the instruction.
`code_language`	`string`	The programming language of the code snippet (e.g., `python`).
`required_task_responses`	`integer`	The number of responses you want this task to have before being considered completed.
`status`	`string`	The status of the task, such as `canceled`. Please refer to the Task statuses

Question categories

The table below outlines the different question categories that a question can have within the API.

Category	Description
multiple_choices	Allows the selection of multiple answers from a predefined list.
single_choice	Permits the selection of a single answer from a predefined list.
single_choice_code_snippet	Ideal for scenarios where you need to compare two code snippets and determine which one is superior.
free_text	Enables the entry of open-ended responses, providing more detailed and nuanced feedback.
code_snippet	Allows the entry of code snippets as responses, useful for direct input of code for evaluation.

Task statuses

The table below outlines the different statuses that a task can have within the API.

Status	Description
draft	The task is in its initial stage and is not yet visible to labelers. It can be edited and reviewed before being published.
published	The task is active and visible to labelers, who can now start working on it and provide responses.
canceled	The task has been canceled and is no longer available for responses. It cannot be interacted with further.
completed	The task has received the required number of responses and is considered finished. No further actions are needed.

Task Imports API

Create tasks in bulk

Endpoint

POST https://api.revelo.com/v1/projects/<projectId>/task_imports

Example request using tasks array

import requests
import json

url = "https://api.revelo.com/v1/projects/<projectId>/task_imports"

payload = json.dumps({
  "tasks": [{
    "external_key": "your_unique_id",
    "code_snippet": "def factorial(n):\n    return 1 if n == 0 else n * factorial(n - 1)\n\nprint(factorial(5))",
    "code_language": "python",
    "required_task_responses": 5,
    "instructions": "Assess the following code",
    "questions": [
      {
        "external_key": "purpose_question_key",
        "category": "single_choice",
        "instructions": "How would you categorize the purpose of this function?",
        "options": [
          {
            "value": "0",
            "label": "Mathematical Computation"
          },
          {
            "value": "1",
            "label": "String Manipulation"
          },
          {
            "value": "2",
            "label": "File I/O"
          },
          {
            "value": "3",
            "label": "None of the Above"
          }
        ]
      },
      {
        "external_key": "explanation_question_key",
        "category": "free_text",
        "instructions": "If you selected 'None of the above', please explain what this code does."
      }
    ]
  }
]})
headers = {
  'Content-Type': 'application/json'
  'Authorization': '<your_api_key>'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

curl --location 'https://api.revelo.com/v1/projects/<projectId>/task_imports' \
--header 'Content-Type: application/json' \
--header 'Authorization: your_api_key' \
--data '{
  "tasks": [
    "external_key": "your_unique_id",
    "code_snippet": "def factorial(n):\n    return 1 if n == 0 else n * factorial(n - 1)\n\nprint(factorial(5))",
    "code_language": "python",
    "required_task_responses": 5,
    "instructions": "Assess the following code",
    "questions": [
      {
        "external_key": "purpose_question_key",
        "category": "single_choice",
        "instructions": "How would you categorize the purpose of this function?",
        "options": [
          {
            "value": "0",
            "label": "Mathematical Computation"
          },
          {
            "value": "1",
            "label": "String Manipulation"
          },
          {
            "value": "2",
            "label": "File I/O"
          },
          {
            "value": "3",
            "label": "None of the Above"
          }
        ]
      },
      {
        "external_key": "explanation_question_key",
        "category": "free_text",
        "instructions": "If you selected '\''None of the above'\'', please explain what this code does."
      }
    ]
  ]
}'

Example request using file parameter

import requests
import json

url = "https://api.revelo.com/v1/projects/<projectId>/task_imports"

files = {'file': open('path/to/my_json_file.json','rb')}
headers = { 'Authorization': '<your_api_key>' }

response = requests.request("POST", url, headers=headers, files=files)

print(response.text)

curl --location 'https://api.revelo.com/v1/projects/<projectId>/task_imports' \
--header 'Authorization: your_api_key' \
-F ‘file=@path/to/my_json_file.json

Example response

{
  "data": [
    {
      "type": "task_import",
      "id": "19d583b4-b4d7-47d8-8e28-da50718cc0d2",
      "attributes": {
        "status": "pending",
        "created_at": "2015-05-22T14:56:29.000Z",
        "updated_at": "2015-05-22T14:56:28.000Z"
      }
    }
  ]
}

Create multiple tasks in bulk by importing a JSON file containing the tasks or sending a tasks array parameter. This endpoint is useful when you need to create a large number of tasks at once in background.

Query Parameters

Parameter	Default	Description
file	-	A `.json` file containing the tasks array. This parameter has precedence over tasks array
tasks	-	Array of tasks to be created.

Retrieve a task import

Endpoint

GET https://api.revelo.com/v1/projects/<projectId>/task_imports/<taskImportId>

Example request

import requests

url = "https://api.revelo.com/v1/projects/<projectId>/task_imports/<taskImportId>"

payload = {}
headers = {
  'Content-Type': 'application/json',
  'Authorization': '<your_api_key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

curl --location 'https://api.revelo.com/v1/projects/<projectId>/task_imports/<taskImportId>' \
--header 'Authorization: your_api_key'

Example response:

{
  "data": [
    {
      "type": "task_import",
      "id": "19d583b4-b4d7-47d8-8e28-da50718cc0d2",
      "attributes": {
        "status": "pending",
        "created_at": "2015-05-22T14:56:29.000Z",
        "updated_at": "2015-05-22T14:56:28.000Z"
      }
    }
  ]
}

This endpoint retrieves a specific task import to check status of the bulk operation.

URL Parameters

Parameter	Description
taskImportId	The ID of the task import batch

Task import statuses

The table below outlines the different statuses that a task import can have within the API.

Status	Description
pending	The task import batch is created and waiting to be processed.
processing	The task import is on process at this time and soon will finish.
finished	The task import is finished and all tasks are created into the project.
failed	The task import failed for some reason (probably missing parameter) and no task is created.

Retrieving responses

Callback on task response

Example request Revelo's API will submit

{
  "data": {
    "type": "tasker_response",
    "id": "78a3e098-01f4-495c-826e-9666e023b5a9",
    "attributes": {
      "task_id": "9b58544a-a850-4122-87dd-1b58da274393",
      "created_at": "2015-05-22T14:56:29.000Z",
      "updated_at": "2015-05-22T14:56:28.000Z"
    },
    "relationships": {
      "task": {
        "data": {
          "id": "9b58544a-a850-4122-87dd-1b58da274393",
          "type": "tasks"
        }
      },
      "answers": {
        "data": [
          {
            "id": "45bdf701-9300-4902-9187-8ca4f5ec79ff",
            "type": "answers"
          },
          {
            "id": "7c998b53-13b6-43ca-b561-34912d5c4666",
            "type": "answers"
          }
        ]
      }
    }
  },
  "included": [
    {
      "type": "tasks",
      "id": "9b58544a-a850-4122-87dd-1b58da274393",
      "attributes": {
        "external_key": "your_unique_id",
        "code_snippet": "def factorial(n):\n    return 1 if n == 0 else n * factorial(n - 1)\n\nprint(factorial(5))  ",
        "code_language": "python",
        "required_task_responses": 5,
        "project_id": "8ca8edc4-91e6-4c21-be18-e783c7296880",
        "instructions": "Assess the following code"
      }
    },
    {
      "type": "questions",
      "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
      "attributes": {
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393",
        "external_key": "purpose_question_key",
        "category": "single_choice",
        "instructions": "How would you categorize the purpose of this function?"
      }
    },
    {
      "type": "questions",
      "id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
      "attributes": {
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393",
        "external_key": "explanation_question_key",
        "category": "free_text",
        "instructions": "If you selected 'None of the above', please explain what this code does."
      }
    },
    {
      "type": "answers",
      "id": "7c998b53-13b6-43ca-b561-34912d5c4666",
      "attributes": {
        "question_id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
        "value": [
          {
            "value": "0",
            "label": "Mathematical Computation"
          }
        ]
      },
      "relationships": {
        "question": {
          "data": {
            "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
            "type": "questions"
          }
        }
      }
    },
    {
      "type": "answers",
      "id": "45bdf701-9300-4902-9187-8ca4f5ec79ff",
      "attributes": {
        "question_id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
        "value": [
          "This code calculates the factorial of a given number"
        ]
      },
      "relationships": {
        "question": {
          "data": {
            "id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
            "type": "questions"
          }
        }
      }
    }
  ]
}

When a task receives a new response, Revelo's API will send a POST request to the callback URL specified during project creation or editing. This request will include all relevant information about the response.

Please note that all values will be saved as strings, and the answers node will always be an array, even if there is only one value.

Retrieve all responses for a task

Endpoint

GET https://api.revelo.com/v1/tasks/<taskId>/tasker_responses

Example request

import requests

url = "https://api.revelo.com/v1/tasks/<taskId>/tasker_responses?page=1&size=20"

payload = {}
headers = {
  'Content-Type': 'application/json',
  'Authorization': '<your_api_key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

curl --location 'https://api.revelo.com/v1/tasks/<taskId>/tasker_responses?page=1&size=20' \
--header 'Authorization: your_api_key'

Example response

{
  "data": [{
    "type": "tasker_responses",
    "id": "78a3e098-01f4-495c-826e-9666e023b5a9",
    "attributes": {
      "task_id": "9b58544a-a850-4122-87dd-1b58da274393",
      "created_at": "2015-05-22T14:56:29.000Z",
      "updated_at": "2015-05-22T14:56:28.000Z"
    },
    "relationships": {
      "task": {
        "data": {
          "id": "9b58544a-a850-4122-87dd-1b58da274393",
          "type": "tasks"
        }
      },
      "answers": {
        "data": [
          {
            "id": "45bdf701-9300-4902-9187-8ca4f5ec79ff",
            "type": "answers"
          },
          {
            "id": "7c998b53-13b6-43ca-b561-34912d5c4666",
            "type": "answers"
          }
        ]
      }
    }
  }],
  "included": [
    {
      "type": "tasks",
      "id": "9b58544a-a850-4122-87dd-1b58da274393",
      "attributes": {
        "external_key": "your_unique_id",
        "code_snippet": "def factorial(n):\n    return 1 if n == 0 else n * factorial(n - 1)\n\nprint(factorial(5))  ",
        "code_language": "python",
        "required_task_responses": 5,
        "project_id": "8ca8edc4-91e6-4c21-be18-e783c7296880",
        "instructions": "Assess the following code"
      }
    },
    {
      "type": "questions",
      "id": "dabe2aab-dcf4-4bda-b5f6-327824fb0a6f",
      "attributes": {
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393",
        "external_key": "purpose_question_key",
        "category": "single_choice",
        "instructions": "How would you categorize the purpose of this function?"
      }
    },
    {
      "type": "questions",
      "id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
      "attributes": {
        "task_id": "9b58544a-a850-4122-87dd-1b58da274393",
        "external_key": "explanation_question_key",
        "category": "free_text",
        "instructions": "If you selected 'None of the above', please explain what this code does."
      }
    },
    {
      "type": "answers",
      "id": "7c998b53-13b6-43ca-b561-34912d5c4666",
      "attributes": {
        "question_id": "da611b23-ac8f-4842-aa41-fccda3ea2f1f",
        "value": [
          {
            "value": "0",
            "label": "Mathematical Computation"
          }
        ]
      },
      "relationships": {
        "question": {
          "data": {
            "id": "da611b23-ac8f-4842-aa41-fccda3ea2f1f",
            "type": "questions"
          }
        }
      }
    },
    {
      "type": "answers",
      "id": "45bdf701-9300-4902-9187-8ca4f5ec79ff",
      "attributes": {
        "question_id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
        "value": [
          {
            "value": "This code calculates the factorial of a given number"
          }
        ]
      },
      "relationships": {
        "question": {
          "data": {
            "id": "a6caafe0-dca4-4823-a415-05147f59e4c4",
            "type": "questions"
          }
        }
      }
    }
  ],
  "meta": {
    "total_count": 40,
    "total_pages": 2
  },
  "links": {
    "prev": null,
    "next": "https://api.revelo.com/v1/tasks/9b58544a-a850-4122-87dd-1b58da274393/tasker_responses?page=2&size=20",
    "self": "https://api.revelo.com/v1/tasks/9b58544a-a850-4122-87dd-1b58da274393/tasker_responses?page=1&size=20",
    "first": "https://api.revelo.com/v1/tasks/9b58544a-a850-4122-87dd-1b58da274393/tasker_responses?page=1&size=20",
    "last": "https://api.revelo.com/v1/tasks/9b58544a-a850-4122-87dd-1b58da274393/tasker_responses?page=2&size=20"
  }
}

This endpoint retrieves all task's responses.

Query Parameters

Parameter	Default	Description
taskId	-	The ID of the task.
size	20	The number of rows returned per page. Adjust this to control the size of each page.
page	1	The page cursor used to navigate through paginated results.