📘

Instructions should be a living document!

Writing task instructions is an iterative process. We very highly recommend multiple rounds of small batch submissions, reviewing the annotated data, and updating the instructions to clarify edge cases and examples.

Instruction Audience

The team doing your tasks and reading your instructions are comprised of talented, trained workers around the world.

To help us both out, we ask that you use clear, concise, and easy to understand language to describe your labeling requirements.

Keep in mind that English may not be our Tasker’s first language and there may be significant differences in cultural norms or environments.

Instruction Types

We have two main ways of generating instructions for our Taskers today:

1.Markdown

We are able to render markdown content to taskers. To leverage this option, you'd pass Markdown as a string directly to the instruction parameter on the project or task, or paste it directly into the Scale application when creating a project.

Tactically, when it comes to newline characters in your Markdown string, simply using \n (newline character) is sufficient.

An example instruction set using Markdown is at NuScenes (rendered | raw)

2.Google Docs

We generally recommend Google Docs as the preferred approach to generating instructions given the rich formatting and easy collaboration offered. The below instructions detail how to set up and use a Google Doc on our platform.

Getting Started with Task Instructions

1. The easiest way to start writing and manage instructions is simply using Google Docs. We made a template to get you started (click File → Make a Copy to start your own). We have created a helpful guide here.
2. After you have written instructions, you can embed the instructions into the Task Creation API call or save the instructions to the Project. You’ll want to follow our for instructions on embedding the Google Doc in your Task or Project.
3. Submit a small sample of tasks to Scale using your instructions. Once you get annotated data back, refine your instructions with additional examples or common errors as needed.

Embedding Instructions to Your Tasks

To embed your google docs instructions to your tasks:

1. In your Google Docs, click File →Publish To Web → Embed (in the window that appears)
2. Copy the iFrame tag that’s provided: <iframe src="[YOUR_CUSTOM_DOC_LINK]"></iframe>
3. Paste the iFrame tag in the instruction parameter on the task, or paste it into your project instructions in the UI.

📘

Project or Task Level Instructions?

In most cases, we recommend you to embed the instructions on the project level, not task level.

Instructions that are embedded on the project level will cascade down to all of the tasks within that project.

Example Instruction Sets

Example Template

Fictional Examples Below:

• Semantic Segmentation
• Polygon
• Categorization
• Transcription

Tips for Writing Instructions

We recommend our customers to use this template when creating instructions.

1. Include plenty of pictures and examples. Remember to include both positive (annotations to include) AND negative (annotations to not include) examples.
2. Use green to demonstrate correct annotations and red for incorrect annotations
3. If there are instructions that you want special attention, feel free to highlight or use bold text
4. Allow our team to quickly navigate your instructions by adding bookmarks to highlighted text and then linking them.
5. Use the relevant sections of the template effectively:
   • Summary of the task provide a concise overview of the task to provide context
   • Labels and attributes table create a list of your category, label, and attributes so our team can review them quickly
   • Annotation rules write general guidelines for how you want your labels to be drawn
   • Labels and attributes description and examples use negative and positive examples with lots of pictures
   • Common errors illustrate common errors so that our team can avoid them in the future

Example Dataset Workflow

After you have created instructions, it’s time to submit for your tasks to be labeled. Depending on your task complexity, it might be take a few iterations to create a comprehensive set of instructions.

If you don’t feel that we are aligned with your requirements, please refine the instructions document. It is helpful to:

1. Use simple and concise english to illustrate your task
2. Provide more examples or screenshots of correct and incorrect annotations.
3. Highlight common errors in a separate section.
4. Gradually increase batch size until you feel that we are aligned with your requirements

🚧

Instructions are iterative!

For your first batch, we recommend submitting a small sample (between 10-20) of tasks to be annotated. After you receive the results, please use our quality assurance tools to reject or approve tasks.