Annotation tasks include:

All of these annotations present a degree of subjectivity. As far as possible, subjective aspects of annotation tasks have been reduced by:

Despite taking care to reduce subjectivity, annotations will always have variability. For example, pixel level annotations, such as masks, will not be drawn identically by two different people. Even physical and observable labels, such as lighting conditions or eye color, may be interpreted differently by different people.

Annotations that were not self-reported are the following:

We assumed that vendors would be able to meet Sony’s specifications for annotator diversity.

Data collection commenced in incremental batches, initially beginning with small ones and gradually scaling up to larger batch sizes over time. This approach helped us to address issues during annotation, such as questions, misunderstandings, inconsistencies and quality assurance challenges.

We provided vendor QA workers with example images for all annotation types and choice options. We sourced multiple images for each choice for each annotation type (e.g., short head hairstyle), with examples spanning across images of people with different skin tones and perceived genders. Where possible, we disambiguated text-based options by including images during the annotation task (e.g., color palettes for skin tone selection).

The following training materials were provided to annotators:

This question is also answered in Section 3.13 of FHIBE’s datasheet. This is a copy of the response from the datasheet.

Risk for data subjects

There exists a possibility that upon public release of the dataset, image information or annotations could be:

The likelihood of harm from these activities is comparable to the risks that individuals take in everyday life when posting images and personal information to public websites or on social media platforms. Sony has taken extensive mitigating actions to protect data subjects:

Data access control: The dataset will be made available on request to users after they agree to our terms of use.

Risk for annotators

Mitigating actions taken by Sony to protect data annotators:

Other risks

There exists a possibility that some of the images that appear in the FHIBE dataset were submitted by bad actors who scraped them from the web.

Mitigating actions taken by Sony to protect users and people from other risks:

Sony deployed a reverse image match using Google’s image web search API. While helpful, such automated checks are not totally accurate when applied to person detection.

Extensive guidelines were provided to vendors to create or validate the following annotations:

To standardize self-reported attributes, Sony provided a document with recommendations on how to report these attributes and any external annotations. The document lists the available choices for attribute values, the range of the values and their format. To avoid mistakes during annotation, attributes had to be enumerated following the numeration convention provided by Sony.

The FHIBE dataset collection and annotation requirements have been intentionally designed with diversity in mind. As the purpose of this dataset is to provide a general-purpose consensual human-centric image dataset of human bodies and faces, there were no particular perspectives that were considered more important than others. However, representation of diverse image subjects was prioritized and carefully considered during data collection. Similarly, we strived to obtain a diverse set of annotators by placing diversity requirements on the recruitment of data annotators.

Image subjects were specifically instructed not to submit images that contain offensive content whatsoever, including, but not limited to: explicit nudity or sexual content; violence; visually disturbing content; rude gestures; drugs; hate symbols; and vulgar text. Vendors checked for this content during their internal quality assurance process, and our internal quality assurance team conducted additional checks. Offensive content, images with third party IP like logos and trademarks, or images that did not meet specification criteria were withdrawn from the data collection. Sony also performed an automated check for child sexual abuse materials (CSAM) against the National Center for Missing & Exploited Children’s hashed database of known CSAM.

In addition, we did not include annotations or images from people under the age of majority as it is difficult to ensure their informed consent. We also did not obtain images or annotations from residents or nationals of (OFAC) sanctioned countries.

Recruitment

Before starting recruitment, vendors were contractually required to submit any text or images that were used for project-related recruiting activities to Sony for approval. Furthermore, before starting any data collection or annotation, vendors were contractually required to submit a methodology report to Sony, which included the following details on recruitment:

We continuously engaged vendors to ensure that reports were submitted. The submitted documents were validated on an ongoing basis to ensure that our requirements are met. We initiated correcting actions from vendors where necessary.

Diversity requirements

Sony requested that vendors use demographically diverse annotators with regards to their age, pronouns, and ancestry. If a vendor had multiple annotation and quality assurance stages, annotators at each stage were required to be demographically diverse. The following categories were considered for diversity:

Note regarding pronouns:

Note regarding ancestry:

While we strived to obtain a diverse and representative set of annotators across intersectional demographic subgroups, this was not always feasible for operational reasons. We thus relaxed our requirements and accommodated the annotator pools that were accessible to vendors, allowing for a 'best effort' approach from vendors to meet the diversity requirements.

Self-reported annotator details

We collected the following self-reported demographic metadata of annotators:

Providing self-reported demographic information was a requirement for crowdsourced annotations, which was further detailed in the project’s informed consent required to join the project as an annotator. For vendor employees who worked as annotators, submitting demographic information was optional.

Annotator demographics are provided in Section D of FHIBE’s supplementary materials.

We collected annotator demographics where possible because we acknowledged a priori that annotator attributes may influence perception and thus also annotation. We strived to obtain a diverse set of annotators, but it was not always feasible for operational reasons. Additionally we took the following measure to reduce the influence of annotator bias:

FHIBE is intended to be used as a diverse human-centric dataset to evaluate a variety of computer vision tasks. The dataset has not been created for a specific context of use. However, the individuals represented in the dataset are diverse across multiple demographic axes and the annotators that labeled the dataset are as diverse as possible, thus supporting the use of FHIBE for evaluating human-centric computer vision models in geographically diverse contexts.

The choice of annotation platform was at the discretion of vendors, provided that it met the specifications that we provided.

The specifications that we provided to vendors described the desired properties of our dataset and associated annotations. The specifications relevant for the vendors’ choice of annotation platform included:

Yes. In some minor cases, when a vendor’s annotation platform was unable to meet our specifications, we slightly relaxed our requirements. For example, our specifications required that if an annotator self-selected a pronoun of 'none of the above' or 'prefer not to say', then they were prohibited from selecting an additional pronoun option. One of the vendor platforms could not limit the number of pronoun selections. We allowed the vendor to contact the annotator to resolve ambiguity when this situation arose. 

During the annotation process, we held weekly FAQ sessions with data vendors to address any questions that arose. Furthermore, the informed consent form included an email address for Sony that data subjects could contact if there were any questions about the project. We closely monitored the email account for any correspondence. 

Sony required vendors to pay, at minimum, the legal minimum wage per hour of work to annotators. Minimum wage was based on the country in which the annotator resides. In the case of countries with no legal minimum wage, vendors were instructed to propose a rate and gain Sony’s approval in advance.

Before commencing any annotation work, vendors were obligated to submit a methods report as part of the deliverables. If vendors required updates to their methods, the report had to undergo re-approval before any changes could be implemented. The report documented:

We conducted intermittent test-driven consistency and reliability checks during the data collection process, and used the results as a measure of annotation quality. The methods for annotation quality assurance are described in detail in the FHIBE paper’s Methods supplement. The process for testing intra- and inter-vendor annotation reliability is expanded on here:

For all tests, the calculation of values took place at the final output of the annotation pipeline, i.e., after the last QA round.

Additionally, we compared the vendor-provided annotations with the average annotations from three internal expert annotators on a randomly sampled set of 500 images for each annotation type.

The type of tests that were conducted depended on the type of annotations, as described below.

These features are: age, pronoun, nationality, country of residence, ancestry, disability/difficulty, natural skin tone, natural eye color, natural head hair type, natural head hair color, natural face hair color, weight, height, biologically related subjects, pregnancy status

These features are: nonconsensual person visible, apparent skin tone, apparent eye color, apparent head hair type, apparent head hair color, hairstyle, facial hairstyle, apparent face hair color, facial marks, body pose, subject-object interaction, subject-subject interaction, illumination, one-hour time window of capture, date of capture, weather, scene, camera position

Keypoints

Segmentation masks

Face bounding box

We evaluated the robustness of some of the image/subject attributes as well as the pixel level annotations (face bounding boxes, segmentation masks, and keypoints). The test results are summarized in Section E of the FHIBE paper’s Supplementary Materials.

For pixel level annotations, agreement between collected and in-house expert annotations are above 90 %, at a similar or higher level as related works, proving the robustness and quality of collected annotations. Furthermore, intra- and inter-vendor agreement is also close to or above 90%, confirming a high quality of collected annotations.

For person and image-specific attributes, intra-vendor agreement is above 80% and inter-vendor agreement is at 70%, which indicates that labels are more noisy.

Test results were discussed with vendors to provide feedback for quality control during annotation. Our ongoing feedback mechanism with vendors ensured that all vendors passed intra-vendor tests regarding annotator agreement of keypoint annotations, mask annotations, union of face bounding boxes, and dice scores of masks. While collecting the data, we corrected disagreement patterns that we observed.

Not assessed.

FHIBE will be updated in connection with consent revocations that impact images in the dataset.

With the exception of self-reported demographic labels, all annotations are of physical, observable traits. Culture, language and geographic borders are dynamic. Thus, it is possible that certain labeling choices that were made, such as pronouns, will need to be adapted over time. Similarly, fashion and places change. Thus, clothing, face masks, objects and places of cultural and social significance in this decade may look very different to the same artifacts and places several decades from now. New artifacts may also be invented that are not captured in this dataset. This is to be expected, and the dataset will need to be assessed against how the world has changed in future.

Yes, in order to download FHIBE, dataset users must agree to the license and the terms of use. Furthermore, dataset users also need to state their intended use of the dataset in the access form and provide a contact email address. The license and terms of use are available in full at the FHIBE web platform.

The terms of use stipulate, among other restrictions, that FHIBE cannot be used to train machine learning or artificial intelligence software, algorithms, or other technologies. The sole exception is the usage of the data for the explicit development of tools to assess fairness and mitigate biases in machine learning models and implementations.

Yes, each annotator signed an informed consent form as well as other documentation describing the nature of the FHIBE project and how their data will be used. No personally identifiable information of the annotator will be externalized.

No personally identifiable information of the annotators will be disclosed in the dataset. However, crowdsourced annotators can revoke their consent with respect to the personal information controlled by us. Such requests will not result in the removal of other annotations, such as keypoints and segmentation masks that they drew or evaluated for quality assurance.