The Fair Human-Centric Image Benchmark (FHIBE) dataset was created to evaluate fairness and robustness in human-centric computer vision applications across dimensions, such as data subjects (e.g., demographic information), instruments (e.g., camera hardware and software), and environments (e.g., illumination, camera distance), which are known to influence model performance.

The creation of FHIBE is motivated by the need for ethically-sourced, human-centric image datasets to audit and mitigate bias in computer vision models for a variety of relevant tasks. FHIBE is the first publicly available, demographically balanced, fairness benchmark with appropriate informed consent, licensing, compensation, and diverse, global representation. FHIBE was created intentionally with the following human-centric applications in mind:

FHIBE is strictly an evaluation dataset and may not be used to train machine learning (ML) or artificial intelligence (AI) software, algorithms, or other technologies. The one exception is that the dataset can be used for training bias detection or mitigation methods. The dataset is intended to support research, development, evaluation and improvement of commercial and noncommercial ML and AI software, algorithms, and other technologies, provided that these are aligned with the terms of use of the dataset. FHIBE can enable researchers and practitioners to carry out disaggregated analysis of computer vision models along relevant axes, such as ancestry and pronouns, to identify the specific groups for which a model may be underperforming. Furthermore, FHIBE can enable efforts to develop corrective measures that address these biases and minimize the potential harm that model performance disparities can cause.

The dataset was created by Alice Xiang, Jerone T.A. Andrews, Rebecca L. Bourke, William Thong, Julienne M. LaChance, Tiffany Georgievski, Apostolos Modas, Aida Rahmattalabi, Yunhao Ba, Shruti Nagpal, Orestis Papakyriakopoulos, Dora Zhao, Jinru Xue, Victoria Matthews, Linxia Gong, Austin T. Hoag, Mircea Cimpoi, Swami Sankaranarayanan, Wiebke Hutiri, Morgan K. Scheuerman, Albert S. Abedi, Peter Stone, Peter R. Wurman, Hiroaki Kitano, Michael Spranger at Sony AI.

The dataset was funded by Sony AI.

FHIBE can be accessed free of charge here.

FHIBE consists of diverse, crowdsourced digital photographs of 1 or 2 people, conducting actions in real-world environments, under various environmental conditions and camera devices, across the world. In addition to the primary dataset, we provide two derivative face datasets: an unaligned face-cropped dataset and an aligned face-cropped dataset.

FHIBE contains 10,318 images of 1,981 unique consensual image subjects. Of these, 1,711 are primary image subjects and 417 are secondary subjects. On average there are 6 images per primary subject. The cropped face dataset has 10,941 images for 1,981 image subjects. The cropped-and-aligned set has 8,370 images from 1,824 subjects.

FHIBE contains unique instances that have been directly collected from human subjects. The data collection from which FHIBE was curated contains more images than those included in the dataset.We excluded submitted images where consent could not be validated or that materially contained third party IP, including logos and copyrighted materials.

Each instance consists of a digital photograph file in PNG format. Accompanying each photograph is a separate metadata file stored in JavaScript Object Notation (JSON). The JSON metadata files contain information about the image subjects such as annotations of their physical attributes and actions, the camera settings used, and environmental factors such as lighting and the weather. An example photograph and the accompanying metadata are shown in Fig. 1 of the paper introducing FHIBE. An example JSON metadata file can be found on the FHIBE web portal where the dataset will be hosted.

The JSON files contain the following pixel level observational annotations for each consensual image subject in each image, obtained from a demographically diverse set of human annotators:

To accommodate fairness evaluations of a wide range of computer vision tasks, there are many labels associated with each instance, all contained within the JSON metadata file accompanying each image file. Refer to Section 3.1 for a full list of the additional labels collected for each image.

No information is missing from individual instances.

The relationship between each image and its corresponding JSON metadata file is made explicit through FHIBE’s file structure and file naming convention. Each primary consensual image subject has a unique identifier and folder, which contains subfolders for metadata and images of that subject. In addition, within each JSON metadata file, the relationship between images containing the same subject is made explicit.

FHIBE is strictly an evaluation dataset (with the narrow exception of training bias detection and mitigation methods) and may not be used to train machine learning or artificial intelligence software, algorithms, or other technologies. There are thus no recommended dataset splits.

The quality of the images and annotations was validated as described in the Methods section of the FHIBE paper. The self-reported attributes in the dataset, such as gender pronouns and age, are taken at face value as there is no way to validate them. The dataset also contains pixel level annotations drawn by humans, which contain some level of inherent noise.

An intentional redundancy is that the same subject appears in multiple photographs, as this is useful for evaluating some computer vision tasks. Furthermore, we recommend excluding images with two image subjects from face verification tasks, as primary and secondary image subject IDs have been allocated separately, meaning that in rare cases, a primary image subject can also be a secondary image subject with a different subject ID.

FHIBE is entirely self-contained.

No.

Vendors were explicitly instructed that images should not contain objectionable content, and QA specialists were hired by Sony to verify that such content was not present including, but not limited to, graphic violence, nudity, explicit sexual activity and/or themes, cruelty, and/or obscene gestures. The vendor’s annotators flagged images if they contained objectionable content, including, but not limited to, graphic violence, nudity, explicit sexual activity and/or themes, cruelty, and/or obscene gestures. Any image that was flagged was removed before delivery. As an additional check, the FHIBE authors and QA specialists verified the contents of the images delivered by the vendors and removed any remaining objectionable images. Before releasing the dataset, we performed an automated check for child sexual abuse materials (CSAM) against the National Center for Missing & Exploited Children’s hashed database of known CSAM.

Yes, FHIBE is a human-centric image dataset. Each image contains 1 or 2 consensual image subjects.

The dataset contains many demographic attributes for image subjects. While FHIBE was specified to have an approximately uniform distribution across 50 intersectional groups based on age, pronouns, and ancestry, the actual distribution diverges as a result of our crowdsourced data collection process. FHIBE is approximately balanced on pronouns and apparent skin color, though lighter skin colors are under-represented. Similarly, African and Asian ancestries, and younger age groups are overrepresented, while older age groups are underrepresented.The distribution of FHIBE across intersectional groups is shown in Section G.2 of the Supplementary Information accompanying the paper.

Subpopulations were identified based on the rationales described in the FHIBE paper and “Ethical Considerations for Responsible Data Curation”, Andrews et al., 2023.

Neither subjects nor annotators' names and email addresses are shared publicly, reducing the risk of re-identification. Additionally, we took privacy-enhancing measures to reduce the risk of re-identification, including: eliminating sensitive information related to the non-consensual individuals through full-body anonymization and text/multimodal personally identifiable information (PII) redaction, as well as redacting PII information from images with consensual subjects. We used a Subject ID and Annotator ID for unique (but not personal) identification:

While these measures reduce the risk of re-identification and our terms of use prohibit re-identification, anonymity of image subjects and annotators cannot be guaranteed. The dataset contains personally identifiable information (namely, images), and it will be made public. Image subjects and annotators were informed of the low risk of re-identification when they read and signed their consent forms to participate in the dataset.

Yes, for each consensual image subject, FHIBE contains the following data that is sensitive:

Each annotator optionally provided the following sensitive demographic information:

No.

The images were collected from 23 April 2023 onwards. The dataset was first published in 2025. Data subjects were able to submit photos they had previously taken, provided they were captured with a digital device released in or after 2011. The images in the dataset thus span a time frame from 2011 - 2024.

Data vendors were carefully selected and vetted by Sony, following a three-stage process. The initial vendor vetting process assessed the crowd diversity, annotation offering, timelines, and pricing of 10 candidate vendors. In addition, we checked online for any significant complaints or legal action against vendors by any previously contracted or employed image subjects and annotators.

Six shortlisted vendors then participated in a small-scale trial, where the quality of their deliverables, adherence to requirements and specifications, privacy standards and operational efficiency were evaluated. In the final stage, four vendors were selected after an in-depth review of their logistical setup for security-related risks and ability to meet relevant regulations, in particular with regards to privacy.

The following criteria were considered in the selection of vendors:

Prior to launching the data collection process, Sony validated for each participating vendor that:

Image subject and data annotator recruitment

The recruitment of data subjects was conducted by data vendors. Sony provided the data vendors with a set of diversity specifications that the collected images have to meet. These diversity requirements were placed at the aggregate level and not at the individual subject level. Vendors were required to abide by Sony's guidelines for recruitment.

Data subjects were eligible to participate if:

All project materials were provided in English. Data vendors were required to validate the English language ability of all data subjects. The English language proficiency test consisted of the vendor selecting three multiple choice questions to present to each data subject before they began working on the project. Vendors were provided with a question bank, and required to randomly select three questions to ask each data subject. In order for a data subject to qualify to participate in the project, they had to answer at least two out of the three questions correctly. Data subjects had two opportunities to take and pass the test. The results of the English language test were not shared with Sony.

In total, 296 annotators and 184 QA workers participated in the project.

In addition to vendor-contracted workers, we contracted QA specialists who worked with us internally to validate collected data.

Image creation

Image subjects could use a camera of their choice to create photographic images. As the type of camera used presents important instrument level metadata, image subjects had to report this, and it is included as metadata with each data instance. To ensure that images are of high enough quality to use in the dataset, data subjects were instructed to take images with a device that meets the following requirements:

In addition:

Uploaded images were retrospectively checked to ensure that they meet these requirements. For the majority of requirements the checks were automated.

The collection of the images cost around $308,500 USD. There were additional fixed costs of around $450,000 USD for quality assurance, legal services, and the cost of building the data platform. These cost estimates do not include the work of the 25 researchers, engineers, and program managers who contributed to the project.

We included all collected images that met our quality assessment, consent and privacy criteria in the released dataset. Furthermore, data records were selected to ensure that the final dataset has a balanced representation across intersectional demographic groups. For intersectional groups that had more images than what were required, we selected images using random downsampling.

The data collection was conducted by external, professional data vendors, including their in-house annotators and QA workers, and by existing and new crowd workers recruited by data vendors. The crowdworkers are image subjects and may be data annotators. We refer to image subjects and data annotators that provide personal information to the project as data subjects. Data vendors were permitted to subcontract data collection with prior authorization from Sony. All data subjects were compensated for their time at rates at or above the equivalent hourly minimum wage in their country of residence. Data vendors reported on their data collection progress on a regular basis.

Yes, ethical review processes and a data privacy impact assessment were conducted by in-house legal counsel, outside counsel, and Sony's AI Ethics Office. Before the start of the data collection, the project obtained IRB approval, which was independently reviewed by WCG (https://www.wcgclinical.com/about/). The collection of the data was considered to pose no more than minimal risk to data subjects, as it is unlikely that the action of image capture would result in physical or financial harm beyond the risks associated with the data subject's own daily life. The project also underwent Sony Research’s internal Research Ethics Review Process.

Yes, FHIBE is a human-centric image dataset.

The dataset was collected directly from individuals, through a compensated crowdsourcing approach. The crowd workers were recruited and managed by data vendors.

Yes, data subjects were aware of the data collection. Further, crowdsourced data subjects could only participate in the data collection after signing a consent form. In the consent form image data subjects were informed that their data was being collected on behalf of Sony. No data was collected from image subjects whose ability to protect themself from neglect, abuse, or violence could have been significantly impaired on account of disability, illness, cognitive impairment, or otherwise (i.e., vulnerable adults). Sony provided specifications for the content to be included in messaging sent to potential primary image data subjects. Messages sent to image subjects had to state the following:

Sony required explicit informed consent from each person depicted in or otherwise identifiable by the dataset using a form designated by Sony. Image data subjects voluntarily consented to the following:

The consent form was made available to, and signed by, image subjects electronically. All subjects were provided with a hardcopy or an electronic version of the consent form.

Withdrawal from the project during data collection

While data collection was ongoing, data subjects could discontinue their participation without any repercussions by exiting out of the user interface. All data of data subjects was deleted if they submitted a request for this.

Consent revocation after data collection

Sony has made provisions to fulfill ongoing operational obligations to service consent revocation requests from data subjects within a 30 day timeframe for as long as the dataset is available. In this way we uphold a relationship with, and enable ongoing participation from, data subjects, as well as dataset users, who must agree to and comply with data processing obligations while they are licensees of the dataset.

The provisions that have been made entail the following:

Data subjects were informed and agreed that while a revocation of consent would result in the deletion of their data from the dataset maintained by Sony, the revocation could not affect any publications, software, algorithms or technology derived from their data by third parties prior to the revocation.

In addition to the right to revoke consent, individuals can request a copy of their data, can submit complaints, and may have other rights depending on the applicable law of their country of residence. Data subjects were provided with a copy of their Participant Informed Consent and Release Form.

The various assessments that Sony conducted are described in Question 3.7. Here we list risks that have been identified, and actions taken by Sony to mitigate these risks.

Risk for data subjects

There exists a possibility that upon public release of the dataset, image information or annotations could be:

The likelihood of harm from these activities is comparable to the risks that individuals take in everyday life when posting images and personal information to public websites or on social media platforms. Sony has taken extensive mitigating actions to protect data subjects:

Data access control: The dataset will be made available on request to users after they agree to our terms of use.

Risk for annotators

Mitigating actions taken by Sony to protect data annotators:

Other risks

There exists a possibility that some of the images that appear in the FHIBE dataset were submitted by bad actors who scraped them from the web.

Mitigating actions taken by Sony to protect users and people from other risks:

Sony deployed a reverse image match using Google’s image web search API. While helpful, such automated checks are not totally accurate when applied to person detection.

No.

Preprocessing

Prior to the release of the dataset, we took extensive privacy-preserving measures to reduce the risk of re-identification of data subjects. This includes the following:

In particular, based on a list of 42 image-level privacy attributes assembled from sources such as the US Privacy Act 1974 and the EU Data Protection Directive 95/46/EC, the following image-level attributes were identified as containing personally identifiable information, and have been redacted:

Yes, the raw data has been saved but is not publicly available.

https://github.com/SonyResearch/fairness-benchmark

Users are able to create derivatives of the dataset annotations, provided that these annotations are explicitly identified as having been added by the user, and that they comply with the dataset's acceptable uses. We do not prescribe a license for derived annotations. For clarity, users are not able redistribute or create derivative works of the dataset images themselves.

Yes, the dataset has been validated for use as an evaluation dataset on the following tasks:

The dataset and all supplementary information are available at https://ai.sony/fairness-benchmark. We will keep an internal record of all third parties who download the dataset, their affiliation and intended use so that third parties can receive dataset updates in connection with consent revocations. We will not track technical usage of the dataset.

The dataset could also be used for:

Images capture a moment in time. In this regard, FHIBE reflects the image subjects and their environment as they were at the moment of data capture, not as they will be in future. This may be particularly relevant to ornamental, cultural, contextual and environmental information contained in images. While these changes may only be observable over time horizons of a decade or more, they may result in distribution shifts that lead to outdated and stereotypical representations of people.

It is unlikely that the action of image capture would result in physical or financial harm beyond the risks associated with the subject's own daily life, as subjects are able to submit pre-existing images. Subjects’ names and email addresses will not be shared publicly, and individuals accessing the image set will be contractually restricted from attempting to re-identify subjects, along with other reasonable use restrictions.

FHIBE is strictly to be used for evaluation and for fairness purposes. FHIBE cannot be used to train machine learning or artificial intelligence software, algorithms, or other technologies. The sole exception is the usage of the data for the explicit development of tools to assess fairness and mitigate biases in machine learning models and implementations.

It is prohibited to attempt to re-identify data subjects. Furthermore, the extraction, inference or prediction of image subjects' sensitive or subjective, non-observable features, such as gender, race, ethnicity, political opinion, religious beliefs, genetic data and any other group membership is forbidden.

The dataset may not be used to evaluate tools that perform biometric categorisation, emotion recognition, physiognomic recognition, that are used for law enforcement purposes, suspicious behavior detection, and specific high risk applications that appear in settings such as education, work, social credit systems, immigration, and law.

No.

Yes, FHIBE will be publicly available for download and use by varying audiences, including research, academic, and for-profit entities.

FHIBE is accessible via an online platform. To download the data, users will need to sign up and create a user account on the platform by providing their name, organization, intention for using the data, and email address. Users will also need to agree to the license and terms of use of the FHIBE dataset. The FHIBE platform contains additional information on how to obtain the dataset.

FHIBE will be released with a DOI.

FHIBE will be made publicly available here.

FHIBE is entirely owned by Sony AI and is distributed under applicable terms of use and custom license.

FHIBE Terms of Use can be found here.

No.

Users are restricted from accessing FHIBE from a country that is subject to a U.S. Government embargo.

N/A

No.

FHIBE is supported, maintained, and hosted by Sony AI.

Sony AI can be contacted using the contact form on the FHIBE platform.

There is not an explicit erratum. In the event that any errors are found, FHIBE will be updated and a new version will be released. Updates regarding FHIBE will be communicated via the FHIBE platform and via email to FHIBE users (i.e., those who agreed to the FHIBE license) if necessary.

FHIBE will be updated when image subjects withdraw their consent. In such a case, instances where consent has been revoked will be replaced and a new dataset will be released under a new version. When FHIBE is updated, FHIBE users will be contacted via email and instructed to delete the prior version of FHIBE. Further details on how images will be replaced in the case of consent revocation are detailed in FHIBE’s Methods.

FHIBE is designed to comply with the GDPR, including its data minimization principles. The retention of an individual’s particular data in FHIBE is subject to applicable law and data subject revocation, and in no way will be retained for more than 20 years. The FHIBE project itself may exist beyond 20 years, as data gets replaced over time.

Older versions of FHIBE will be withdrawn. FHIBE users must delete the prior version of FHIBE as agreed in Terms of Use.

Yes, the terms of use permit the addition and modification of the annotations, provided that the modifications do not otherwise violate the terms of use. Any modified version of the annotations must clearly identify the modifications from the official FHIBE dataset. Sony does not intend to validate or verify versions of the dataset created by others containing modified annotations.

No.