Study: Understanding How Diversity and Inclusion Statements Perform

We had a question: What is the impact of diversity and inclusion statements, or Equal Opportunity Employer (EOE) statements, on candidates from different backgrounds?

To answer that question and a couple of others, we decided to conduct an experiment. Studies are nothing unusual for TapRecruit. We rely heavily on behavioral science in addition to data science, although we don’t usually publish our findings. And we never incorporate research into our products unless we validate it, completely.

But with this study, we wanted to share our findings. Our ultimate goal was to understand whether the statements impacted how candidates felt about the job or the company.


Why do a study?

Data science has its limitations when it comes to recruiting, so we use behavioral science as well. Basically, we can use data science to bring up questions that behavioral science can then try to answer.

Over the years, TapRecruit has collected data from 10s of millions of job descriptions from more than 40,000 employers. We can use this data to classify jobs (for a better candidate experience) and to provide content and language guidance in our products. And we can use the data in our customers’ talent funnels to provide them with insightful recruiting analytics.

But we can’t work with data we can’t see. Once someone applies to a job, for example, we can learn about their candidate experience and qualifications. But if they never apply, we can’t ask them anything, especially (and crucially) about why they didn’t apply. That’s one limitation of data science in recruiting.

Another limitation has to do with job description content. Job descriptions have two main types of content: job-specific and company-specific. Job-specific content includes the title, responsibilities, requirements, and descriptions of the team. Company-specific content includes benefits, perks, a description of the company, and a diversity and inclusion statement.

Job-specific content is unique for every job description, which means we can begin to understand the impact of individual elements like the title. Company-specific content, however, is uniform across all company jobs, making it harder to single out the impact of individual elements. It’s also hard to delineate between the impact of company-specific content versus the impact of the company’s overall reputation. It’s a hard problem for data science to answer.


So, what did we find?

We found that job descriptions should definitely include a statement supporting diversity and inclusion. What it should say, though, is a little more complicated. (Our team had several theories about the type of statement that candidates might prefer. But, for the most part, our instincts were wrong. Which is exactly why we do research!)

Anyway, here’s an overview of our study.


Study Mechanics

How did we structure our study? We wrote a generic job description and attached one of five different diversity statements. We split participants randomly into six groups, one for each diversity statement plus a control group with no diversity statement.

STUDY MECHANICS / 01

A generic job description

We started by writing a job description for an Office Manager position. We made it generic and entry-level in the hopes that most participants would feel like they qualify for the position. (So they wouldn’t focus on or get hung up by whether they qualify.) 


STUDY MECHANICS / 02

5 distinct diversity statements

Next, we wrote five different diversity statements, each with a different slant or intent. We wanted to study whether the inclusion of a diversity statement had an impact on job seekers. But we also wanted to know how different types of diversity statements (in otherwise identical job descriptions) could impact job seeker perceptions. 

No Judgment

This one asserts that the company won’t judge you no matter who you are, so everyone gets a fair shot. It states that the company cares deeply about fairness and will never judge a candidate based on gender, race, age, religion, or anything else.

Accommodation

This one is all about accommodating workers to enable them to perform the job. It asserts that the company will provide reasonable accommodations to ensure that candidates can apply and employees can succeed in the job.

Legalese

This is fairly standard and lengthy legalese you may see a large corporation use. However, it goes a step beyond most legalese statements by including an exhaustive list of groups the company protects. The list includes things that candidates and employees may never divulge and are not required to divulge to employers (e.g., sexual orientation, genetic information, HIV/AIDS status).

Preference

This one attempts to welcome by singling out certain groups. It states that the company is ‘particularly’ welcoming to ethnic and racial minorities as well as other candidates. (And, yes, it’s problematic on its face because it shows a preference for certain groups.)

Results

This one attempts to prove a commitment to diversity by listing the company’s ongoing diversity efforts. It includes the results of the efforts right in the statement (e.g., ‘helped us increase the number of minority candidates we interview by 15%’).


STUDY MECHANICS / 03

A quick pilot survey

Any scientific endeavor starts with curiosity and certain assumptions. We were curious about the impact of a diversity and inclusion statement. And we assumed that job seekers would prefer the Results statement, which is written in accessible language (as opposed to Legalese). We’ve seen in other content that accessibility is appealing, which is why we suspected this might be the case here.

Had we stopped there, we would have carried on with our assumption. But we would have been wrong. And we would have risked making other incorrect assumptions based on that one, and on it would go. But, again, we don’t include anything in our products without validating the science behind it.

At this point, we decided to run a quick pilot survey to see if we could get a sense of people’s perceptions. We asked our own TapRecruit crew to read the various statements, rank them, and provide comments. It was a quick and dirty survey, but the results changed our expectations for the study overall.

As you now know, the Results statement was not the most preferred statement among our TapRecruit group. A couple of the comments we received about it:


It’s genuine but too transparent. It makes me feel like they’re still trying to get it right and that I am part of their experiment.

I’m not sure why, but this statement reads like somewhat of a vanity metric.


Instead, the Accommodation statement ranked highest. A couple of the comments we received about that one:


This feels like the most authentic statement because it describes the “what” and the “how” of a diverse culture that supports the success of all people. It’s easy to copy a list of identifiers from an EEOC site for the sake of compliance, but this statement explains how diversity and inclusion is part of the fabric of the company.

This statement goes beyond how I will be treated as a candidate and shows me how their culture is built and how I can participate. It makes me think I won’t be the only Person of Color or queer person to arrive on the scene.


Right. So, as you’ll soon learn, we got that wrong. And, again, this is why we do research. No matter how much experience is behind it, our ‘gut’ only gets us so far.


STUDY MECHANICS / 04

3 attributes + questions

After we had the diversity statements, we needed to decide on what attributes to measure. Our study ended up focusing on three attributes: Belonging, Inclusion, and Process Fairness.

Belonging was how comfortable participants would feel working at the company (e.g., “I could fit in well with this company”). Inclusion was how inclusive participants perceived the company to be (e.g., “I think the company is welcoming for People of Color”). Process Fairness was how fair the process seemed (e.g., “How fair do you think the company is in its recruiting processes?).

Note: These questions didn’t just come out and say what it is we were measuring. That’s because we don’t really want participants to understand what we’re after. There is a well-known phenomenon in social psychology called “participant bias” or “response bias” where participants may guess what the researcher is after and alter their answers, or unconsciously change their answers. As you can imagine, this can have a huge impact on research findings.


STUDY MECHANICS / 05

Translating questions into metrics

A note on consistency. Any study has to be reliable to provide usable data. (Reliability is how consistent the study is at measuring what it’s supposed to measure.) For our questions to translate into usable metrics, we needed to ensure that every question within a group measured the same concept. (In other words, every question in the Inclusiveness group had to measure perceptions of inclusiveness.) 

We calculated the reliability of our questions in capturing an idea using Cronbach’s alpha, a calculation used commonly in behavioral research. Calculating alpha yields a numerical range from 0 (no agreement among the elements of the group) to 1 (total agreement). For behavioral studies like this, the alpha value should be a minimum of 0.7. In our study, the alpha values of our groupings ranged from 0.81 to 0.91. (Note: Process Fairness only had one question.)


STUDY MECHANICS / 06

Conducting the study

This part was pretty straightforward. Using an online crowdsourcing platform, we enlisted approximately 1,200 participants from a wide spectrum of intersectionalities. (We asked about age, gender, LGBTQ status, race/ethnicity, proficiency in English, and whether the participant had earned a Bachelor’s degree.)

Everyone was over 18 years old and gave us informed consent to participate in the study. We split the group up randomly, which resulted in around 200 people assigned to each of the job descriptions.

Each participant’s job description had either no diversity and inclusion statement at all or one of the five statements above. (We know that many companies require EOE statements. But when we do controlled experiments, it’s important to compare to a control. This enables us to compare all of the conditions to a baseline and better understand the impact of the variables.)

Participants saw only one job description each. We asked them to look at the job description, answer our questions, and provide comments if they chose. Once we had all of our responses, we analyzed the data set using a statistical threshold that accounts for our multiple questions.


STUDY MECHANICS / 07

Caveats and considerations

No study is perfect and, of course, we have some caveats. Here are some of our considerations about this study:

  • Each participant saw only one job description, which is atypical for real-world job searches. It’s likely that participants over-interpreted or over-emphasized the impact of the job description. (This is what we call a “high false positive” context.)
  • It was hard to know which attributes to study. Therefore, we tested multiple attributes. We focused on three (belonging, inclusion, and process fairness) for the sake of storytelling in this article.
  • We were testing using natural language, which is incredibly complex and makes 100% conclusiveness impossible to attain.
  • We need more than 1,200 participants and will continue to do more testing to ensure robustness and reproducibility.
  • Just because we didn’t see a difference doesn’t mean there was no difference. For example, a job description that mentions specific groups in an EOE statement doesn’t perform worse than a job description that has no EOE at all, it just doesn’t perform any better. (Rather, the difference isn’t statistically significant.) In other words, “the absence of evidence is not evidence of absence.” That’s why it’s important we repeat the study.

Study Findings

What did we discover? Well, for one thing, we discovered that our theory about which statement participants would prefer was wrong. (We thought the Results statement would win out, but it didn’t. Again, why we do research.)

And while it’s important to emphasize that these results are not absolutely conclusive, they do offer some good insight. We had one big takeaway and several smaller takeaways. Here they are:

STUDY FINDINGS / 01

A diverse group of study participants 

The demographics of our participants aren’t exactly a finding, but we want to include them here because they’re interesting. In short, we were pleasantly surprised to find such a diverse group of individuals. Here they are:


STUDY FINDINGS / 02

Big takeaway: Any diversity statement is better than no diversity statement 

A diversity statement can increase a job or company’s perceived inclusiveness. That may just seem like common sense and something people in diversity and inclusion (maybe you) have assumed for a while now. But it’s important to see data back that up. In our study, the presence of a diversity statement significantly increased perceived inclusiveness among participants. In other words, just having it there made the company appear more inclusive. 


STUDY FINDINGS / 03

Non-male respondents showed diverse preferences

Non-male respondents were literally anyone who did not self-identify as male. Among this grouping, preferences were very diverse, although we do see some common ground on fairness and belonging.


STUDY FINDINGS / 04

LGBTQ responders showed split preferences 

LGBTQ individuals self-identify as lesbian, gay, bisexual, transgender, or queer/questioning. Other labels exist regarding sexuality, such as pansexual (attracted to all genders/identities), but we gave participants a choice of identifying as either LGBTQ or not. For respondents who self-identified as LGBTQ, the Legalese statement significantly increased company appeal and perceived belonging over no statement. Meanwhile, the Results statement increased perceptions of inclusiveness over no statement. 


STUDY FINDINGS / 05

People of Color showed a strong preference for the Legalese statement

Participants who self-identified as a race other than Caucasian, Asian, or Asian-American showed a strong preference for the Legalese diversity statement. As a group, the Legalese statement significantly increased perceptions of inclusiveness and fairness compared to no diversity statement. In fact, the Legalese statement was the only one that was significantly different from no diversity statement for People of Color. 


STUDY FINDINGS / 06

How overall preferences performed 

Overall, the Legalese diversity statement was viewed as both fair and inclusive by the widest spectrum of participants. It was particularly true for People of Color and for women and individuals self-identifying as non-binary. Among LGBTQ participants, the Legalese statement also increased company appeal and a perception of belonging.


STUDY FINDINGS / 07

Current takeaway on preference and intersectionality

Mentioning a range of comprehensive subgroups likely helps the performance of a diversity statement. So, listing race, gender, age, et cetera, can help.


STUDY FINDINGS / 08

Remaining questions

As with any study, we do have some lingering questions. We’ll be continuing to explore them and other related topics down the road.

  • We focused on a handful of EOE statement types which we saw frequently among our customers (and prospective customers). However, there are other archetypes of EOE statements that we did not use in this initial study. Are there other archetypes that communicate belonging, fairness and inclusion?
  • It wasn’t our intention, but the study was conducted shortly after the killing of George Floyd, in the midst of protests against racial discrimination and social upheaval in the United States. What impact did that broader context have on our study? Did it increase sensitivity to racial injustice among participants? We suspect it might have, and we plan to repeat the study to determine whether the preferences we observed were robust.
  • We were surprised by how popular the Legalese statement was with many different populations! We have more questions about what specifically increased its appeal. For example, did its length (2x the others) play a significant role in its overall performance? Did its more formal tone play a role? Or does the reference to legal outcomes create psychological safety among study participants? We will be further studying these questions, with a focus on populations that have been historically underrepresented in the workforce.
  • Lastly, while 1,200 participants is large for a behavioral experiment, it only captures a very small sample of U.S.-based jobseekers. To ensure that the results are robust, we need to replicate the study with a larger sample size of people (who hadn’t participated in this original study).

Conclusion: Yes, include a diversity statement

We started with the simple question of whether diversity statements on job descriptions impact perceived inclusiveness. While our study is not absolutely comprehensive, we feel we got a revealing answer to that question. Yes, a diversity statement (any diversity statement) is better than no diversity statement.

We hope to answer other questions with additional studies down the road. In particular, we want to explore the Legalese diversity statement more because it seemed to outperform all the others. For now, we know that hiring teams should include diversity statements on their job descriptions. With more research, we may be able to gauge what type of statement works best. Stay tuned!

Posted on Tags / Diversity and Inclusion|Recruiting Basics
Maryam J.
Research Scientist
Suggested Reading