Definition

Content moderation occurs when an online platform monitors user-generated content based on specific rules and guidelines to determine if the content should be published on the platform, or not.

Overview

Social media operators maintain policies that prohibit users from posting certain content, such as content that exhibits graphic violence, child sexual exploitation, and hateful content or speech. An operator may temporarily or permanently ban users that violate its policies, depending on the operator's perspective on the severity of the users' violation(s). There is no uniform standard for content moderation, resulting in practices varying across social media sites. Some operators have chosen to release reports containing information on their content moderation practices, such as the amount of content removed and the number of appeals, but operators are not required to release this information.

Social media operators rely on several sources to identify content to flag or remove: (1) users, (2) content moderators, and (3) automated systems, also known as artificial intelligence (AI) technologies. Users can flag or mark inappropriate posts for content moderators to review and remove when applicable. Automated systems can also flag and remove posts. Content moderators, primarily contractors, may be able to identify nuanced violations of content policy, such as taking into account the context of a statement.

Some social media operators may be compelled to rely more heavily on AI technologies to moderate content. Some commentators have raised concern about whether repeatedly reviewing graphic, explicit, and violent materials harms content moderators' mental health.

Source

• Overview section: Social Media: Misinformation and Content Moderation Issues for Congress, at 6.