The essential guide to content moderation

Content moderation is the act of applying a set of guidelines to text, images and video that appear on a website, often with a particular focus on user submissions. It involves monitoring and identifying potentially harmful content, assessing whether it complies with the site’s guidelines and filtering out anything inappropriate.

Moderating content is a complex task, often involving many processes. A clear understanding of the process can help you to build a closer relationship with your customers and protect your brand’s reputation.

What constitutes sensitive content?

The main purpose of content moderation is to remove inappropriate content from a website. This often comes in the form of graphic or extreme content that contains violence, hate speech or nudity. Depending on each site’s specific requirements, moderation can be undertaken to a greater or lesser extent. Some message boards pride themselves on freedom of speech, while others, such as social networking sites, have to strike a difficult balance between ease of use and protecting their younger users.

However, there are also a range of user experience (UX) challenges to keep in mind when moderating content. Users have come to expect consistently high standards across the websites they frequent, which also stretches to user submissions. As such, it’s important to quickly identify and remove duplicate content through a process called deduplication. Similarly, low-quality image or video submissions are also a frequent target of moderation efforts.

A large number of sites also have legal issues for their moderators to consider. For example, libelous or copyrighted content can cost a company thousands of dollars in legal reparations. It’s a matter of urgency for many businesses to identify and take down this content before it’s flagged by any external party.

Which content types can you moderate?

In addition to the many different ways that content can be sensitive, there are also several different types of content that moderation services have to contend with. Content moderators or algorithms will usually have to deal with at least one of the following content types.

Text

The sheer variety of text that requires moderation on a site can be staggering. From comments and forum threads to full-length articles hosted on your site, almost any type of text can require assessment. As such, moderators and moderation algorithms must be adept at scanning texts of varying lengths and styles for unwanted content.

Furthermore, text moderation can be an extremely difficult task due to the complex nature of language. In order to detect cyberbullying or hate speech, for example, it’s necessary to move beyond explicit keywords and look at whether phrases, sentences or even paragraphs as a whole breach your community code of conduct. While text may not contain any obvious indicators of sensitive content, it may still contain behavior that is extremely damaging to your site.

Image

Although it might seem simple to identify inappropriate images, there are many challenges to consider when moderating them. For starters, detecting nudity or explicit imagery in user-submitted content can sometimes depend on context. What constitutes an indecent image in the U.S. is very different to an indecent image in Saudi Arabia. Some companies may also draw the line in different places depending on their product, such as lingerie brands. As a result, image moderaters also have to consider the target audience, market and company in question as they monitor a site.

Video

Video is one of the most difficult types of content to moderate. While images and text can often be vetted quickly, video can be extremely time-consuming, forcing moderators to watch a video all the way to the end. If only a few frames of video are explicit, it could drastically change the viewer’s perception of the site that hosted it. If your platform allows video submissions, searching for these hidden breaches of community guidelines can have a significant impact on your moderation efforts.

Video moderators are also required to perform several tasks simultaneously. In addition to the actual video content, any attached audio or subtitles need to be vetted for explicit language or hate speech. Separate from the actual video content, audio or subtitles may not match the video accurately or meet the necessary quality thresholds. Due to the many market, audience and quality concerns outlined above, video moderation is a formidable challenge.

Types of content moderation

The content moderation method that makes the most sense for you will depend on your website’s goals. It’s important to consider whether you want people to be able to communicate quickly and easily, or whether it’s more important to keep your site completely free of sensitive content at all times. There are a range of different types of content moderation which fall at varying points on the spectrum between these two goals. The most common types are outlined below.

• Pre-moderation: Unsurprisingly, this involves all user submissions being placed in a queue for moderation before they are displayed on the site. Through pre-moderation, it’s possible to keep all sensitive content off a site by checking every single comment, image or video. However, for online communities that prize immediacy and barrier-free engagement, this moderation method can cause challenges. It’s best suited to sites which need high levels of protection, such as those frequented by children.
• Post-moderation: In cases where user engagement is important but a comprehensive moderation program is still required, post-moderation is often a good choice. It allows users to publish their submissions immediately, but also adds it to a queue for moderation. Since every comment is approved by a moderator, scalability can be an issue.
• Reactive moderation: For a scalable program that relies on community members, reactive moderation is a possible solution. This type of moderation asks users to flag any content that they find offensive or that breaches community guidelines. By involving users in the process, reactive moderation directs moderator efforts towards the content that most needs their attention. However, there’s also the risk that offensive content will remain on site for long periods of time and possibly damage the reputation of the brand.
• Supervisor moderation: Similar to reactive moderation, supervisor moderation involves selecting a group of moderators from the online community. Also known as unilateral moderation, this system gives certain users special privileges to edit or delete submissions as they use the site. If supervisors are selected carefully, this method can promptly remove sensitive content and is easily scaled as the community grows. However, it is also prone to the negative effects outlined above if moderators miss offensive text, images or video.
• Commercial content moderation (CCM): CCM mainly involves monitoring content for social media platforms. It is often outsourced to specialists, who are tasked with ensuring that the content on a platform abides by community guidelines, user agreements and legal frameworks for that particular site and market.
• Distributed moderation: As one of the most hands-off moderation systems, distributed moderation places a lot of trust and control in the hands of the community. It usually involves allowing users to rate or vote on submissions that they see, flagging content that goes against any guidelines that are in place. This often takes place under the guidance of experienced moderators and can work well if a site has a large and active community. Distributed moderation systems remain somewhat rare given the possible risks of allowing a community to almost entirely self-moderate.
• Automated moderation Automated moderation is an increasingly popular moderation method. As the name suggests, it involves the use of a variety of tools to filter, flag and reject user submissions. These tools can range from simple filters, which search for banned words or block certain IP addresses, to machine learning algorithms, which detect inappropriate content in images and video. Many of these tools are used in addition to some kind of human moderation.

How is machine learning used in content moderation?

Machine learning algorithms are implemented in a variety of ways to help remove the burden of content moderation from human workers. These algorithms are built on the work of their human predecessors, since they are trained using large datasets of previously tagged content. From this huge bank of relevant examples, the algorithm extrapolates the rules that govern the distinction between safe and sensitive content. An intricate understanding of these rules allows algorithms to flag explicit material with increasing accuracy.

The next step in the development of these models involves building out the capabilities to deal with some of the more complicated instances of inappropriate content, such as cyberbullying. Future content moderation tools will be able to calculate a relative ‘risk’ score for a piece of content, before determining when and if it should be reviewed.

For now, the sheer complexity of content moderation means that full automation remains a dream from the distant future. However, as the industry evolves, human-in-the-loop moderation workflows are becoming far more common. These build on the strengths of both human and machine, allowing the algorithm to square away a large proportion of inappropriate content while referring difficult, subjective content to their human overseers. In this way, the continuing expansion of AI in this field will help to protect not only internet users, but also moderators from the worst of the world of content.

Why is content moderation important?

It’s absolutely crucial to build a solid and lasting connection with your customers. Creating an online presence where people can engage with your brand will not only expand your customer base, but also give you valuable insight into how to improve your product.

However, these strategies come with risks attached. By opening your business up and creating a community, you also open the doors to inappropriate content from some of the darker sides of the internet. It’s easy to argue that this only affects companies with certain products, but in reality it affects all businesses. It only takes one piece of inappropriate content to destroy a relationship with a potential customer, after which it’s difficult to win them back.

By using content moderation to enforce high standards upon your site, you protect yourself from a variety of legal issues, protect your growing community and ultimately show that you care; not just about the environment you create, but about the way that your company engages with the world.