Should search engines index your feature request board?

A public feature request board is an odd kind of website. Most of its words are written by customers, many of its URLs are the same list in a different order, and some pages exist only for the person who just submitted something. Letting search engines in is a real decision, and one answer for the whole board is usually the wrong one.

The short version: index the pages that answer a question a stranger might search for (each request, the plain list, the roadmap and the changelog), and only once they have something on them. Keep out every form, every personal state, every re-sorted copy and every empty shell. Use a crawlable noindex for that, not robots.txt.

Why index a request board at all

Requests are written in your customers' words rather than your marketing vocabulary. A prospect who searches for your product's name plus "export to CSV" or "SSO" is asking exactly what a request page answers. "Planned, 48 votes, here is the thread" is a better answer than nothing, and a lot better than a competitor's comparison page guessing on your behalf.

Changelog entries do the same job for features that already exist. They are often the only page on your site that names a small feature at all, and they carry a date, which is what someone checking whether a gap has closed needs.

There is a cost, and it comes before any tag. Google's spam policies list user-generated spam (spammy posts on forum threads, spammy accounts on services anyone can join) as the site owner's problem, whether or not the owner noticed. A board that publishes every submission the moment it arrives is an open comment form with your domain on it. Only index a board whose new posts are reviewed before they appear, or at least watched closely.

The policy, page by page

The sections below are the reasoning behind each row.

Use noindex, not robots.txt

A request board links to its own sign-in page, its submit form and its sorted views from every page. Disallow them and you get the worst combination: URLs that can be listed with no title and no description, because the crawler knows they exist and was told not to look.

So allow crawling, and mark the pages to keep out with <meta name="robots" content="noindex">. For responses that are not HTML, such as the RSS feed, send the same rule as an X-Robots-Tag: noindex header, which the same documentation covers.

Re-sorted and filtered lists

A board usually offers "top" and "new" orderings and a filter per status. Each combination is a URL, and each repeats requests that already have their own pages. Google's pagination guidance has a section for exactly this, "Avoid indexing URLs with filters or alternative sort orders", and recommends a noindex robots meta tag on those variations.

Keep follow on those pages, though. A sorted view is often the only route a crawler has to a request that sits low in the default order, and noindex, nofollow throws that route away for nothing.

Pagination: give every page its own canonical

The tempting shortcut is to point page 2, 3 and 4 at page 1 with rel="canonical". The same Google page says not to: "Don't use the first page of a paginated sequence as the canonical page. Instead, give each page its own canonical URL." Canonicalizing later pages to the first tells Google they are copies of page 1, and the requests listed only on them lose their crawl path.

What works:

• Each page has its own URL, such as ?page=3, and names itself as canonical.
• Pages link to the next one with a real <a href>, not a button that loads more with script.
• A page number past the end does not render an empty list with a 200. Return a 404, or at least noindex it.
• rel="next" and rel="prev" are optional. Google states it no longer uses them, though other engines may.

Empty pages are soft 404s

A board that was switched on yesterday has a list with nothing in it, a roadmap with three empty columns and a changelog that says "No updates yet". To a person that is fine. To a search engine it is a page whose content "suggests an error", which Search Console reports as a soft 404. Indexing it gains nothing and spends a little of the board's credibility.

The fix is mechanical. Decide indexability from the same data the page renders: noindex while the list is empty, and drop it the first time an item is public. Do the same for a roadmap whose columns are all empty and for a changelog with no published entries.

Requests that are not public must not exist

Pending, declined and hidden requests are a privacy question before they are a search question. If a request is not public, its URL should answer 404, the same 404 as a request that never existed. A noindex page that still renders a declined request's text is a leak with a polite tag on it.

The same goes for the board itself. A board that has been switched off, or whose project is archived, should answer exactly like an address that was never used. Anything else confirms to a stranger that something was there.

One URL per request

Request titles change. Someone fixes a typo or merges two requests and renames the survivor. If the slug is part of the URL, a renamed request either breaks old links or ends up with two live addresses.

A pattern that holds up: put a stable number in the path and treat the slug as decoration, as in /p/acme/api/42/csv-export. When a request is fetched with an old or mistyped slug, answer a permanent redirect to the current one. The number keeps old links working, the slug keeps the URL readable, and there is only ever one canonical address. Google's guide to consolidating duplicate URLs ranks canonicalization signals by strength and puts redirects first, ahead of rel="canonical".

Links inside user posts

Customers paste links into requests and comments. Google asks for rel="ugc" on links in user-generated content, and it can be combined with nofollow (rel="ugc nofollow"). That tells search engines the link is not your endorsement, and it removes most of the reason anyone would post spam links on your board in the first place.

Structured data for request pages

A request with votes and replies has the shape of a forum thread. Google reads DiscussionForumPosting markup for forum-style pages and requires an author, a publish date and the post's text. Vote and comment counts fit interactionStatistic, and the replies fit comment.

Two cautions. Mark up only what the page shows: if your team answers as "Acme team" rather than as a named employee, the markup should say the same. And escape the JSON: a request title containing </script> will close a naive inline script tag, so write < as \u003c when embedding user text.

A short checklist

• Board-level opt-in, off by default. Not every customer wants their requests in search results.
• Review new requests before they appear, or keep the board out of the index.
• noindex for forms, sign-in, unsubscribe pages and empty lists. Crawlable, never disallowed.
• noindex, follow for sorted and filtered views.
• Every page of the default list self-canonical, linked with <a href>.
• 404 for anything not public, identical for every reason.
• Stable number plus redirecting slug for each request.
• rel="ugc nofollow" on links people post.
• Forum markup that matches the visible page, with user text escaped.

None of this needs a plugin or a ranking trick. It needs the indexing decision to be made from the same data that renders the page, so the two cannot disagree.