WordPress.org

Resolved fartbrat
(@fartbrat)

4 months ago

Hi, may i know how do i add “Disallow: /cdn-cgi/” to the robots.txt provided by the plugin? I tried to create my own robots.txt in the root folder but on my website.com/robots.txt it is not overwritten.

Thanks

The page I need help with: [log in to see the link]

Viewing 1 replies (of 1 total)

Plugin Author Sybre Waaijer
(@cybr)

4 months ago

Hello!

As for the title of this topic, this appears to be a widespread issue with Lighthouse, regardless of the SEO plugin. Please see https://wordpress.org/support/topic/robots-txt-error-2/#post-17521699.

As for the content of your request, TSF’s robots.txt output is virtual; there’s no stored file. This is done to support complex networks, and WordPress itself actually manages the output.

Still, I recommend blocking robots to /cdn-cgi/ via .htaccess or a firewall rule instead of exposing the endpoint in the robots.txt file. But you can edit the robots.txt virtual output via this plugin: https://wordpress.org/plugins/robots-txt-quick-editor/.

Viewing 1 replies (of 1 total)

You must be logged in to reply to this topic.

In: Plugins
1 reply
2 participants
Last reply from: Sybre Waaijer
Last activity: 4 months ago
Status: resolved