Sign in

Update robots.txt: New OpenAI Web Crawler (User-agent: GPTBot)

Forum › Feature Suggestions

  • This topic has 4 replies, 4 voices, and was last updated 1 month, 1 week ago by .
Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts
  • August 9, 2023 at 4:00 am #10442
    Avatar photocandidexmedia

    Hi Publii team!

    I recently learned that OpenAI allows website managers to opt-out of their web crawler, User-agent: GPTBot.

    My feature suggestion would be to allow publii users to customize whether or not they would like their content to be used by Open AI. The specs can be found here: https://platform.openai.com/docs/gptbot

    August 9, 2023 at 10:47 pm #10451
    Avatar photoSar6e

    Agreed.  I came here this morning after opening Publii to see if I could manually apply the required setting to the robots.txt file.  I can’t see that I can in Site Settings…

    August 10, 2023 at 12:50 pm #10462
    Avatar photoBob

    Hey,

    We will add it in the upcoming Publii release.

    --
    Do you appreciate the support you've received today? If so, consider donating to the Publii team by clicking here; we'll be sure to use your donation to make Publii even better!

    August 19, 2023 at 1:05 pm #10507
    Avatar photoTomasz Dziuda

    Hi,

    I can confirm that in the Publii v.0.43 you will be able to block both ChatGPT-User, GPTBot and CommonCrawl bots (separately)

    --
    Do you appreciate the support you've received today? If so, consider donating to the Publii team by clicking here; we'll be sure to use your donation to make Publii even better!

    August 19, 2023 at 4:04 pm #10514
    Avatar photocandidexmedia

    Fantastic! Thank you Tomasz and Bob!

  • Author
    Posts
Viewing 5 posts - 1 through 5 (of 5 total)
  • You must be logged in to reply to this topic.