Download
We're evolving to serve you better! This current forum has transitioned to read-only mode. For new discussions, support, and engagement, we've moved to GitHub Discussions.

robots.txt

  • #8701
    Avatar photo[anonymous]

    is there a way to edit the robots.txt file before uploading to production?

    #8709
    Avatar photoBob

    Publii automatically generates a robot.txt file. The best way to edit this file is “override” it by creating a new robot.txt file in the root directory using the File Manager. Publii, when encounters this file, will not generate its own robots.txt file.

    #8717
    Avatar photo[anonymous]
    [anonymous] wrote:

    Publii automatically generates a robot.txt file. The best way to edit this file is “override” it by creating a new robot.txt file in the root directory using the File Manager. Publii, when encounters this file, will not generate its own robots.txt file.

    thanks

    #9932
    Avatar photo[anonymous]

    Hello,

    It seems this doesn’t work on amazon S3 host. When publii uploads, it overrides my robots.txt. This seems to be a bug.

    #9934
    Avatar photo[anonymous]

    Nevermind, I checked some more and it works.

    #9936
    Avatar photo[anonymous]

    Sometimes the whole website is synced, that’s when robots.txt is override even if I have the file there.

    #10082
    Avatar photo[anonymous]

    @lich2king – that’s very weird, because Publii definitely checks for robots.txt existence: https://github.com/GetPublii/Publii/blob/fde8b48eefb409aae9a0d0f4270653b51c41521c/app/back-end/modules/render-html/renderer.js#L1811-L1813

    I supposed that we can have some race condition here, but as I see operation of copying root files is fully synchronous:
    https://github.com/GetPublii/Publii/blob/fde8b48eefb409aae9a0d0f4270653b51c41521c/app/back-end/modules/render-html/helpers/files.js#L21-L33

    So currently I have no idea why your override for robots.txt file is not working :/

    #10086
    Avatar photo[anonymous]

    It happened only when publii synced the whole website (all files to be uploaded). I don’t know what is the case for a full resync of the files but that’s when the robots.txt is also replaced.

    This didn’t happen again since may.

    checking this code: https://github.com/GetPublii/Publii/blob/fde8b48eefb409aae9a0d0f4270653b51c41521c/app/back-end/modules/render-html/renderer.js#L1811-L1813

    If publii uploads the robots.txt, this means it won’t be able to update it again with new values as the robots.txt file already exists.

    #10087
    Avatar photo[anonymous]

    But form of sync has nothing to do here in my opinion – Publii during generating output files should always use your overrided robots.txt file.

    Full sync can caused by global website changes (in footer/header) or due problem with checking publii.files.json file which is used to check which files should be synced.