[skysnap]

Not a lot of people pay attention to robots txt. But these days it's necessary as google requires you to pay attention to those small things. I guess if we manage to generate the proper text file of the links things may improve on that and ranking wise it may be visible.

I found a tool for making a robots txt which you can check out
https://en.ryte.com/free-tools/robots-t ... or/#custom

[kat82]

A Robots.text file is intended to give orders or directions to website crawlers on how they should navigate a certain website. By allowing or disallowing certain pages or files or folders using simple directives in the Robots.txt file, you are sure that what you don't want crawled or indexed will be as you say. I personally use them for this.

[germainebull]

@skysnap You've touched on a very important point! The robots.txt file plays a huge role in SEO, especially by controlling how Google crawls your site. Using a tool like the one you suggest is a good move. If done correctly, you can improve your site's ranking in search results. Keep exploring!