Why Do You Need a Robots.txt File for WordPress Website?
Even without a robots.txt file, your site will be crawled and indexed. However, you cannot restrict which content search engines can index. If you are just starting off with your blog and don't yet have a sizable amount of content, this won't make a huge difference. However, as your website expands and more content is added, you'll need to have greater control over how it's crawled and indexed.
To prevent irrelevant pages like admin pages, plugins, themes, or posts from being indexed by search engines, you can use the robots.txt file. It's not the most secure method, but it will get the job done and keep your private data from the public and search engines.