4xx errors often point to a problem on a website. For example, if you have a broken link on a page, and visitors click it, they may see a 4xx error. It's important to regularly monitor and fix these errors, because they may have negative impact and lower your site's authority in users' eyes.

5xx error messages are sent when the server has a problem or error. It's important to regularly monitor these errors and investigate their causes, because they may have negative impact and lower the site's authority in search engines' eyes.

A custom 404 error page can help you keep users on the website. In a perfect world, it should inform users that the page they are looking for doesn't exist, and feature such elements as your HTML sitemap, the navigation bar and a search field. But more importantly, a 404 error page should return the 404 response code. This may sound obvious, but unfortunately it's rarely so.

According to Google Search Console:

"Returning a code other than 404 or 410 for a non-existent page... can be problematic. Firstly, it tells search engines that there's a real page at that URL. As a result, that URL may be crawled and its content indexed. Because of the time Googlebot spends on non-existent pages, your unique URLs may not be discovered as quickly or visited as frequently and your site's crawl coverage may be impacted. We recommend that you always return a 404 (Not found) or a 410 (Gone) response code in response to a request for a non-existing page."

The robots.txt file is automatically crawled by robots when they arrive at your website. This file should contain commands for robots, such as which pages should or should not be indexed. If you want to disallow indexing of some content (for example, pages with private or duplicate content), just use an appropriate rule in the robots.txt file. For more information on such rules, check out http://www.robotstxt.org/robotstxt.html.

Please note that commands placed in the robots.txt file are more like suggestions rather than absolute rules for robots to follow. There's no guarantee that some robot will not check the content that you have disallowed.

An XML sitemap should contain all of the website pages that you want to be indexed, and should be located on the website one directory structure away from the homepage (ex. http://www.site.com/sitemap.xml). In general, it serves to aid indexing. You should update it each time you add new pages to your website. Besides, the sitemap should follow particular syntax.

The sitemap allows you to set the priority of each page, telling search engines which pages they are supposed to crawl more often (i.e. they are more frequently updated). Learn how to create an .xml sitemap at http://www.sitemaps.org/.