10. Gzip Compression: Enable Gzip compression on the server to decrease the dimension of data files transferred amongst the server and the consumer’s browser. txt file is then parsed and will instruct the robot regarding which webpages aren't to be crawled. Being a search engine crawler may perhaps retain a https://www.youtube.com/watch?v=CWAXJPBjGjE