For webmasters, a links.txt file often serves as the output of a crawl.
: You can use grep or wget with a spider option to crawl a website and save every discovered URL into a text file for auditing. Links.txt
Developers frequently use links.txt as a simple data source for scripts. For webmasters, a links
