David Naylor Launches Robots.txt File Builder
I was actually outside the Hilton discussing this project less than two weeks ago with David Naylor and he has already delivered the first part of his idea.
Many people screw up their robots.txt file and deny the search engine spiders access to their sites. Dave thought it would be a great idea to create a central site that acts autonomously where people can have their robots.txt file created and stored to insure good interaction with the spiders.
His initial offering allows people to create the file and then copy and paste it into a page they can upload to their own site. Eventually Dave wants to host the pages himself and make sure the spiders correctly spider them. The site would be the central location for all spiders to get the right written file for any website.
The subtle differences between the spiders can be adapted, but Dave felt it would also be a way to get uniformity from the engines once they saw people using the site in sizable numbers.
The other ancilary benefits he was discussing was the ability to determine load times for a given site and get the spiders to visit at low traffic times so as to not overload the client’s site capabilities.
I am impressed how quickly he got started on this. But then again he did share it at SES NYC with other fast to market players…. maybe he correctly guessed he better move on it quick before someone else did.
Great job so far Dave… now don’t forget the rest!
More about:
The Merkle B2B 2023 Superpowers Index outlines what drives competitive advantage within the business culture and subcultures that are critical to success. It is the indispensable guide for B2B marketers to deliver world-class experiences and keep pace with the dynamic environment. Download Now
The ClicData survey found that various challenges exist that prevent organizations from achieving such gains. These challenges included inaccessible data formats and limited flexibility in displaying data in dashboards. Download Now
The need for fraud prevention in the digital world is critical now more than ever. Why? Thinking about your own behavior, consider how you complete transactions and how this has changed over the last 5 years. Download Now
The need for fraud prevention in the digital world is critical now more than ever. Why? Thinking about your own behavior, consider how you complete transactions and how this has changed over the last 5 years. Download Now