Spiders Spiders Spiders

Mr-Ice-Man · 10-15-2007, 09:12 AM

Can anyone tell me please what i need to do to gt spiders coming to my site please.
Ive heard on some forums people say about makeing a file called robots.txt but what should i put in it?

Hope someone can help.
Thankyou.

10-15-2007, 11:49 AM

robots.txt is for spiders not to crawl in certain locations. to get them their in the first place, you need to have content

10-15-2007, 06:49 PM

Quote:

Originally Posted by whitemike

to get them their in the first place, you need to have content

Not necessarily. They will still go to the site without content, they just won't have anything to catalog

Mr-Ice-Man, the bots will go to your site automatically in around a month or two from when you first registered it, unless it's on a ban list. I'd suggest you add that content that Mike was talking about or it will hurt you. If they don't see fresh content they will stay away longer, defeating what you're trying to accomplish

night2dark2 · 10-23-2007, 08:35 AM

hmm yes that is quite intersting question. Spiders crawl my site for at least 42-90 hours a month. With iuno abot 20 Spiders. Same for yahoo and all other search engines.

10-23-2007, 03:45 PM

Add back links to your site, and quality content.

BlazingPC · 10-23-2007, 04:19 PM

Submit your URL(s) to all the major search engines and sign up for Googles webmasters tools and create a site map.
This will also tell you how to create a robot.txt file, Then its a matter of getting your links out there on other sites to help the bots find your site from other places on the net.

10-24-2007, 03:28 PM

Yes, don't forget about that site map. This is a great tool left out by many webmasters.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
eXtend Show Spiders	Tim	vBulletin 3.5 / 3.6 Hack Releases	19	11-11-2007 04:08 PM
Spiders have same rights as members.	deadeye	vBulletin Modification Requests	0	12-07-2006 08:09 AM
Spiders as members	Vaz	General vBulletin Support	6	07-18-2006 02:29 PM
Do spiders love archives?	Rex	Growing Your vB Forum	7	06-03-2006 03:48 PM

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)