Too Few Pages Indexed -- FreeFind.com

Free search engine

home
features
pricing
sign-up

FAQ
library
search
site map
policies
contact

Too Few Pages Indexed

Library > Reference > FAQ > Too Few

If the search engine did not locate as many pages as you expected look here for the most common reasons why.

You have not enabled PDF indexing.

You are not using the actual address of your site.

Your hosting company changed the actual location of your site.

Your site has an advertising frameset.

Your are using a nofollow robots meta tag.

There are parts of your site which cannot be found by following links.

Your site uses applets for links.

Your site has links exclusively in flash content.

The only links to some pages are built using javascript.

The only links to some pages are in included JS files.

You have links embedded in some type of browser plugin.

Your site is not fully contained in the starting directory.

Your site is not fully contained within the starting domain.

You are trying to index a type of file that is not supported.

There was internet congestion when you last indexed your site.

You are not consistently using (or not using) the "www".

Problem: You have not enabled PDF indexing.
Solution: If your PDF files are not being indexed be sure that PDF indexing is enabled. To do this simply log in, go to the page and then use the PDF indexing link. [top]

Problem: You are not using the actual address of your site.
Solution: It is fairly common for sites to use a redirector to "give the site a more memorable name". If your site is like this, or if it uses a redirector for some other reason, then your account needs to specify the actual location of your site and not the location of the redirector.

For most sites, you can determine the actual location of your site by waiting for your home page to finish displaying, then using the address as shown at the top of your browser's window.

Some "name" services don't redirect but actually place your entire site within a frame so that the original name is still displayed in the browser's address field. For these types of arrangements you still need to use the actual location of your site and not the location of the "name".

Note that you do not need to sign up again to change your site's address! Simply log in, go to the page and then use the change URL link.
[top]

Problem: Your hosting company changed the actual location of your site.
Solution: This happens more frequently than you might guess (and they usually do it without telling you!), so double-check that your web site address is still the same.

For most sites, you can determine the actual location of your site by waiting for your home page to finish displaying, then using the address as shown at the top of your browser's window.

Note that you do not need to sign up again to change your site's address! Simply log in, go to the page and then use the change URL link.
[top]

Problem: Your site has an advertising frameset.
Solution: It is common for hosting companies which provide free web space to place your site within an "advertising frameset". Sites like these have only one page at your home address - the advertising frameset. All your pages are located someplace else.

To see if this is being done, you can browse to your online home page and take a look at the HTML source using your browser. Look for the frameset tag and, if found, look for a frame tag indicating the actual location of your site.

For example if your site is at:
   http://hometown.aol.com/somesite/
when you look at the HTML you might find something like:
   <frame src="http://members.aol.com/_ht_a/somesite/" name=contentFrame>
This indicates that your pages are actually at:
   http://members.aol.com/_ht_a/somesite/

Once you determine where your site is actually located you can specify this location as an additional starting point for the spider. To do this simply log in, go to the page and then use the set starting points link.
[top]

Problem: Your are using a nofollow robots meta tag.
Solution: The search engine spider honors your "robots nofollow" meta tags. If your site has one or more pages which contain the tag:
<meta name=robots content="nofollow">
then the links on those pages will be ignored, possibly resulting in incomplete indexing of your site.

There a couple ways of fixing this:
1. You can override a particular robots meta tag by adding the FreeFind version of the tag after the robots meta tag. The FreeFind version has the same possible content values, but has the name "FreeFind" instead of "robots". Use something like:
<meta name=FreeFind content="all">

2. You can make FreeFind ignore all robots meta-tags on all pages by adding the following tag on the first page the FreeFind spiders reads:
<meta name=FreeFind content="noRobotsTag">
It should be added before any robots meta-tags are encountered.
[top]

Problem: There are parts of your site which cannot be found by following links.
Solution: The spider (indexer) locates the pages of your site by following all the links it can find on your site which lead to other pages in your site. It does not just scan your server's hard disk!

If there are parts of your site which don't have links to them but need to be indexed, you must let the spider know about them. There are two ways of doing this:
1. You can use the FreeFind Links tag to provide the spider with links to follow but which are not displayed. This tag needs to be placed in a page that is already found, so the spider can find and processes it.
2. You can provide one or more additional starting points for the spider. To do this simply log in, go to the page and then use the set starting points link. Note that each starting point is treated as if it were a separate site, not a single page.
After letting the spider know where to start looking for the missing areas of your site, you will need to re-spider your site to see the results. To do this simply log in, go to the page and use the index now link.
[top]

Problem: Your site uses applets for links.
Solution: The spider (indexer) locates the pages of your site by following all the links it can find on your site which lead to other pages in your site. It does not just scan your server's hard disk!

Links embedded in applets (or expressed in applet parameters) cannot be followed by our spider (nor any other one we're aware of). This means that if all your links to a page are applet links, that page will not be found.

There are two ways around this problem:
1. The recommended way is to use the FreeFind Links tag to provide the spider with links to follow but which are not displayed. This tag should be placed in the page(s) with the applet links.
2. A less optimal way is to provide one or more additional starting points for the spider. To do this simply log in, go to the page and then use the set starting points link. Although each starting point is treated as if it were a separate site and not a single page, if your site uses applet links exclusively you will end up needing to provide a comprehensive list of all your pages!
After providing "spider-friendly" links, you will need to re-spider your site to see the results. To do this simply log in, go to the page and use the index now link.
[top]

Problem: Your site has links exclusively in flash content.
Solution: The spider (indexer) locates the pages of your site by following all the links it can find on your site which lead to other pages in your site. It does not just scan your server's hard disk!

Links embedded in flash content cannot be followed by our spider. This means that if all your links to a page are flash links, that page will not be found.

There are two ways around this problem:
1. The recommended way is to use the FreeFind Links tag to provide the spider with links to follow but which are not displayed. This tag should be placed in the page(s) with the flash links.
2. A less optimal way is to provide one or more additional starting points for the spider. To do this simply log in, go to the page and then use the set starting points link. Although each starting point is treated as if it were a separate site and not a single page, if your site uses flash links exclusively you will end up needing to provide a comprehensive list of all your pages!
After providing "spider-friendly" links, you will need to re-spider your site to see the results. To do this simply log in, go to the page and use the index now link.
[top]

Problem: The only links to some pages are built using javascript.
Solution: The spider (indexer) locates the pages of your site by following all the links it can find on your site which lead to other pages in your site. It does not just scan your server's hard disk!

Although FreeFind is unusual in that it tries to extract links from javascript, if your links are built up out of separate pieces they will not be found.

There are two ways around this problem:
1. The recommended way is to use the FreeFind Links tag to provide the spider with links to follow but which are not displayed. This tag should be placed in the page(s) with the built-up javascript links.
2. A less optimal way is to provide one or more additional starting points for the spider. To do this simply log in, go to the page and then use the set starting points link. Note that each starting point is treated as if it were a separate site, not a single page.
After providing "spider-friendly" links, you will need to re-spider your site to see the results. To do this simply log in, go to the page and use the index now link.
[top]

Problem: The only links to some pages are in included JS files.
Solution: The spider (indexer) never reads JS files, so links your included javascript will not be found. There are two ways around this problem:
1. The recommended way is to use the FreeFind Links tag to provide the spider with links to follow but which are not displayed. This tag should be placed in the page(s) with the built-up javascript links.
2. A less optimal way is to provide one or more additional starting points for the spider. To do this simply log in, go to the page and then use the set starting points link. Note that each starting point is treated as if it were a separate site, not a single page.
After providing "spider-friendly" links, you will need to re-spider your site to see the results. To do this simply log in, go to the page and use the index now link.
[top]

Problem: You have links embedded in some type of browser plugin.
Solution: The spider (indexer) locates the pages of your site by following all the links it can find on your site which lead to other pages in your site. It does not just scan your server's hard disk!

Links embedded in plugin content cannot be followed by our spider. This means that if all your links to a page are plugin links, that page will not be found.

There are two ways around this problem:
1. The recommended way is to use the FreeFind Links tag to provide the spider with links to follow but which are not displayed. This tag should be placed in the page(s) with the plugin links.
2. A less optimal way is to provide one or more additional starting points for the spider. To do this simply log in, go to the page and then use the set starting points link. Although each starting point is treated as if it were a separate site and not a single page, if your site uses plugin links exclusively you will end up needing to provide a comprehensive list of all your pages!
After providing "spider-friendly" links, you will need to re-spider your site to see the results. To do this simply log in, go to the page and use the index now link.
[top]

Problem: Your site is not fully contained in the starting directory.
Solution: By default, the spider constrains itself to the directory tree that it starts in. For example, if it starts at your home page:
   http://members.someisp.com/~imauser/index.html
the it will follow links to here:
   http://members.someisp.com/~imauser/archive/...
but not to here:
   http://members.someisp.com/~somebodyelse/...

Typically this is exactly what's desired. In some cases, however, you may want to the spider to jump to some other directory tree. For example, if your home page is actually down a directory:
   http://members.someisp.com/~imauser/home/index.html
and you want to follow links to here:
   http://members.someisp.com/~imauser/archive/...
then you will need to let the spider know that it's OK to jump to the archive directory. You do this by providing an additional starting point for the spider that is within the archive directory, i.e.:
   http://members.someisp.com/~imauser/archive/index.html

To set additional starting points simply log in, go to the page and then use the set starting points link. Note that each starting point is treated as if it were a separate site and not a single page. After specifying additional starting points, you will need to re-spider your site to see the results. To do this use the index now link.
[top]

Problem: Your site is not fully contained within the starting domain.
Solution: By default, the spider constrains itself to the domain that it starts in. For example, if it starts at your home page:
   http://myfirstsite.com/index.html
the it will follow links to here:
   http://myfirstsite.com/archive/...
but not to here:
   http://mysecondsite.com/...

Typically this is exactly what's desired (if it didn't do this, it would end up indexing the entire web!). In some cases, however, you may want to the spider to jump to some other domain. An example might be a site:
   http://myfirstsite.com/index.html
which has a separate domain for the user support area:
   http://support.myfirstsite.com/index.html
In cases like these you will need to let the spider know that it's OK to jump to the other domain. You do this by providing an additional starting point for the spider, like:
   http://support.myfirstsite.com/index.html

To set additional starting points simply log in, go to the page and then use the set starting points link. Note that each starting point is treated as if it were a separate site and not a single page. After specifying additional starting points, you will need to re-spider your site to see the results. To do this use the index now link.
[top]

Problem: You are trying to index a type of file that is not supported.
Solution: Currently FreeFind indexes HTML and text files (regardless of file extension), and PDF files if you have subscribed to the ad-free version. Other file types (DOC, etc) are not yet supported.
[top]

Problem: There was internet congestion when you last indexed your site.
Solution: Yes, it happens. Sometimes the internet has a traffic jam! Although the search engine spider (indexer) will try repeatedly, sometime it just cannot get through to your server. The solution is simple: just respider your site.

Before you do it is often a good idea to make sure that your site is actually online. Go online to your home page and "refresh" the page to make sure that everything seems to be working.

Your site is up? Then log in, go to the page and then use the index now link to respider your site.
[top]

Problem: You are not consistently using (or not using) the "www".
Solution: Some folks have "www" links and "non-www" links throughout their site as if they are the same. They are not. Although you may have both your "www" domain name and your non-www domain name pointed to the same site, the search engine does not know this and so by default will not follow a link from one to the other.

You may try letting the spider know that it's OK to follow links to either domain name by adding the other domain name to the list of additional starting points. Sometimes this works, but often it results in spidering twice as many pages as expected.

If this is happening to you, the fix is to standardize on one or the other; either use "www" in all your site's links or don't. Do not mix and match. As a bonus, fixing this problem may improve how well your site is indexed by the big web search engines.
[top]

home | search engine sign-up | features | pricing | search | site map

control center login | FAQ | library | policies | advertising | contact | link to us

Questions? See our search engine FAQ or check out our library. Need to get in touch with us? See our contact info.
Before you use FreeFind or add FreeFind to your site, read and agree to the site policy.
Visit our other sites Findia Net Search and Change Detection and our partners
Search engine and site navigation technology by FreeFind.com

FreeFind and FreeFind.com are trademarks of FreeFind.com.
Copyright 1998 - 2004