Top 10 Tips for better indexing by crawlers:

1) Unique and relevant content in the site and update content on regular basis…updating content daily or weekly is considered fine.

2) Get backlinks from high PR sites or which is >= “your site current PR”. The link should be relevant to your site theme, never try to link from so many websites that is irrelevant to your site theme. Minimum links from high PR sites are much beneficial then many backlinks from low PR and irrelevant sites.

3) Avoid any broken link or “Page not found 404”error in the site. There should be no any broken link in the site and all the links should have related destination page. Do not create under construction pages.

4) Avoid temporary redirect until highly required. Instead use permanent redirect in the site for all pages that is indexed by search engine but do not exist now. Redirect such urls to other most relevant page or request search engine to remove it from indexed database.

5) Easy navigation & site structure, user convenient is a must. You site should have easy navigation, so it is helpful to visitors. You visitors should be able to access every related page from any page or section in the site.

6) Do Google/Yahoo sitemap submissions with daily/weekly/Monthly changing frequency according how frequently changes are done; MSN is still need to implement sitemap submissions utility.

7) You may also include “Sitemap: ” in the robots.txt file of your site. The should be the complete URL to the Sitemap, such as: http://www.example.com/sitemap.xml. This directive is independent of the user-agent line, so it doesn't matter where you place it in your file. If you have a Sitemap index file, you can include the location of just that file. You don't need to list each individual Sitemap listed in the index file.

8) Page download time and size also matters a lot. Heavy size pages or that takes long time to appear should be avoided. A light page comprising everything required for visitors is always preferred. This also has great impact in the company sales

9) Create keywords rich Title and Meta tags, but avoid any kind of stuffing. Optimize all pages depending on the main theme of that page. Do no develop duplicate pages in the site. Theme based site development can be a basis. Develop theme of every page with unique and relevant content.

10) Do strong off page submissions using to create backlinks for the site and increase Internet presence. You can do directory submissions, article submissions, search engine submissions (although it is not required), product directory submission (for products site basically), press release submission and many more to do the off page work for a site.

Think out-side the box too. Do not use traditional optimization ways. Search engine world is changing everyday, so update buddy :)