refers to the search engine spiders crawling out of the link to the URL analysis from the known on the page, and then along the link found new page (that is the link to the URL process). Of course, spiders are not found new URL immediately climb to grab new pages, but to find the URL store to grab spider address database, according to a certain order from the address database extraction URL to crawl.
is the search engine spiders crawl extraction to catch the URL address from a library to be arrested, visit the URL, read the HTML code stored in the database. The spider is like a browser to open the page, and the user browser, will also have a record of the original server log.
crawling is what?
spiders crawl What is the What is the
is a reader of this post what message in quota:
message that the reader does not understand what is what is the crawl, index, and what is the significance of index and Noindex tags. The Noindex tag is not allowed to crawl the page, the page index is not allowed, the two are different, have different functions.
no, the index label refers to tell the spider can crawl the page, then Noindex is not allowed to crawl the page?! So why the last few articles in "Noindex tags cannot save grab share. The search engine to know Noindex tags on the page, you must first grab this page, so do not save to grab share."
index refers to a URL information, which is stored in the database, index library, user search, search engine URL information extraction and sorting out from the index in the library. The English index is index. The index is used to search, so the indexed U>
see Shanghai dragon related blogs and forums can feel a lot of Shanghai dragon did not understand, crawling crawling, indexing, included what is these concepts really means, what is the difference between where, Noindex, nofollow, robots file function. No precise understanding of these concepts, large website structure, decide what pages should be crawled, what need to be indexed, which pages need to prohibit crawling, indexing and so on, it is difficult to know what to do. A lot of messages even like grasping the post quota said, mentioned in the handling of these, simply cannot read what to say.
is so basic and important, and relatively easy to confuse the concept of Shanghai dragon, I thought before in the blog wrote, read the message, read previous posts that had previously not written. Shanghai dragon book is written in the actual password, but every one of Shanghai dragon is not written. Today make up.