babyspiderΒΆ
- babyspider package
- Module contents
- Submodules
- babyspider.crawler module
CrawlerCrawler.base_output_dirCrawler.parserCrawler.crawl_delayCrawler.use_sitemapsCrawler.max_redirectsCrawler.http_timeoutCrawler.chunk_sizeCrawler.index_filenameCrawler.check_parser()Crawler.chunk_sizeCrawler.crawl()Crawler.domain_from_url()Crawler.download_resource()Crawler.download_sitemap_urls()Crawler.get_canonical_url()Crawler.get_local_path()Crawler.get_new_domain_urls()Crawler.get_urls_from_sitemap()Crawler.http_timeoutCrawler.index_filenameCrawler.max_redirectsCrawler.read_robots_txt()Crawler.strip_url()