怎么下载小说 笔趣阁txt下载网站入口
时间:2023-04-28 09:11/span>
作者:tiger
分类:
新知
浏览:8322
评论:0
爬虫脚本把这个小说网上的几乎所有小说都下载到了本地,一共27000+本小说,一共40G。
自此,大功告成,打包发了过去。另外说一句,百度云真坑,每次上传数量有限制,分享给朋友文件数量有限制,分享的文件夹大小也有限制,害的我还得整成压缩版才能分享。
下载界面
源码展示:
coding=utf-8 2 import urllib 3 import urllib2 4 import re 5 import os 6 7 webroot = 'http://www.xuanshu.com' 8 9 for page in range(20,220): 10 print '正在下载第'+str(page)+'页小说' 11 12 url = 'http://www.xuanshu.com/origin/soft/sort02/index_'+str(page)+'.html' 13 headers = {'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6' } 14 try: 15 request = urllib2.Request(url,headers=headers) 16 response = urllib2.urlopen(request,timeout=180) 17 print response.read() 18 except urllib2.URLError, e: 19 if hasattr(e,"code"): 20 print e.code 21 if hasattr(e,"reason"): 22 print e.reason 23 24 html = response.read().decode('utf-8') 25 print html 26 pattern = re.compile(u'<li>.*?<div >.*?target="_blank">(.*?)</a><br />大小:(.*?)<br>.*?</em><br>更新:(.*?)</div>.*?<a href="(.*?)"><img.*?>(.*?)</a>.*?<div >(.*?)</div>',re.S) 27 items = re.findall(pattern,html) 28 print items 29 30 for item in items: 31 try: 32 book_auther = item[0].encode('gbk') 33 book_size = item[1].encode('gbk') 34 book_updatetime = item[2].encode('gbk') 35 book_link = item[3].encode('gbk') 36 book_name = item[4].encode('gbk') 37 book_note = item[5].encode('gbk') 38 39 book_full_link = webroot + book_link 构建书的绝对地址 40 41 请求地址 42 try: 43 request = urllib2.Request(book_full_link,headers=headers) 44 response = urllib2.urlopen(request,timeout=180) 45 except urllib2.URLError, e: 46 if hasattr(e,"code"): 47 print e.code 48 if hasattr(e,"reason"): 49 print e.reason 50 html = response.read().decode('utf-8') 51 print html 52 pattern = re.compile('<a downButton" href=\'(.*?)\'.*?Txt.*?</a>',re.S) 53 down_link = re.findall(pattern,html) 54 print book_name 55 print down_link 56 57 down txt 58 try: 59 request = urllib2.Request(down_link[0].encode('utf-8'),headers=headers) 60 response = urllib2.urlopen(request,timeout=180) 61 except urllib2.URLError, e: 62 if hasattr(e,"code"): 63 print e.code 64 if hasattr(e,"reason"): 65 print e.reason 66 try: 67 fp = open(book_name+'.txt','w') 68 except IOError,e: 69 pattern = re.compile('<strong>.*?>(.*?)<.*?</strong>',re.S) 70 book_name = re.findall(pattern,book_name) 71 fp = open(book_name[0]+'.txt','w') 72 print 'start download' 73 fp.write(response.read()) 74 print 'down finish\n' 75 fp.close() 76 except Exception,e: 77 print '该条目解析出现错误,忽略' 78 print e 79 print '' 80 fp = open('error.log','a') 81 fp.write('page:'+str(page)+'\n') 82 fp.write(item[4].encode('gbk')) 83 fp.write(e) 84 fp.write('\nThere is an error in parsing process.\n\n') 85 fp.close()
ps:在学习python过程有不懂的,需要python教程,电子书籍,开发工具等,可以关注我搜索号,私信我”python资料”几个字 免费领取,24小时在线!