⊟
132 KB
/srv/reproducible-results/rbuild-debian/r-b-build.xW3jcoMw/b1/python-scrapy_2.4.1-2+deb11u1_amd64.changes vs.
/srv/reproducible-results/rbuild-debian/r-b-build.xW3jcoMw/b2/python-scrapy_2.4.1-2+deb11u1_amd64.changes
⊟
319 B
Files
    
Offset 1, 3 lines modifiedOffset 1, 3 lines modified
  
1 ·6e1752b11018c80a1c1dfd6727bcbc60·709268·doc·optional·python-scrapy-doc_2.4.1-2+deb11u1_all.deb1 ·2d3c29c6404180e12190e993f37f243c·709288·doc·optional·python-scrapy-doc_2.4.1-2+deb11u1_all.deb
2 ·39e5bfdc1c8b4d72e0b68b1fa3ea3d77·214112·python·optional·python3-scrapy_2.4.1-2+deb11u1_all.deb2 ·39e5bfdc1c8b4d72e0b68b1fa3ea3d77·214112·python·optional·python3-scrapy_2.4.1-2+deb11u1_all.deb
⊟
131 KB
python-scrapy-doc_2.4.1-2+deb11u1_all.deb
⊟
452 B
file list
    
Offset 1, 3 lines modifiedOffset 1, 3 lines modified
1 -rw-r--r--···0········0········0········4·2022-05-20·20:11:00.000000·debian-binary1 -rw-r--r--···0········0········0········4·2022-05-20·20:11:00.000000·debian-binary
2 -rw-r--r--···0········0········0·····4448·2022-05-20·20:11:00.000000·control.tar.xz2 -rw-r--r--···0········0········0·····4444·2022-05-20·20:11:00.000000·control.tar.xz
3 -rw-r--r--···0········0········0···704628·2022-05-20·20:11:00.000000·data.tar.xz3 -rw-r--r--···0········0········0···704652·2022-05-20·20:11:00.000000·data.tar.xz
⊟
98.0 B
control.tar.xz
⊟
70.0 B
control.tar
⊟
48.0 B
./md5sums
⊟
30.0 B
./md5sums
Files differ
⊟
131 KB
data.tar.xz
⊟
131 KB
data.tar
⊟
983 B
./usr/share/doc/python-scrapy-doc/html/README.html
    
Offset 274, 15 lines modifiedOffset 274, 15 lines modified
274 ··········</div>274 ··········</div>
275 ··········<footer>275 ··········<footer>
  
276 ··<hr/>276 ··<hr/>
  
277 ··<div·role="contentinfo">277 ··<div·role="contentinfo">
278 ····<p>278 ····<p>
279 ········&#169;·Copyright·2008–2024,·Scrapy·developers.279 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
280 ······<span·class="lastupdated">280 ······<span·class="lastupdated">
281 ········Last·updated·on·May·20,·2022.281 ········Last·updated·on·May·20,·2022.
282 ······</span>282 ······</span>
  
283 ····</p>283 ····</p>
284 ··</div>284 ··</div>
285 ····285 ····
⊟
545 B
html2text {}
    
Offset 90, 9 lines modifiedOffset 90, 9 lines modified
90 *****·Alternative·method·using·tox¶·*****90 *****·Alternative·method·using·tox¶·*****
91 To·compile·the·documentation·to·HTML·run·the·following·command:91 To·compile·the·documentation·to·HTML·run·the·following·command:
92 tox·-e·docs92 tox·-e·docs
93 Documentation·will·be·generated·(in·HTML·format)·inside·the·.tox/docs/tmp/html93 Documentation·will·be·generated·(in·HTML·format)·inside·the·.tox/docs/tmp/html
94 dir.94 dir.
  
95 ===============================================================================95 ===============================================================================
96 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.96 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
97 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.97 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
929 B
./usr/share/doc/python-scrapy-doc/html/_modules/index.html
    
Offset 284, 15 lines modifiedOffset 284, 15 lines modified
284 ··········</div>284 ··········</div>
285 ··········<footer>285 ··········<footer>
  
286 ··<hr/>286 ··<hr/>
  
287 ··<div·role="contentinfo">287 ··<div·role="contentinfo">
288 ····<p>288 ····<p>
289 ········&#169;·Copyright·2008–2024,·Scrapy·developers.289 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
290 ······<span·class="lastupdated">290 ······<span·class="lastupdated">
291 ········Last·updated·on·May·20,·2022.291 ········Last·updated·on·May·20,·2022.
292 ······</span>292 ······</span>
  
293 ····</p>293 ····</p>
294 ··</div>294 ··</div>
295 ····295 ····
⊟
475 B
html2text {}
    
Offset 128, 9 lines modifiedOffset 128, 9 lines modified
128 ····*·scrapy.statscollectors128 ····*·scrapy.statscollectors
129 ····*·scrapy.utils.curl129 ····*·scrapy.utils.curl
130 ····*·scrapy.utils.log130 ····*·scrapy.utils.log
131 ····*·scrapy.utils.reactor131 ····*·scrapy.utils.reactor
132 ····*·scrapy.utils.trackref132 ····*·scrapy.utils.trackref
  
133 ===============================================================================133 ===============================================================================
134 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.134 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
135 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.135 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
947 B
./usr/share/doc/python-scrapy-doc/html/_modules/itemadapter/adapter.html
    
Offset 483, 15 lines modifiedOffset 483, 15 lines modified
483 ··········</div>483 ··········</div>
484 ··········<footer>484 ··········<footer>
  
485 ··<hr/>485 ··<hr/>
  
486 ··<div·role="contentinfo">486 ··<div·role="contentinfo">
487 ····<p>487 ····<p>
488 ········&#169;·Copyright·2008–2024,·Scrapy·developers.488 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
489 ······<span·class="lastupdated">489 ······<span·class="lastupdated">
490 ········Last·updated·on·May·20,·2022.490 ········Last·updated·on·May·20,·2022.
491 ······</span>491 ······</span>
  
492 ····</p>492 ····</p>
493 ··</div>493 ··</div>
494 ····494 ····
⊟
465 B
html2text {}
    
Offset 346, 9 lines modifiedOffset 346, 9 lines modified
346 ········return·obj.asdict()346 ········return·obj.asdict()
347 ····elif·is_item(obj):347 ····elif·is_item(obj):
348 ········return·ItemAdapter(obj).asdict()348 ········return·ItemAdapter(obj).asdict()
349 ····else:349 ····else:
350 ········return·obj350 ········return·obj
  
351 ===============================================================================351 ===============================================================================
352 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.352 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
353 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.353 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1010 B
./usr/share/doc/python-scrapy-doc/html/_modules/itemadapter/utils.html
    
Offset 333, 15 lines modifiedOffset 333, 15 lines modified
333 ··········</div>333 ··········</div>
334 ··········<footer>334 ··········<footer>
  
335 ··<hr/>335 ··<hr/>
  
336 ··<div·role="contentinfo">336 ··<div·role="contentinfo">
337 ····<p>337 ····<p>
338 ········&#169;·Copyright·2008–2024,·Scrapy·developers.338 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
339 ······<span·class="lastupdated">339 ······<span·class="lastupdated">
340 ········Last·updated·on·May·20,·2022.340 ········Last·updated·on·May·20,·2022.
341 ······</span>341 ······</span>
  
342 ····</p>342 ····</p>
343 ··</div>343 ··</div>
344 ····344 ····
⊟
536 B
html2text {}
    
Offset 188, 9 lines modifiedOffset 188, 9 lines modified
188 (item_class.__name__,·field_name))188 (item_class.__name__,·field_name))
189 ····elif·issubclass(item_class,·dict):189 ····elif·issubclass(item_class,·dict):
190 ········return·MappingProxyType({})190 ········return·MappingProxyType({})
191 ····else:191 ····else:
192 ········raise·TypeError("%s·is·not·a·valid·item·class"·%·(item_class,))192 ········raise·TypeError("%s·is·not·a·valid·item·class"·%·(item_class,))
  
193 ===============================================================================193 ===============================================================================
194 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.194 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
195 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.195 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
982 B
./usr/share/doc/python-scrapy-doc/html/_modules/itemloaders/processors.html
    
Offset 454, 15 lines modifiedOffset 454, 15 lines modified
454 ··········</div>454 ··········</div>
455 ··········<footer>455 ··········<footer>
  
456 ··<hr/>456 ··<hr/>
  
457 ··<div·role="contentinfo">457 ··<div·role="contentinfo">
458 ····<p>458 ····<p>
459 ········&#169;·Copyright·2008–2024,·Scrapy·developers.459 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
460 ······<span·class="lastupdated">460 ······<span·class="lastupdated">
461 ········Last·updated·on·May·20,·2022.461 ········Last·updated·on·May·20,·2022.
462 ······</span>462 ······</span>
  
463 ····</p>463 ····</p>
464 ··</div>464 ··</div>
465 ····465 ····
⊟
494 B
html2text {}
    
Offset 307, 9 lines modifiedOffset 307, 9 lines modified
307 ····def·__init__(self,·separator='·'):307 ····def·__init__(self,·separator='·'):
308 ········self.separator·=·separator308 ········self.separator·=·separator
  
309 ····def·__call__(self,·values):309 ····def·__call__(self,·values):
310 ········return·self.separator.join(values)310 ········return·self.separator.join(values)
  
311 ===============================================================================311 ===============================================================================
312 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.312 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
313 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.313 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
996 B
./usr/share/doc/python-scrapy-doc/html/_modules/itemloaders.html
    
Offset 661, 15 lines modifiedOffset 661, 15 lines modified
661 ··········</div>661 ··········</div>
662 ··········<footer>662 ··········<footer>
  
663 ··<hr/>663 ··<hr/>
  
664 ··<div·role="contentinfo">664 ··<div·role="contentinfo">
665 ····<p>665 ····<p>
666 ········&#169;·Copyright·2008–2024,·Scrapy·developers.666 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
667 ······<span·class="lastupdated">667 ······<span·class="lastupdated">
668 ········Last·updated·on·May·20,·2022.668 ········Last·updated·on·May·20,·2022.
669 ······</span>669 ······</span>
  
670 ····</p>670 ····</p>
671 ··</div>671 ··</div>
672 ····672 ····
⊟
530 B
html2text {}
    
Offset 533, 9 lines modifiedOffset 533, 9 lines modified
  
533 ····def·_get_cssvalues(self,·csss,·**kw):533 ····def·_get_cssvalues(self,·csss,·**kw):
534 ········self._check_selector_method()534 ········self._check_selector_method()
535 ········csss·=·arg_to_iter(csss)535 ········csss·=·arg_to_iter(csss)
536 ········return·flatten(self.selector.css(css).getall()·for·css·in·csss)536 ········return·flatten(self.selector.css(css).getall()·for·css·in·csss)
  
537 ===============================================================================537 ===============================================================================
538 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.538 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
539 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.539 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
996 B
./usr/share/doc/python-scrapy-doc/html/_modules/parsel/selector.html
    
Offset 624, 15 lines modifiedOffset 624, 15 lines modified
624 ··········</div>624 ··········</div>
625 ··········<footer>625 ··········<footer>
  
626 ··<hr/>626 ··<hr/>
  
627 ··<div·role="contentinfo">627 ··<div·role="contentinfo">
628 ····<p>628 ····<p>
629 ········&#169;·Copyright·2008–2024,·Scrapy·developers.629 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
630 ······<span·class="lastupdated">630 ······<span·class="lastupdated">
631 ········Last·updated·on·May·20,·2022.631 ········Last·updated·on·May·20,·2022.
632 ······</span>632 ······</span>
  
633 ····</p>633 ····</p>
634 ··</div>634 ··</div>
635 ····635 ····
⊟
522 B
html2text {}
    
Offset 498, 9 lines modifiedOffset 498, 9 lines modified
498 ····def·__str__(self):498 ····def·__str__(self):
499 ········data·=·repr(shorten(self.get(),·width=40))499 ········data·=·repr(shorten(self.get(),·width=40))
500 ········return·"<%s·xpath=%r·data=%s>"·%·(type(self).__name__,·self._expr,500 ········return·"<%s·xpath=%r·data=%s>"·%·(type(self).__name__,·self._expr,
501 data)501 data)
502 ····__repr__·=·__str__502 ····__repr__·=·__str__
  
503 ===============================================================================503 ===============================================================================
504 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.504 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
505 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.505 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
987 B
./usr/share/doc/python-scrapy-doc/html/_modules/parsel/xpathfuncs.html
    
Offset 280, 15 lines modifiedOffset 280, 15 lines modified
280 ··········</div>280 ··········</div>
281 ··········<footer>281 ··········<footer>
  
282 ··<hr/>282 ··<hr/>
  
283 ··<div·role="contentinfo">283 ··<div·role="contentinfo">
284 ····<p>284 ····<p>
285 ········&#169;·Copyright·2008–2024,·Scrapy·developers.285 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
286 ······<span·class="lastupdated">286 ······<span·class="lastupdated">
287 ········Last·updated·on·May·20,·2022.287 ········Last·updated·on·May·20,·2022.
288 ······</span>288 ······</span>
  
289 ····</p>289 ····</p>
290 ··</div>290 ··</div>
291 ····291 ····
⊟
509 B
html2text {}
    
Offset 125, 9 lines modifiedOffset 125, 9 lines modified
125 ····node_cls·=·replace_html5_whitespaces('·',·node_cls)125 ····node_cls·=·replace_html5_whitespaces('·',·node_cls)
126 ····for·cls·in·classes:126 ····for·cls·in·classes:
127 ········if·'·'·+·cls·+·'·'·not·in·node_cls:127 ········if·'·'·+·cls·+·'·'·not·in·node_cls:
128 ············return·False128 ············return·False
129 ····return·True129 ····return·True
  
130 ===============================================================================130 ===============================================================================
131 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.131 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
132 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.132 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.05 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/contracts/default.html
    
Offset 329, 15 lines modifiedOffset 329, 15 lines modified
329 ··········</div>329 ··········</div>
330 ··········<footer>330 ··········<footer>
  
331 ··<hr/>331 ··<hr/>
  
332 ··<div·role="contentinfo">332 ··<div·role="contentinfo">
333 ····<p>333 ····<p>
334 ········&#169;·Copyright·2008–2024,·Scrapy·developers.334 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
335 ······<span·class="lastupdated">335 ······<span·class="lastupdated">
336 ········Last·updated·on·May·20,·2022.336 ········Last·updated·on·May·20,·2022.
337 ······</span>337 ······</span>
  
338 ····</p>338 ····</p>
339 ··</div>339 ··</div>
340 ····340 ····
⊟
587 B
html2text {}
    
Offset 177, 9 lines modifiedOffset 177, 9 lines modified
177 ················missing·=·[arg·for·arg·in·self.args·if·arg·not·in·ItemAdapter177 ················missing·=·[arg·for·arg·in·self.args·if·arg·not·in·ItemAdapter
178 (x)]178 (x)]
179 ················if·missing:179 ················if·missing:
180 ····················missing_fields·=·",·".join(missing)180 ····················missing_fields·=·",·".join(missing)
181 ····················raise·ContractFail(f"Missing·fields:·{missing_fields}")181 ····················raise·ContractFail(f"Missing·fields:·{missing_fields}")
  
182 ===============================================================================182 ===============================================================================
183 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.183 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
184 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.184 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1000 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/contracts.html
    
Offset 398, 15 lines modifiedOffset 398, 15 lines modified
398 ··········</div>398 ··········</div>
399 ··········<footer>399 ··········<footer>
  
400 ··<hr/>400 ··<hr/>
  
401 ··<div·role="contentinfo">401 ··<div·role="contentinfo">
402 ····<p>402 ····<p>
403 ········&#169;·Copyright·2008–2024,·Scrapy·developers.403 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
404 ······<span·class="lastupdated">404 ······<span·class="lastupdated">
405 ········Last·updated·on·May·20,·2022.405 ········Last·updated·on·May·20,·2022.
406 ······</span>406 ······</span>
  
407 ····</p>407 ····</p>
408 ··</div>408 ··</div>
409 ····409 ····
⊟
529 B
html2text {}
    
Offset 245, 9 lines modifiedOffset 245, 9 lines modified
245 ············return·f"[{spider}]·{method.__name__}·({desc})"245 ············return·f"[{spider}]·{method.__name__}·({desc})"
  
246 ····name·=·f'{spider}_{method.__name__}'246 ····name·=·f'{spider}_{method.__name__}'
247 ····setattr(ContractTestCase,·name,·lambda·x:·x)247 ····setattr(ContractTestCase,·name,·lambda·x:·x)
248 ····return·ContractTestCase(name)248 ····return·ContractTestCase(name)
  
249 ===============================================================================249 ===============================================================================
250 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.250 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
251 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.251 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.02 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/crawler.html
    
Offset 563, 15 lines modifiedOffset 563, 15 lines modified
563 ··········</div>563 ··········</div>
564 ··········<footer>564 ··········<footer>
  
565 ··<hr/>565 ··<hr/>
  
566 ··<div·role="contentinfo">566 ··<div·role="contentinfo">
567 ····<p>567 ····<p>
568 ········&#169;·Copyright·2008–2024,·Scrapy·developers.568 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
569 ······<span·class="lastupdated">569 ······<span·class="lastupdated">
570 ········Last·updated·on·May·20,·2022.570 ········Last·updated·on·May·20,·2022.
571 ······</span>571 ······</span>
  
572 ····</p>572 ····</p>
573 ··</div>573 ··</div>
574 ····574 ····
⊟
574 B
html2text {}
    
Offset 424, 9 lines modifiedOffset 424, 9 lines modified
424 ····def·_handle_twisted_reactor(self):424 ····def·_handle_twisted_reactor(self):
425 ········if·self.settings.get("TWISTED_REACTOR"):425 ········if·self.settings.get("TWISTED_REACTOR"):
426 ············install_reactor(self.settings["TWISTED_REACTOR"],·self.settings426 ············install_reactor(self.settings["TWISTED_REACTOR"],·self.settings
427 ["ASYNCIO_EVENT_LOOP"])427 ["ASYNCIO_EVENT_LOOP"])
428 ········super()._handle_twisted_reactor()428 ········super()._handle_twisted_reactor()
  
429 ===============================================================================429 ===============================================================================
430 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.430 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
431 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.431 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.04 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/ajaxcrawl.html
    
Offset 312, 15 lines modifiedOffset 312, 15 lines modified
312 ··········</div>312 ··········</div>
313 ··········<footer>313 ··········<footer>
  
314 ··<hr/>314 ··<hr/>
  
315 ··<div·role="contentinfo">315 ··<div·role="contentinfo">
316 ····<p>316 ····<p>
317 ········&#169;·Copyright·2008–2024,·Scrapy·developers.317 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
318 ······<span·class="lastupdated">318 ······<span·class="lastupdated">
319 ········Last·updated·on·May·20,·2022.319 ········Last·updated·on·May·20,·2022.
320 ······</span>320 ······</span>
  
321 ····</p>321 ····</p>
322 ··</div>322 ··</div>
323 ····323 ····
⊟
548 B
html2text {}
    
Offset 164, 9 lines modifiedOffset 164, 9 lines modified
  
164 ····text·=·html.remove_tags_with_content(text,·('script',·'noscript'))164 ····text·=·html.remove_tags_with_content(text,·('script',·'noscript'))
165 ····text·=·html.replace_entities(text)165 ····text·=·html.replace_entities(text)
166 ····text·=·html.remove_comments(text)166 ····text·=·html.remove_comments(text)
167 ····return·_ajax_crawlable_re.search(text)·is·not·None167 ····return·_ajax_crawlable_re.search(text)·is·not·None
  
168 ===============================================================================168 ===============================================================================
169 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.169 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
170 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.170 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.08 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/cookies.html
    
Offset 329, 15 lines modifiedOffset 329, 15 lines modified
329 ··········</div>329 ··········</div>
330 ··········<footer>330 ··········<footer>
  
331 ··<hr/>331 ··<hr/>
  
332 ··<div·role="contentinfo">332 ··<div·role="contentinfo">
333 ····<p>333 ····<p>
334 ········&#169;·Copyright·2008–2024,·Scrapy·developers.334 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
335 ······<span·class="lastupdated">335 ······<span·class="lastupdated">
336 ········Last·updated·on·May·20,·2022.336 ········Last·updated·on·May·20,·2022.
337 ······</span>337 ······</span>
  
338 ····</p>338 ····</p>
339 ··</div>339 ··</div>
340 ····340 ····
⊟
594 B
html2text {}
    
Offset 178, 9 lines modifiedOffset 178, 9 lines modified
178 ············cookies·=·request.cookies178 ············cookies·=·request.cookies
179 ········formatted·=·filter(None,·(self._format_cookie(c,·request)·for·c·in179 ········formatted·=·filter(None,·(self._format_cookie(c,·request)·for·c·in
180 cookies))180 cookies))
181 ········response·=·Response(request.url,·headers={"Set-Cookie":·formatted})181 ········response·=·Response(request.url,·headers={"Set-Cookie":·formatted})
182 ········return·jar.make_cookies(response,·request)182 ········return·jar.make_cookies(response,·request)
  
183 ===============================================================================183 ===============================================================================
184 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.184 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
185 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.185 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.01 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/defaultheaders.html
    
Offset 241, 15 lines modifiedOffset 241, 15 lines modified
241 ··········</div>241 ··········</div>
242 ··········<footer>242 ··········<footer>
  
243 ··<hr/>243 ··<hr/>
  
244 ··<div·role="contentinfo">244 ··<div·role="contentinfo">
245 ····<p>245 ····<p>
246 ········&#169;·Copyright·2008–2024,·Scrapy·developers.246 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
247 ······<span·class="lastupdated">247 ······<span·class="lastupdated">
248 ········Last·updated·on·May·20,·2022.248 ········Last·updated·on·May·20,·2022.
249 ······</span>249 ······</span>
  
250 ····</p>250 ····</p>
251 ··</div>251 ··</div>
252 ····252 ····
⊟
507 B
html2text {}
    
Offset 85, 9 lines modifiedOffset 85, 9 lines modified
85 ········return·cls(headers.items())85 ········return·cls(headers.items())
  
86 ····def·process_request(self,·request,·spider):86 ····def·process_request(self,·request,·spider):
87 ········for·k,·v·in·self._headers:87 ········for·k,·v·in·self._headers:
88 ············request.headers.setdefault(k,·v)88 ············request.headers.setdefault(k,·v)
  
89 ===============================================================================89 ===============================================================================
90 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.90 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
91 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.91 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.07 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/downloadtimeout.html
    
Offset 245, 15 lines modifiedOffset 245, 15 lines modified
245 ··········</div>245 ··········</div>
246 ··········<footer>246 ··········<footer>
  
247 ··<hr/>247 ··<hr/>
  
248 ··<div·role="contentinfo">248 ··<div·role="contentinfo">
249 ····<p>249 ····<p>
250 ········&#169;·Copyright·2008–2024,·Scrapy·developers.250 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
251 ······<span·class="lastupdated">251 ······<span·class="lastupdated">
252 ········Last·updated·on·May·20,·2022.252 ········Last·updated·on·May·20,·2022.
253 ······</span>253 ······</span>
  
254 ····</p>254 ····</p>
255 ··</div>255 ··</div>
256 ····256 ····
⊟
563 B
html2text {}
    
Offset 88, 9 lines modifiedOffset 88, 9 lines modified
88 ········self._timeout·=·getattr(spider,·'download_timeout',·self._timeout)88 ········self._timeout·=·getattr(spider,·'download_timeout',·self._timeout)
  
89 ····def·process_request(self,·request,·spider):89 ····def·process_request(self,·request,·spider):
90 ········if·self._timeout:90 ········if·self._timeout:
91 ············request.meta.setdefault('download_timeout',·self._timeout)91 ············request.meta.setdefault('download_timeout',·self._timeout)
  
92 ===============================================================================92 ===============================================================================
93 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.93 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
94 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.94 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.06 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/httpauth.html
    
Offset 269, 15 lines modifiedOffset 269, 15 lines modified
269 ··········</div>269 ··········</div>
270 ··········<footer>270 ··········<footer>
  
271 ··<hr/>271 ··<hr/>
  
272 ··<div·role="contentinfo">272 ··<div·role="contentinfo">
273 ····<p>273 ····<p>
274 ········&#169;·Copyright·2008–2024,·Scrapy·developers.274 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
275 ······<span·class="lastupdated">275 ······<span·class="lastupdated">
276 ········Last·updated·on·May·20,·2022.276 ········Last·updated·on·May·20,·2022.
277 ······</span>277 ······</span>
  
278 ····</p>278 ····</p>
279 ··</div>279 ··</div>
280 ····280 ····
⊟
566 B
html2text {}
    
Offset 116, 9 lines modifiedOffset 116, 9 lines modified
116 ················self.domain·=·domain116 ················self.domain·=·domain
117 ················self.domain_unset·=·False117 ················self.domain_unset·=·False
118 ············if·not·self.domain·or·url_is_from_any_domain(request.url,118 ············if·not·self.domain·or·url_is_from_any_domain(request.url,
119 [self.domain]):119 [self.domain]):
120 ················request.headers[b'Authorization']·=·auth120 ················request.headers[b'Authorization']·=·auth
  
121 ===============================================================================121 ===============================================================================
122 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.122 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
123 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.123 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.12 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/httpcache.html
    
Offset 352, 15 lines modifiedOffset 352, 15 lines modified
352 ··········</div>352 ··········</div>
353 ··········<footer>353 ··········<footer>
  
354 ··<hr/>354 ··<hr/>
  
355 ··<div·role="contentinfo">355 ··<div·role="contentinfo">
356 ····<p>356 ····<p>
357 ········&#169;·Copyright·2008–2024,·Scrapy·developers.357 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
358 ······<span·class="lastupdated">358 ······<span·class="lastupdated">
359 ········Last·updated·on·May·20,·2022.359 ········Last·updated·on·May·20,·2022.
360 ······</span>360 ······</span>
  
361 ····</p>361 ····</p>
362 ··</div>362 ··</div>
363 ····363 ····
⊟
630 B
html2text {}
    
Offset 204, 9 lines modifiedOffset 204, 9 lines modified
204 ········if·self.policy.should_cache_response(response,·request):204 ········if·self.policy.should_cache_response(response,·request):
205 ············self.stats.inc_value('httpcache/store',·spider=spider)205 ············self.stats.inc_value('httpcache/store',·spider=spider)
206 ············self.storage.store_response(spider,·request,·response)206 ············self.storage.store_response(spider,·request,·response)
207 ········else:207 ········else:
208 ············self.stats.inc_value('httpcache/uncacheable',·spider=spider)208 ············self.stats.inc_value('httpcache/uncacheable',·spider=spider)
  
209 ===============================================================================209 ===============================================================================
210 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.210 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
211 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.211 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.06 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/httpcompression.html
    
Offset 301, 15 lines modifiedOffset 301, 15 lines modified
301 ··········</div>301 ··········</div>
302 ··········<footer>302 ··········<footer>
  
303 ··<hr/>303 ··<hr/>
  
304 ··<div·role="contentinfo">304 ··<div·role="contentinfo">
305 ····<p>305 ····<p>
306 ········&#169;·Copyright·2008–2024,·Scrapy·developers.306 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
307 ······<span·class="lastupdated">307 ······<span·class="lastupdated">
308 ········Last·updated·on·May·20,·2022.308 ········Last·updated·on·May·20,·2022.
309 ······</span>309 ······</span>
  
310 ····</p>310 ····</p>
311 ··</div>311 ··</div>
312 ····312 ····
⊟
549 B
html2text {}
    
Offset 148, 9 lines modifiedOffset 148, 9 lines modified
148 ············#·where·there·is·content·size·data·embedded·in·the·frame148 ············#·where·there·is·content·size·data·embedded·in·the·frame
149 ············reader·=·zstandard.ZstdDecompressor().stream_reader(io.BytesIO149 ············reader·=·zstandard.ZstdDecompressor().stream_reader(io.BytesIO
150 (body))150 (body))
151 ············body·=·reader.read()151 ············body·=·reader.read()
152 ········return·body152 ········return·body
  
153 ===============================================================================153 ===============================================================================
154 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.154 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
155 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.155 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.05 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/httpproxy.html
    
Offset 294, 15 lines modifiedOffset 294, 15 lines modified
294 ··········</div>294 ··········</div>
295 ··········<footer>295 ··········<footer>
  
296 ··<hr/>296 ··<hr/>
  
297 ··<div·role="contentinfo">297 ··<div·role="contentinfo">
298 ····<p>298 ····<p>
299 ········&#169;·Copyright·2008–2024,·Scrapy·developers.299 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
300 ······<span·class="lastupdated">300 ······<span·class="lastupdated">
301 ········Last·updated·on·May·20,·2022.301 ········Last·updated·on·May·20,·2022.
302 ······</span>302 ······</span>
  
303 ····</p>303 ····</p>
304 ··</div>304 ··</div>
305 ····305 ····
⊟
558 B
html2text {}
    
Offset 138, 9 lines modifiedOffset 138, 9 lines modified
138 ····def·_set_proxy(self,·request,·scheme):138 ····def·_set_proxy(self,·request,·scheme):
139 ········creds,·proxy·=·self.proxies[scheme]139 ········creds,·proxy·=·self.proxies[scheme]
140 ········request.meta['proxy']·=·proxy140 ········request.meta['proxy']·=·proxy
141 ········if·creds:141 ········if·creds:
142 ············request.headers['Proxy-Authorization']·=·b'Basic·'·+·creds142 ············request.headers['Proxy-Authorization']·=·b'Basic·'·+·creds
  
143 ===============================================================================143 ===============================================================================
144 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.144 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
145 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.145 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.06 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/redirect.html
    
Offset 352, 15 lines modifiedOffset 352, 15 lines modified
352 ··········</div>352 ··········</div>
353 ··········<footer>353 ··········<footer>
  
354 ··<hr/>354 ··<hr/>
  
355 ··<div·role="contentinfo">355 ··<div·role="contentinfo">
356 ····<p>356 ····<p>
357 ········&#169;·Copyright·2008–2024,·Scrapy·developers.357 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
358 ······<span·class="lastupdated">358 ······<span·class="lastupdated">
359 ········Last·updated·on·May·20,·2022.359 ········Last·updated·on·May·20,·2022.
360 ······</span>360 ······</span>
  
361 ····</p>361 ····</p>
362 ··</div>362 ··</div>
363 ····363 ····
⊟
566 B
html2text {}
    
Offset 202, 9 lines modifiedOffset 202, 9 lines modified
202 ········if·url·and·interval·<·self._maxdelay:202 ········if·url·and·interval·<·self._maxdelay:
203 ············redirected·=·self._redirect_request_using_get(request,·url)203 ············redirected·=·self._redirect_request_using_get(request,·url)
204 ············return·self._redirect(redirected,·request,·spider,·'meta·refresh')204 ············return·self._redirect(redirected,·request,·spider,·'meta·refresh')
  
205 ········return·response205 ········return·response
  
206 ===============================================================================206 ===============================================================================
207 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.207 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
208 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.208 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.06 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/retry.html
    
Offset 316, 15 lines modifiedOffset 316, 15 lines modified
316 ··········</div>316 ··········</div>
317 ··········<footer>317 ··········<footer>
  
318 ··<hr/>318 ··<hr/>
  
319 ··<div·role="contentinfo">319 ··<div·role="contentinfo">
320 ····<p>320 ····<p>
321 ········&#169;·Copyright·2008–2024,·Scrapy·developers.321 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
322 ······<span·class="lastupdated">322 ······<span·class="lastupdated">
323 ········Last·updated·on·May·20,·2022.323 ········Last·updated·on·May·20,·2022.
324 ······</span>324 ······</span>
  
325 ····</p>325 ····</p>
326 ··</div>326 ··</div>
327 ····327 ····
⊟
576 B
html2text {}
    
Offset 166, 9 lines modifiedOffset 166, 9 lines modified
166 ············logger.error("Gave·up·retrying·%(request)s·(failed·%(retries)d166 ············logger.error("Gave·up·retrying·%(request)s·(failed·%(retries)d
167 times):·%(reason)s",167 times):·%(reason)s",
168 ·························{'request':·request,·'retries':·retries,·'reason':168 ·························{'request':·request,·'retries':·retries,·'reason':
169 reason},169 reason},
170 ·························extra={'spider':·spider})170 ·························extra={'spider':·spider})
  
171 ===============================================================================171 ===============================================================================
172 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.172 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
173 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.173 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.05 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/robotstxt.html
    
Offset 328, 15 lines modifiedOffset 328, 15 lines modified
328 ··········</div>328 ··········</div>
329 ··········<footer>329 ··········<footer>
  
330 ··<hr/>330 ··<hr/>
  
331 ··<div·role="contentinfo">331 ··<div·role="contentinfo">
332 ····<p>332 ····<p>
333 ········&#169;·Copyright·2008–2024,·Scrapy·developers.333 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
334 ······<span·class="lastupdated">334 ······<span·class="lastupdated">
335 ········Last·updated·on·May·20,·2022.335 ········Last·updated·on·May·20,·2022.
336 ······</span>336 ······</span>
  
337 ····</p>337 ····</p>
338 ··</div>338 ··</div>
339 ····339 ····
⊟
558 B
html2text {}
    
Offset 176, 9 lines modifiedOffset 176, 9 lines modified
176 ············key·=·f'robotstxt/exception_count/{failure.type}'176 ············key·=·f'robotstxt/exception_count/{failure.type}'
177 ············self.crawler.stats.inc_value(key)177 ············self.crawler.stats.inc_value(key)
178 ········rp_dfd·=·self._parsers[netloc]178 ········rp_dfd·=·self._parsers[netloc]
179 ········self._parsers[netloc]·=·None179 ········self._parsers[netloc]·=·None
180 ········rp_dfd.callback(None)180 ········rp_dfd.callback(None)
  
181 ===============================================================================181 ===============================================================================
182 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.182 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
183 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.183 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.11 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/stats.html
    
Offset 253, 15 lines modifiedOffset 253, 15 lines modified
253 ··········</div>253 ··········</div>
254 ··········<footer>254 ··········<footer>
  
255 ··<hr/>255 ··<hr/>
  
256 ··<div·role="contentinfo">256 ··<div·role="contentinfo">
257 ····<p>257 ····<p>
258 ········&#169;·Copyright·2008–2024,·Scrapy·developers.258 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
259 ······<span·class="lastupdated">259 ······<span·class="lastupdated">
260 ········Last·updated·on·May·20,·2022.260 ········Last·updated·on·May·20,·2022.
261 ······</span>261 ······</span>
  
262 ····</p>262 ····</p>
263 ··</div>263 ··</div>
264 ····264 ····
⊟
629 B
html2text {}
    
Offset 100, 9 lines modifiedOffset 100, 9 lines modified
100 ····def·process_exception(self,·request,·exception,·spider):100 ····def·process_exception(self,·request,·exception,·spider):
101 ········ex_class·=·global_object_name(exception.__class__)101 ········ex_class·=·global_object_name(exception.__class__)
102 ········self.stats.inc_value('downloader/exception_count',·spider=spider)102 ········self.stats.inc_value('downloader/exception_count',·spider=spider)
103 ········self.stats.inc_value(f'downloader/exception_type_count/{ex_class}',103 ········self.stats.inc_value(f'downloader/exception_type_count/{ex_class}',
104 spider=spider)104 spider=spider)
  
105 ===============================================================================105 ===============================================================================
106 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.106 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
107 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.107 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.06 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/downloadermiddlewares/useragent.html
    
Offset 242, 15 lines modifiedOffset 242, 15 lines modified
242 ··········</div>242 ··········</div>
243 ··········<footer>243 ··········<footer>
  
244 ··<hr/>244 ··<hr/>
  
245 ··<div·role="contentinfo">245 ··<div·role="contentinfo">
246 ····<p>246 ····<p>
247 ········&#169;·Copyright·2008–2024,·Scrapy·developers.247 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
248 ······<span·class="lastupdated">248 ······<span·class="lastupdated">
249 ········Last·updated·on·May·20,·2022.249 ········Last·updated·on·May·20,·2022.
250 ······</span>250 ······</span>
  
251 ····</p>251 ····</p>
252 ··</div>252 ··</div>
253 ····253 ····
⊟
563 B
html2text {}
    
Offset 85, 9 lines modifiedOffset 85, 9 lines modified
85 ········self.user_agent·=·getattr(spider,·'user_agent',·self.user_agent)85 ········self.user_agent·=·getattr(spider,·'user_agent',·self.user_agent)
  
86 ····def·process_request(self,·request,·spider):86 ····def·process_request(self,·request,·spider):
87 ········if·self.user_agent:87 ········if·self.user_agent:
88 ············request.headers.setdefault(b'User-Agent',·self.user_agent)88 ············request.headers.setdefault(b'User-Agent',·self.user_agent)
  
89 ===============================================================================89 ===============================================================================
90 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.90 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
91 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.91 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
928 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/exceptions.html
    
Offset 308, 15 lines modifiedOffset 308, 15 lines modified
308 ··········</div>308 ··········</div>
309 ··········<footer>309 ··········<footer>
  
310 ··<hr/>310 ··<hr/>
  
311 ··<div·role="contentinfo">311 ··<div·role="contentinfo">
312 ····<p>312 ····<p>
313 ········&#169;·Copyright·2008–2024,·Scrapy·developers.313 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
314 ······<span·class="lastupdated">314 ······<span·class="lastupdated">
315 ········Last·updated·on·May·20,·2022.315 ········Last·updated·on·May·20,·2022.
316 ······</span>316 ······</span>
  
317 ····</p>317 ····</p>
318 ··</div>318 ··</div>
319 ····319 ····
⊟
450 B
html2text {}
    
Offset 161, 9 lines modifiedOffset 161, 9 lines modified
  
  
161 [docs]class·ContractFail(AssertionError):161 [docs]class·ContractFail(AssertionError):
162 ····"""Error·raised·in·case·of·a·failing·contract"""162 ····"""Error·raised·in·case·of·a·failing·contract"""
163 ····pass163 ····pass
  
164 ===============================================================================164 ===============================================================================
165 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.165 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
166 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.166 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1010 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/exporters.html
    
Offset 557, 15 lines modifiedOffset 557, 15 lines modified
557 ··········</div>557 ··········</div>
558 ··········<footer>558 ··········<footer>
  
559 ··<hr/>559 ··<hr/>
  
560 ··<div·role="contentinfo">560 ··<div·role="contentinfo">
561 ····<p>561 ····<p>
562 ········&#169;·Copyright·2008–2024,·Scrapy·developers.562 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
563 ······<span·class="lastupdated">563 ······<span·class="lastupdated">
564 ········Last·updated·on·May·20,·2022.564 ········Last·updated·on·May·20,·2022.
565 ······</span>565 ······</span>
  
566 ····</p>566 ····</p>
567 ··</div>567 ··</div>
568 ····568 ····
⊟
536 B
html2text {}
    
Offset 421, 9 lines modifiedOffset 421, 9 lines modified
421 ····def·export_item(self,·item):421 ····def·export_item(self,·item):
422 ········result·=·dict(self._get_serialized_fields(item))422 ········result·=·dict(self._get_serialized_fields(item))
423 ········if·self.binary:423 ········if·self.binary:
424 ············result·=·dict(self._serialize_item(result))424 ············result·=·dict(self._serialize_item(result))
425 ········return·result425 ········return·result
  
426 ===============================================================================426 ===============================================================================
427 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.427 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
428 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.428 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
989 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/extensions/closespider.html
    
Offset 287, 15 lines modifiedOffset 287, 15 lines modified
287 ··········</div>287 ··········</div>
288 ··········<footer>288 ··········<footer>
  
289 ··<hr/>289 ··<hr/>
  
290 ··<div·role="contentinfo">290 ··<div·role="contentinfo">
291 ····<p>291 ····<p>
292 ········&#169;·Copyright·2008–2024,·Scrapy·developers.292 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
293 ······<span·class="lastupdated">293 ······<span·class="lastupdated">
294 ········Last·updated·on·May·20,·2022.294 ········Last·updated·on·May·20,·2022.
295 ······</span>295 ······</span>
  
296 ····</p>296 ····</p>
297 ··</div>297 ··</div>
298 ····298 ····
⊟
487 B
html2text {}
    
Offset 135, 9 lines modifiedOffset 135, 9 lines modified
  
135 ····def·spider_closed(self,·spider):135 ····def·spider_closed(self,·spider):
136 ········task·=·getattr(self,·'task',·False)136 ········task·=·getattr(self,·'task',·False)
137 ········if·task·and·task.active():137 ········if·task·and·task.active():
138 ············task.cancel()138 ············task.cancel()
  
139 ===============================================================================139 ===============================================================================
140 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.140 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
141 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.141 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.07 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/extensions/corestats.html
    
Offset 265, 15 lines modifiedOffset 265, 15 lines modified
265 ··········</div>265 ··········</div>
266 ··········<footer>266 ··········<footer>
  
267 ··<hr/>267 ··<hr/>
  
268 ··<div·role="contentinfo">268 ··<div·role="contentinfo">
269 ····<p>269 ····<p>
270 ········&#169;·Copyright·2008–2024,·Scrapy·developers.270 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
271 ······<span·class="lastupdated">271 ······<span·class="lastupdated">
272 ········Last·updated·on·May·20,·2022.272 ········Last·updated·on·May·20,·2022.
273 ······</span>273 ······</span>
  
274 ····</p>274 ····</p>
275 ··</div>275 ··</div>
276 ····276 ····
⊟
593 B
html2text {}
    
Offset 111, 9 lines modifiedOffset 111, 9 lines modified
111 ····def·item_dropped(self,·item,·spider,·exception):111 ····def·item_dropped(self,·item,·spider,·exception):
112 ········reason·=·exception.__class__.__name__112 ········reason·=·exception.__class__.__name__
113 ········self.stats.inc_value('item_dropped_count',·spider=spider)113 ········self.stats.inc_value('item_dropped_count',·spider=spider)
114 ········self.stats.inc_value(f'item_dropped_reasons_count/{reason}',114 ········self.stats.inc_value(f'item_dropped_reasons_count/{reason}',
115 spider=spider)115 spider=spider)
  
116 ===============================================================================116 ===============================================================================
117 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.117 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
118 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.118 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
995 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/extensions/debug.html
    
Offset 283, 15 lines modifiedOffset 283, 15 lines modified
283 ··········</div>283 ··········</div>
284 ··········<footer>284 ··········<footer>
  
285 ··<hr/>285 ··<hr/>
  
286 ··<div·role="contentinfo">286 ··<div·role="contentinfo">
287 ····<p>287 ····<p>
288 ········&#169;·Copyright·2008–2024,·Scrapy·developers.288 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
289 ······<span·class="lastupdated">289 ······<span·class="lastupdated">
290 ········Last·updated·on·May·20,·2022.290 ········Last·updated·on·May·20,·2022.
291 ······</span>291 ······</span>
  
292 ····</p>292 ····</p>
293 ··</div>293 ··</div>
294 ····294 ····
⊟
505 B
html2text {}
    
Offset 127, 9 lines modifiedOffset 127, 9 lines modified
127 ············#·win32·platforms·don't·support·SIGUSR·signals127 ············#·win32·platforms·don't·support·SIGUSR·signals
128 ············pass128 ············pass
  
129 ····def·_enter_debugger(self,·signum,·frame):129 ····def·_enter_debugger(self,·signum,·frame):
130 ········Pdb().set_trace(frame.f_back)130 ········Pdb().set_trace(frame.f_back)
  
131 ===============================================================================131 ===============================================================================
132 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.132 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
133 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.133 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1000 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/extensions/httpcache.html
    
Offset 591, 15 lines modifiedOffset 591, 15 lines modified
591 ··········</div>591 ··········</div>
592 ··········<footer>592 ··········<footer>
  
593 ··<hr/>593 ··<hr/>
  
594 ··<div·role="contentinfo">594 ··<div·role="contentinfo">
595 ····<p>595 ····<p>
596 ········&#169;·Copyright·2008–2024,·Scrapy·developers.596 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
597 ······<span·class="lastupdated">597 ······<span·class="lastupdated">
598 ········Last·updated·on·May·20,·2022.598 ········Last·updated·on·May·20,·2022.
599 ······</span>599 ······</span>
  
600 ····</p>600 ····</p>
601 ··</div>601 ··</div>
602 ····602 ····
⊟
502 B
html2text {}
    
Offset 450, 9 lines modifiedOffset 450, 9 lines modified
450 ····try:450 ····try:
451 ········date_str·=·to_unicode(date_str,·encoding='ascii')451 ········date_str·=·to_unicode(date_str,·encoding='ascii')
452 ········return·mktime_tz(parsedate_tz(date_str))452 ········return·mktime_tz(parsedate_tz(date_str))
453 ····except·Exception:453 ····except·Exception:
454 ········return·None454 ········return·None
  
455 ===============================================================================455 ===============================================================================
456 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.456 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
457 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.457 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1020 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/extensions/logstats.html
    
Offset 271, 15 lines modifiedOffset 271, 15 lines modified
271 ··········</div>271 ··········</div>
272 ··········<footer>272 ··········<footer>
  
273 ··<hr/>273 ··<hr/>
  
274 ··<div·role="contentinfo">274 ··<div·role="contentinfo">
275 ····<p>275 ····<p>
276 ········&#169;·Copyright·2008–2024,·Scrapy·developers.276 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
277 ······<span·class="lastupdated">277 ······<span·class="lastupdated">
278 ········Last·updated·on·May·20,·2022.278 ········Last·updated·on·May·20,·2022.
279 ······</span>279 ······</span>
  
280 ····</p>280 ····</p>
281 ··</div>281 ··</div>
282 ····282 ····
⊟
524 B
html2text {}
    
Offset 114, 9 lines modifiedOffset 114, 9 lines modified
114 ········logger.info(msg,·log_args,·extra={'spider':·spider})114 ········logger.info(msg,·log_args,·extra={'spider':·spider})
  
115 ····def·spider_closed(self,·spider,·reason):115 ····def·spider_closed(self,·spider,·reason):
116 ········if·self.task·and·self.task.running:116 ········if·self.task·and·self.task.running:
117 ············self.task.stop()117 ············self.task.stop()
  
118 ===============================================================================118 ===============================================================================
119 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.119 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
120 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.120 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.01 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/extensions/memdebug.html
    
Offset 252, 15 lines modifiedOffset 252, 15 lines modified
252 ··········</div>252 ··········</div>
253 ··········<footer>253 ··········<footer>
  
254 ··<hr/>254 ··<hr/>
  
255 ··<div·role="contentinfo">255 ··<div·role="contentinfo">
256 ····<p>256 ····<p>
257 ········&#169;·Copyright·2008–2024,·Scrapy·developers.257 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
258 ······<span·class="lastupdated">258 ······<span·class="lastupdated">
259 ········Last·updated·on·May·20,·2022.259 ········Last·updated·on·May·20,·2022.
260 ······</span>260 ······</span>
  
261 ····</p>261 ····</p>
262 ··</div>262 ··</div>
263 ····263 ····
⊟
537 B
html2text {}
    
Offset 97, 9 lines modifiedOffset 97, 9 lines modified
97 ········for·cls,·wdict·in·live_refs.items():97 ········for·cls,·wdict·in·live_refs.items():
98 ············if·not·wdict:98 ············if·not·wdict:
99 ················continue99 ················continue
100 ············self.stats.set_value(f'memdebug/live_refs/{cls.__name__}',·len100 ············self.stats.set_value(f'memdebug/live_refs/{cls.__name__}',·len
101 (wdict),·spider=spider)101 (wdict),·spider=spider)
  
102 ===============================================================================102 ===============================================================================
103 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.103 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
104 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.104 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
994 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/extensions/memusage.html
    
Offset 345, 15 lines modifiedOffset 345, 15 lines modified
345 ··········</div>345 ··········</div>
346 ··········<footer>346 ··········<footer>
  
347 ··<hr/>347 ··<hr/>
  
348 ··<div·role="contentinfo">348 ··<div·role="contentinfo">
349 ····<p>349 ····<p>
350 ········&#169;·Copyright·2008–2024,·Scrapy·developers.350 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
351 ······<span·class="lastupdated">351 ······<span·class="lastupdated">
352 ········Last·updated·on·May·20,·2022.352 ········Last·updated·on·May·20,·2022.
353 ······</span>353 ······</span>
  
354 ····</p>354 ····</p>
355 ··</div>355 ··</div>
356 ····356 ····
⊟
498 B
html2text {}
    
Offset 198, 9 lines modifiedOffset 198, 9 lines modified
198 ----·\r\n"198 ----·\r\n"
199 ········s·+=·"\r\n"199 ········s·+=·"\r\n"
200 ········s·+=·pformat(get_engine_status(self.crawler.engine))200 ········s·+=·pformat(get_engine_status(self.crawler.engine))
201 ········s·+=·"\r\n"201 ········s·+=·"\r\n"
202 ········self.mail.send(rcpts,·subject,·s)202 ········self.mail.send(rcpts,·subject,·s)
  
203 ===============================================================================203 ===============================================================================
204 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.204 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
205 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.205 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.04 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/extensions/statsmailer.html
    
Offset 253, 15 lines modifiedOffset 253, 15 lines modified
253 ··········</div>253 ··········</div>
254 ··········<footer>254 ··········<footer>
  
255 ··<hr/>255 ··<hr/>
  
256 ··<div·role="contentinfo">256 ··<div·role="contentinfo">
257 ····<p>257 ····<p>
258 ········&#169;·Copyright·2008–2024,·Scrapy·developers.258 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
259 ······<span·class="lastupdated">259 ······<span·class="lastupdated">
260 ········Last·updated·on·May·20,·2022.260 ········Last·updated·on·May·20,·2022.
261 ······</span>261 ······</span>
  
262 ····</p>262 ····</p>
263 ··</div>263 ··</div>
264 ····264 ····
⊟
567 B
html2text {}
    
Offset 98, 9 lines modifiedOffset 98, 9 lines modified
98 ().items())98 ().items())
99 ········body·+=·f"\n\n{spider.name}·stats\n\n"99 ········body·+=·f"\n\n{spider.name}·stats\n\n"
100 ········body·+=·"\n".join(f"{k:<50}·:·{v}"·for·k,·v·in·spider_stats.items())100 ········body·+=·"\n".join(f"{k:<50}·:·{v}"·for·k,·v·in·spider_stats.items())
101 ········return·self.mail.send(self.recipients,·f"Scrapy·stats·for:101 ········return·self.mail.send(self.recipients,·f"Scrapy·stats·for:
102 {spider.name}",·body)102 {spider.name}",·body)
  
103 ===============================================================================103 ===============================================================================
104 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.104 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
105 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.105 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
983 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/extensions/telnet.html
    
Offset 333, 15 lines modifiedOffset 333, 15 lines modified
333 ··········</div>333 ··········</div>
334 ··········<footer>334 ··········<footer>
  
335 ··<hr/>335 ··<hr/>
  
336 ··<div·role="contentinfo">336 ··<div·role="contentinfo">
337 ····<p>337 ····<p>
338 ········&#169;·Copyright·2008–2024,·Scrapy·developers.338 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
339 ······<span·class="lastupdated">339 ······<span·class="lastupdated">
340 ········Last·updated·on·May·20,·2022.340 ········Last·updated·on·May·20,·2022.
341 ······</span>341 ······</span>
  
342 ····</p>342 ····</p>
343 ··</div>343 ··</div>
344 ····344 ····
⊟
491 B
html2text {}
    
Offset 181, 9 lines modifiedOffset 181, 9 lines modified
181 telnetconsole.html",181 telnetconsole.html",
182 ········}182 ········}
183 ········self.crawler.signals.send_catch_log(update_telnet_vars,183 ········self.crawler.signals.send_catch_log(update_telnet_vars,
184 telnet_vars=telnet_vars)184 telnet_vars=telnet_vars)
185 ········return·telnet_vars185 ········return·telnet_vars
  
186 ===============================================================================186 ===============================================================================
187 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.187 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
188 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.188 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.05 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/http/request/form.html
    
Offset 436, 15 lines modifiedOffset 436, 15 lines modified
436 ··········</div>436 ··········</div>
437 ··········<footer>437 ··········<footer>
  
438 ··<hr/>438 ··<hr/>
  
439 ··<div·role="contentinfo">439 ··<div·role="contentinfo">
440 ····<p>440 ····<p>
441 ········&#169;·Copyright·2008–2024,·Scrapy·developers.441 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
442 ······<span·class="lastupdated">442 ······<span·class="lastupdated">
443 ········Last·updated·on·May·20,·2022.443 ········Last·updated·on·May·20,·2022.
444 ······</span>444 ······</span>
  
445 ····</p>445 ····</p>
446 ··</div>446 ··</div>
447 ····447 ····
⊟
579 B
html2text {}
    
Offset 289, 9 lines modifiedOffset 289, 9 lines modified
289 ········raise·ValueError(f"Multiple·elements·found·({el!r})·matching·the·"289 ········raise·ValueError(f"Multiple·elements·found·({el!r})·matching·the·"
290 ·························f"criteria·in·clickdata:·{clickdata!r}")290 ·························f"criteria·in·clickdata:·{clickdata!r}")
291 ····else:291 ····else:
292 ········raise·ValueError(f'No·clickable·element·matching·clickdata:292 ········raise·ValueError(f'No·clickable·element·matching·clickdata:
293 {clickdata!r}')293 {clickdata!r}')
  
294 ===============================================================================294 ===============================================================================
295 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.295 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
296 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.296 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
975 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/http/request/json_request.html
    
Offset 278, 15 lines modifiedOffset 278, 15 lines modified
278 ··········</div>278 ··········</div>
279 ··········<footer>279 ··········<footer>
  
280 ··<hr/>280 ··<hr/>
  
281 ··<div·role="contentinfo">281 ··<div·role="contentinfo">
282 ····<p>282 ····<p>
283 ········&#169;·Copyright·2008–2024,·Scrapy·developers.283 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
284 ······<span·class="lastupdated">284 ······<span·class="lastupdated">
285 ········Last·updated·on·May·20,·2022.285 ········Last·updated·on·May·20,·2022.
286 ······</span>286 ······</span>
  
287 ····</p>287 ····</p>
288 ··</div>288 ··</div>
289 ····289 ····
⊟
467 B
html2text {}
    
Offset 122, 9 lines modifiedOffset 122, 9 lines modified
122 ········return·json.dumps(data,·**self._dumps_kwargs)122 ········return·json.dumps(data,·**self._dumps_kwargs)
  
  
  
123 JSONRequest·=·create_deprecated_class("JSONRequest",·JsonRequest)123 JSONRequest·=·create_deprecated_class("JSONRequest",·JsonRequest)
  
124 ===============================================================================124 ===============================================================================
125 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.125 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
126 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.126 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
998 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/http/request.html
    
Offset 362, 15 lines modifiedOffset 362, 15 lines modified
362 ··········</div>362 ··········</div>
363 ··········<footer>363 ··········<footer>
  
364 ··<hr/>364 ··<hr/>
  
365 ··<div·role="contentinfo">365 ··<div·role="contentinfo">
366 ····<p>366 ····<p>
367 ········&#169;·Copyright·2008–2024,·Scrapy·developers.367 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
368 ······<span·class="lastupdated">368 ······<span·class="lastupdated">
369 ········Last·updated·on·May·20,·2022.369 ········Last·updated·on·May·20,·2022.
370 ······</span>370 ······</span>
  
371 ····</p>371 ····</p>
372 ··</div>372 ··</div>
373 ····373 ····
⊟
516 B
html2text {}
    
Offset 219, 9 lines modifiedOffset 219, 9 lines modified
219 ·······"""219 ·······"""
220 ········request_kwargs·=·curl_to_request_kwargs(curl_command,220 ········request_kwargs·=·curl_to_request_kwargs(curl_command,
221 ignore_unknown_options)221 ignore_unknown_options)
222 ········request_kwargs.update(kwargs)222 ········request_kwargs.update(kwargs)
223 ········return·cls(**request_kwargs)223 ········return·cls(**request_kwargs)
  
224 ===============================================================================224 ===============================================================================
225 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.225 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
226 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.226 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
938 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/http/response/html.html
    
Offset 233, 15 lines modifiedOffset 233, 15 lines modified
233 ··········</div>233 ··········</div>
234 ··········<footer>234 ··········<footer>
  
235 ··<hr/>235 ··<hr/>
  
236 ··<div·role="contentinfo">236 ··<div·role="contentinfo">
237 ····<p>237 ····<p>
238 ········&#169;·Copyright·2008–2024,·Scrapy·developers.238 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
239 ······<span·class="lastupdated">239 ······<span·class="lastupdated">
240 ········Last·updated·on·May·20,·2022.240 ········Last·updated·on·May·20,·2022.
241 ······</span>241 ······</span>
  
242 ····</p>242 ····</p>
243 ··</div>243 ··</div>
244 ····244 ····
⊟
444 B
html2text {}
    
Offset 75, 9 lines modifiedOffset 75, 9 lines modified
75 from·scrapy.http.response.text·import·TextResponse75 from·scrapy.http.response.text·import·TextResponse
  
  
76 [docs]class·HtmlResponse(TextResponse):76 [docs]class·HtmlResponse(TextResponse):
77 ····pass77 ····pass
  
78 ===============================================================================78 ===============================================================================
79 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.79 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
80 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.80 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1020 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/http/response/text.html
    
Offset 486, 15 lines modifiedOffset 486, 15 lines modified
486 ··········</div>486 ··········</div>
487 ··········<footer>487 ··········<footer>
  
488 ··<hr/>488 ··<hr/>
  
489 ··<div·role="contentinfo">489 ··<div·role="contentinfo">
490 ····<p>490 ····<p>
491 ········&#169;·Copyright·2008–2024,·Scrapy·developers.491 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
492 ······<span·class="lastupdated">492 ······<span·class="lastupdated">
493 ········Last·updated·on·May·20,·2022.493 ········Last·updated·on·May·20,·2022.
494 ······</span>494 ······</span>
  
495 ····</p>495 ····</p>
496 ··</div>496 ··</div>
497 ····497 ····
⊟
526 B
html2text {}
    
Offset 350, 9 lines modifiedOffset 350, 9 lines modified
350 ····href·=·sel.root.get('href')350 ····href·=·sel.root.get('href')
351 ····if·href·is·None:351 ····if·href·is·None:
352 ········raise·_InvalidSelector(f"<{sel.root.tag}>·element·has·no·href352 ········raise·_InvalidSelector(f"<{sel.root.tag}>·element·has·no·href
353 attribute:·{sel}")353 attribute:·{sel}")
354 ····return·strip_html5_whitespace(href)354 ····return·strip_html5_whitespace(href)
  
355 ===============================================================================355 ===============================================================================
356 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.356 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
357 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.357 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
935 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/http/response/xml.html
    
Offset 233, 15 lines modifiedOffset 233, 15 lines modified
233 ··········</div>233 ··········</div>
234 ··········<footer>234 ··········<footer>
  
235 ··<hr/>235 ··<hr/>
  
236 ··<div·role="contentinfo">236 ··<div·role="contentinfo">
237 ····<p>237 ····<p>
238 ········&#169;·Copyright·2008–2024,·Scrapy·developers.238 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
239 ······<span·class="lastupdated">239 ······<span·class="lastupdated">
240 ········Last·updated·on·May·20,·2022.240 ········Last·updated·on·May·20,·2022.
241 ······</span>241 ······</span>
  
242 ····</p>242 ····</p>
243 ··</div>243 ··</div>
244 ····244 ····
⊟
443 B
html2text {}
    
Offset 75, 9 lines modifiedOffset 75, 9 lines modified
75 from·scrapy.http.response.text·import·TextResponse75 from·scrapy.http.response.text·import·TextResponse
  
  
76 [docs]class·XmlResponse(TextResponse):76 [docs]class·XmlResponse(TextResponse):
77 ····pass77 ····pass
  
78 ===============================================================================78 ===============================================================================
79 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.79 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
80 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.80 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
946 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/http/response.html
    
Offset 415, 15 lines modifiedOffset 415, 15 lines modified
415 ··········</div>415 ··········</div>
416 ··········<footer>416 ··········<footer>
  
417 ··<hr/>417 ··<hr/>
  
418 ··<div·role="contentinfo">418 ··<div·role="contentinfo">
419 ····<p>419 ····<p>
420 ········&#169;·Copyright·2008–2024,·Scrapy·developers.420 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
421 ······<span·class="lastupdated">421 ······<span·class="lastupdated">
422 ········Last·updated·on·May·20,·2022.422 ········Last·updated·on·May·20,·2022.
423 ······</span>423 ······</span>
  
424 ····</p>424 ····</p>
425 ··</div>425 ··</div>
426 ····426 ····
⊟
462 B
html2text {}
    
Offset 267, 9 lines modifiedOffset 267, 9 lines modified
267 ················cb_kwargs=cb_kwargs,267 ················cb_kwargs=cb_kwargs,
268 ················flags=flags,268 ················flags=flags,
269 ············)269 ············)
270 ············for·url·in·urls270 ············for·url·in·urls
271 ········)271 ········)
  
272 ===============================================================================272 ===============================================================================
273 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.273 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
274 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.274 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.0 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/item.html
    
Offset 377, 15 lines modifiedOffset 377, 15 lines modified
377 ··········</div>377 ··········</div>
378 ··········<footer>378 ··········<footer>
  
379 ··<hr/>379 ··<hr/>
  
380 ··<div·role="contentinfo">380 ··<div·role="contentinfo">
381 ····<p>381 ····<p>
382 ········&#169;·Copyright·2008–2024,·Scrapy·developers.382 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
383 ······<span·class="lastupdated">383 ······<span·class="lastupdated">
384 ········Last·updated·on·May·20,·2022.384 ········Last·updated·on·May·20,·2022.
385 ······</span>385 ······</span>
  
386 ····</p>386 ····</p>
387 ··</div>387 ··</div>
388 ····388 ····
⊟
561 B
html2text {}
    
Offset 232, 9 lines modifiedOffset 232, 9 lines modified
232 ····about·fields·<topics-items-fields>`·for·additional·information.232 ····about·fields·<topics-items-fields>`·for·additional·information.
  
233 ····Unlike·instances·of·:class:`dict`,·instances·of·:class:`Item`·may·be233 ····Unlike·instances·of·:class:`dict`,·instances·of·:class:`Item`·may·be
234 ····:ref:`tracked·<topics-leaks-trackrefs>`·to·debug·memory·leaks.234 ····:ref:`tracked·<topics-leaks-trackrefs>`·to·debug·memory·leaks.
235 ····"""235 ····"""
  
236 ===============================================================================236 ===============================================================================
237 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.237 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
238 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.238 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
991 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/link.html
    
Offset 272, 15 lines modifiedOffset 272, 15 lines modified
272 ··········</div>272 ··········</div>
273 ··········<footer>273 ··········<footer>
  
274 ··<hr/>274 ··<hr/>
  
275 ··<div·role="contentinfo">275 ··<div·role="contentinfo">
276 ····<p>276 ····<p>
277 ········&#169;·Copyright·2008–2024,·Scrapy·developers.277 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
278 ······<span·class="lastupdated">278 ······<span·class="lastupdated">
279 ········Last·updated·on·May·20,·2022.279 ········Last·updated·on·May·20,·2022.
280 ······</span>280 ······</span>
  
281 ····</p>281 ····</p>
282 ··</div>282 ··</div>
283 ····283 ····
⊟
525 B
html2text {}
    
Offset 120, 9 lines modifiedOffset 120, 9 lines modified
120 ····def·__repr__(self):120 ····def·__repr__(self):
121 ········return·(121 ········return·(
122 ············f'Link(url={self.url!r},·text={self.text!r},·'122 ············f'Link(url={self.url!r},·text={self.text!r},·'
123 ············f'fragment={self.fragment!r},·nofollow={self.nofollow!r})'123 ············f'fragment={self.fragment!r},·nofollow={self.nofollow!r})'
124 ········)124 ········)
  
125 ===============================================================================125 ===============================================================================
126 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.126 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
127 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.127 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.03 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/linkextractors/lxmlhtml.html
    
Offset 383, 15 lines modifiedOffset 383, 15 lines modified
383 ··········</div>383 ··········</div>
384 ··········<footer>384 ··········<footer>
  
385 ··<hr/>385 ··<hr/>
  
386 ··<div·role="contentinfo">386 ··<div·role="contentinfo">
387 ····<p>387 ····<p>
388 ········&#169;·Copyright·2008–2024,·Scrapy·developers.388 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
389 ······<span·class="lastupdated">389 ······<span·class="lastupdated">
390 ········Last·updated·on·May·20,·2022.390 ········Last·updated·on·May·20,·2022.
391 ······</span>391 ······</span>
  
392 ····</p>392 ····</p>
393 ··</div>393 ··</div>
394 ····394 ····
⊟
552 B
html2text {}
    
Offset 232, 9 lines modifiedOffset 232, 9 lines modified
232 ········for·doc·in·docs:232 ········for·doc·in·docs:
233 ············links·=·self._extract_links(doc,·response.url,·response.encoding,233 ············links·=·self._extract_links(doc,·response.url,·response.encoding,
234 base_url)234 base_url)
235 ············all_links.extend(self._process_links(links))235 ············all_links.extend(self._process_links(links))
236 ········return·unique_list(all_links)236 ········return·unique_list(all_links)
  
237 ===============================================================================237 ===============================================================================
238 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.238 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
239 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.239 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.03 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/loader.html
    
Offset 307, 15 lines modifiedOffset 307, 15 lines modified
307 ··········</div>307 ··········</div>
308 ··········<footer>308 ··········<footer>
  
309 ··<hr/>309 ··<hr/>
  
310 ··<div·role="contentinfo">310 ··<div·role="contentinfo">
311 ····<p>311 ····<p>
312 ········&#169;·Copyright·2008–2024,·Scrapy·developers.312 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
313 ······<span·class="lastupdated">313 ······<span·class="lastupdated">
314 ········Last·updated·on·May·20,·2022.314 ········Last·updated·on·May·20,·2022.
315 ······</span>315 ······</span>
  
316 ····</p>316 ····</p>
317 ··</div>317 ··</div>
318 ····318 ····
⊟
582 B
html2text {}
    
Offset 160, 9 lines modifiedOffset 160, 9 lines modified
160 ········if·selector·is·None·and·response·is·not·None:160 ········if·selector·is·None·and·response·is·not·None:
161 ············selector·=·self.default_selector_class(response)161 ············selector·=·self.default_selector_class(response)
162 ········context.update(response=response)162 ········context.update(response=response)
163 ········super().__init__(item=item,·selector=selector,·parent=parent,163 ········super().__init__(item=item,·selector=selector,·parent=parent,
164 **context)164 **context)
  
165 ===============================================================================165 ===============================================================================
166 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.166 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
167 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.167 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
902 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/logformatter.html
    
Offset 366, 15 lines modifiedOffset 366, 15 lines modified
366 ··········</div>366 ··········</div>
367 ··········<footer>367 ··········<footer>
  
368 ··<hr/>368 ··<hr/>
  
369 ··<div·role="contentinfo">369 ··<div·role="contentinfo">
370 ····<p>370 ····<p>
371 ········&#169;·Copyright·2008–2024,·Scrapy·developers.371 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
372 ······<span·class="lastupdated">372 ······<span·class="lastupdated">
373 ········Last·updated·on·May·20,·2022.373 ········Last·updated·on·May·20,·2022.
374 ······</span>374 ······</span>
  
375 ····</p>375 ····</p>
376 ··</div>376 ··</div>
377 ····377 ····
⊟
420 B
html2text {}
    
Offset 229, 9 lines modifiedOffset 229, 9 lines modified
  
  
229 ····@classmethod229 ····@classmethod
230 ····def·from_crawler(cls,·crawler):230 ····def·from_crawler(cls,·crawler):
231 ········return·cls()231 ········return·cls()
  
232 ===============================================================================232 ===============================================================================
233 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.233 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
234 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.234 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
940 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/mail.html
    
Offset 359, 15 lines modifiedOffset 359, 15 lines modified
359 ··········</div>359 ··········</div>
360 ··········<footer>360 ··········<footer>
  
361 ··<hr/>361 ··<hr/>
  
362 ··<div·role="contentinfo">362 ··<div·role="contentinfo">
363 ····<p>363 ····<p>
364 ········&#169;·Copyright·2008–2024,·Scrapy·developers.364 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
365 ······<span·class="lastupdated">365 ······<span·class="lastupdated">
366 ········Last·updated·on·May·20,·2022.366 ········Last·updated·on·May·20,·2022.
367 ······</span>367 ······</span>
  
368 ····</p>368 ····</p>
369 ··</div>369 ··</div>
370 ····370 ····
⊟
474 B
html2text {}
    
Offset 209, 9 lines modifiedOffset 209, 9 lines modified
209 ssl.ClientContextFactory())209 ssl.ClientContextFactory())
210 ········else:210 ········else:
211 ············reactor.connectTCP(self.smtphost,·self.smtpport,·factory)211 ············reactor.connectTCP(self.smtphost,·self.smtpport,·factory)
  
212 ········return·d212 ········return·d
  
213 ===============================================================================213 ===============================================================================
214 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.214 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
215 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.215 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.04 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/pipelines/files.html
    
Offset 733, 15 lines modifiedOffset 733, 15 lines modified
733 ··········</div>733 ··········</div>
734 ··········<footer>734 ··········<footer>
  
735 ··<hr/>735 ··<hr/>
  
736 ··<div·role="contentinfo">736 ··<div·role="contentinfo">
737 ····<p>737 ····<p>
738 ········&#169;·Copyright·2008–2024,·Scrapy·developers.738 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
739 ······<span·class="lastupdated">739 ······<span·class="lastupdated">
740 ········Last·updated·on·May·20,·2022.740 ········Last·updated·on·May·20,·2022.
741 ······</span>741 ······</span>
  
742 ····</p>742 ····</p>
743 ··</div>743 ··</div>
744 ····744 ····
⊟
573 B
html2text {}
    
Offset 594, 9 lines modifiedOffset 594, 9 lines modified
594 ············media_ext·=·''594 ············media_ext·=·''
595 ············media_type·=·mimetypes.guess_type(request.url)[0]595 ············media_type·=·mimetypes.guess_type(request.url)[0]
596 ············if·media_type:596 ············if·media_type:
597 ················media_ext·=·mimetypes.guess_extension(media_type)597 ················media_ext·=·mimetypes.guess_extension(media_type)
598 ········return·f'full/{media_guid}{media_ext}'598 ········return·f'full/{media_guid}{media_ext}'
  
599 ===============================================================================599 ===============================================================================
600 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.600 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
601 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.601 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.0 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/pipelines/images.html
    
Offset 395, 15 lines modifiedOffset 395, 15 lines modified
395 ··········</div>395 ··········</div>
396 ··········<footer>396 ··········<footer>
  
397 ··<hr/>397 ··<hr/>
  
398 ··<div·role="contentinfo">398 ··<div·role="contentinfo">
399 ····<p>399 ····<p>
400 ········&#169;·Copyright·2008–2024,·Scrapy·developers.400 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
401 ······<span·class="lastupdated">401 ······<span·class="lastupdated">
402 ········Last·updated·on·May·20,·2022.402 ········Last·updated·on·May·20,·2022.
403 ······</span>403 ······</span>
  
404 ····</p>404 ····</p>
405 ··</div>405 ··</div>
406 ····406 ····
⊟
539 B
html2text {}
    
Offset 246, 9 lines modifiedOffset 246, 9 lines modified
  
  
246 ····def·thumb_path(self,·request,·thumb_id,·response=None,·info=None):246 ····def·thumb_path(self,·request,·thumb_id,·response=None,·info=None):
247 ········thumb_guid·=·hashlib.sha1(to_bytes(request.url)).hexdigest()247 ········thumb_guid·=·hashlib.sha1(to_bytes(request.url)).hexdigest()
248 ········return·f'thumbs/{thumb_id}/{thumb_guid}.jpg'248 ········return·f'thumbs/{thumb_id}/{thumb_guid}.jpg'
  
249 ===============================================================================249 ===============================================================================
250 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.250 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
251 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.251 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
985 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/robotstxt.html
    
Offset 349, 15 lines modifiedOffset 349, 15 lines modified
349 ··········</div>349 ··········</div>
350 ··········<footer>350 ··········<footer>
  
351 ··<hr/>351 ··<hr/>
  
352 ··<div·role="contentinfo">352 ··<div·role="contentinfo">
353 ····<p>353 ····<p>
354 ········&#169;·Copyright·2008–2024,·Scrapy·developers.354 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
355 ······<span·class="lastupdated">355 ······<span·class="lastupdated">
356 ········Last·updated·on·May·20,·2022.356 ········Last·updated·on·May·20,·2022.
357 ······</span>357 ······</span>
  
358 ····</p>358 ····</p>
359 ··</div>359 ··</div>
360 ····360 ····
⊟
509 B
html2text {}
    
Offset 200, 9 lines modifiedOffset 200, 9 lines modified
  
200 ····def·allowed(self,·url,·user_agent):200 ····def·allowed(self,·url,·user_agent):
201 ········user_agent·=·to_unicode(user_agent)201 ········user_agent·=·to_unicode(user_agent)
202 ········url·=·to_unicode(url)202 ········url·=·to_unicode(url)
203 ········return·self.rp.can_fetch(url,·user_agent)203 ········return·self.rp.can_fetch(url,·user_agent)
  
204 ===============================================================================204 ===============================================================================
205 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.205 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
206 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.206 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1020 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/selector/unified.html
    
Offset 301, 15 lines modifiedOffset 301, 15 lines modified
301 ··········</div>301 ··········</div>
302 ··········<footer>302 ··········<footer>
  
303 ··<hr/>303 ··<hr/>
  
304 ··<div·role="contentinfo">304 ··<div·role="contentinfo">
305 ····<p>305 ····<p>
306 ········&#169;·Copyright·2008–2024,·Scrapy·developers.306 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
307 ······<span·class="lastupdated">307 ······<span·class="lastupdated">
308 ········Last·updated·on·May·20,·2022.308 ········Last·updated·on·May·20,·2022.
309 ······</span>309 ······</span>
  
310 ····</p>310 ····</p>
311 ··</div>311 ··</div>
312 ····312 ····
⊟
533 B
html2text {}
    
Offset 147, 9 lines modifiedOffset 147, 9 lines modified
147 ············text·=·response.text147 ············text·=·response.text
148 ············kwargs.setdefault('base_url',·response.url)148 ············kwargs.setdefault('base_url',·response.url)
  
149 ········self.response·=·response149 ········self.response·=·response
150 ········super().__init__(text=text,·type=st,·root=root,·**kwargs)150 ········super().__init__(text=text,·type=st,·root=root,·**kwargs)
  
151 ===============================================================================151 ===============================================================================
152 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.152 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
153 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.153 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.04 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/settings.html
    
Offset 684, 15 lines modifiedOffset 684, 15 lines modified
684 ··········</div>684 ··········</div>
685 ··········<footer>685 ··········<footer>
  
686 ··<hr/>686 ··<hr/>
  
687 ··<div·role="contentinfo">687 ··<div·role="contentinfo">
688 ····<p>688 ····<p>
689 ········&#169;·Copyright·2008–2024,·Scrapy·developers.689 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
690 ······<span·class="lastupdated">690 ······<span·class="lastupdated">
691 ········Last·updated·on·May·20,·2022.691 ········Last·updated·on·May·20,·2022.
692 ······</span>692 ······</span>
  
693 ····</p>693 ····</p>
694 ··</div>694 ··</div>
695 ····695 ····
⊟
587 B
html2text {}
    
Offset 552, 9 lines modifiedOffset 552, 9 lines modified
552 ····"""Return·a·dict·of·the·settings·that·have·been·overridden"""552 ····"""Return·a·dict·of·the·settings·that·have·been·overridden"""
553 ····for·name,·defvalue·in·iter_default_settings():553 ····for·name,·defvalue·in·iter_default_settings():
554 ········value·=·settings[name]554 ········value·=·settings[name]
555 ········if·not·isinstance(defvalue,·dict)·and·value·!=·defvalue:555 ········if·not·isinstance(defvalue,·dict)·and·value·!=·defvalue:
556 ············yield·name,·value556 ············yield·name,·value
  
557 ===============================================================================557 ===============================================================================
558 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.558 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
559 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.559 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.0 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/signalmanager.html
    
Offset 287, 15 lines modifiedOffset 287, 15 lines modified
287 ··········</div>287 ··········</div>
288 ··········<footer>288 ··········<footer>
  
289 ··<hr/>289 ··<hr/>
  
290 ··<div·role="contentinfo">290 ··<div·role="contentinfo">
291 ····<p>291 ····<p>
292 ········&#169;·Copyright·2008–2024,·Scrapy·developers.292 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
293 ······<span·class="lastupdated">293 ······<span·class="lastupdated">
294 ········Last·updated·on·May·20,·2022.294 ········Last·updated·on·May·20,·2022.
295 ······</span>295 ······</span>
  
296 ····</p>296 ····</p>
297 ··</div>297 ··</div>
298 ····298 ····
⊟
543 B
html2text {}
    
Offset 134, 9 lines modifiedOffset 134, 9 lines modified
134 ········:param·signal:·the·signal·to·disconnect·from134 ········:param·signal:·the·signal·to·disconnect·from
135 ········:type·signal:·object135 ········:type·signal:·object
136 ········"""136 ········"""
137 ········kwargs.setdefault('sender',·self.sender)137 ········kwargs.setdefault('sender',·self.sender)
138 ········return·_signal.disconnect_all(signal,·**kwargs)138 ········return·_signal.disconnect_all(signal,·**kwargs)
  
139 ===============================================================================139 ===============================================================================
140 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.140 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
141 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.141 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
996 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/spiderloader.html
    
Offset 311, 15 lines modifiedOffset 311, 15 lines modified
311 ··········</div>311 ··········</div>
312 ··········<footer>312 ··········<footer>
  
313 ··<hr/>313 ··<hr/>
  
314 ··<div·role="contentinfo">314 ··<div·role="contentinfo">
315 ····<p>315 ····<p>
316 ········&#169;·Copyright·2008–2024,·Scrapy·developers.316 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
317 ······<span·class="lastupdated">317 ······<span·class="lastupdated">
318 ········Last·updated·on·May·20,·2022.318 ········Last·updated·on·May·20,·2022.
319 ······</span>319 ······</span>
  
320 ····</p>320 ····</p>
321 ··</div>321 ··</div>
322 ····322 ····
⊟
514 B
html2text {}
    
Offset 157, 9 lines modifiedOffset 157, 9 lines modified
157 [docs]····def·list(self):157 [docs]····def·list(self):
158 ········"""158 ········"""
159 ········Return·a·list·with·the·names·of·all·spiders·available·in·the·project.159 ········Return·a·list·with·the·names·of·all·spiders·available·in·the·project.
160 ········"""160 ········"""
161 ········return·list(self._spiders.keys())161 ········return·list(self._spiders.keys())
  
162 ===============================================================================162 ===============================================================================
163 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.163 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
164 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.164 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.03 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/spidermiddlewares/depth.html
    
Offset 277, 15 lines modifiedOffset 277, 15 lines modified
277 ··········</div>277 ··········</div>
278 ··········<footer>278 ··········<footer>
  
279 ··<hr/>279 ··<hr/>
  
280 ··<div·role="contentinfo">280 ··<div·role="contentinfo">
281 ····<p>281 ····<p>
282 ········&#169;·Copyright·2008–2024,·Scrapy·developers.282 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
283 ······<span·class="lastupdated">283 ······<span·class="lastupdated">
284 ········Last·updated·on·May·20,·2022.284 ········Last·updated·on·May·20,·2022.
285 ······</span>285 ······</span>
  
286 ····</p>286 ····</p>
287 ··</div>287 ··</div>
288 ····288 ····
⊟
551 B
html2text {}
    
Offset 120, 9 lines modifiedOffset 120, 9 lines modified
120 ············response.meta['depth']·=·0120 ············response.meta['depth']·=·0
121 ············if·self.verbose_stats:121 ············if·self.verbose_stats:
122 ················self.stats.inc_value('request_depth_count/0',·spider=spider)122 ················self.stats.inc_value('request_depth_count/0',·spider=spider)
  
123 ········return·(r·for·r·in·result·or·()·if·_filter(r))123 ········return·(r·for·r·in·result·or·()·if·_filter(r))
  
124 ===============================================================================124 ===============================================================================
125 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.125 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
126 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.126 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.03 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/spidermiddlewares/httperror.html
    
Offset 276, 15 lines modifiedOffset 276, 15 lines modified
276 ··········</div>276 ··········</div>
277 ··········<footer>277 ··········<footer>
  
278 ··<hr/>278 ··<hr/>
  
279 ··<div·role="contentinfo">279 ··<div·role="contentinfo">
280 ····<p>280 ····<p>
281 ········&#169;·Copyright·2008–2024,·Scrapy·developers.281 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
282 ······<span·class="lastupdated">282 ······<span·class="lastupdated">
283 ········Last·updated·on·May·20,·2022.283 ········Last·updated·on·May·20,·2022.
284 ······</span>284 ······</span>
  
285 ····</p>285 ····</p>
286 ··</div>286 ··</div>
287 ····287 ····
⊟
544 B
html2text {}
    
Offset 122, 9 lines modifiedOffset 122, 9 lines modified
122 ················"Ignoring·response·%(response)r:·HTTP·status·code·is·not122 ················"Ignoring·response·%(response)r:·HTTP·status·code·is·not
123 handled·or·not·allowed",123 handled·or·not·allowed",
124 ················{'response':·response},·extra={'spider':·spider},124 ················{'response':·response},·extra={'spider':·spider},
125 ············)125 ············)
126 ············return·[]126 ············return·[]
  
127 ===============================================================================127 ===============================================================================
128 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.128 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
129 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.129 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
900 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/spidermiddlewares/offsite.html
    
Offset 304, 15 lines modifiedOffset 304, 15 lines modified
304 ··········</div>304 ··········</div>
305 ··········<footer>305 ··········<footer>
  
306 ··<hr/>306 ··<hr/>
  
307 ··<div·role="contentinfo">307 ··<div·role="contentinfo">
308 ····<p>308 ····<p>
309 ········&#169;·Copyright·2008–2024,·Scrapy·developers.309 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
310 ······<span·class="lastupdated">310 ······<span·class="lastupdated">
311 ········Last·updated·on·May·20,·2022.311 ········Last·updated·on·May·20,·2022.
312 ······</span>312 ······</span>
  
313 ····</p>313 ····</p>
314 ··</div>314 ··</div>
315 ····315 ····
⊟
392 B
html2text {}
    
Offset 151, 9 lines modifiedOffset 151, 9 lines modified
151 ····pass151 ····pass
  
  
152 class·PortWarning(Warning):152 class·PortWarning(Warning):
153 ····pass153 ····pass
  
154 ===============================================================================154 ===============================================================================
155 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.155 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
156 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.156 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.08 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/spidermiddlewares/referer.html
    
Offset 580, 15 lines modifiedOffset 580, 15 lines modified
580 ··········</div>580 ··········</div>
581 ··········<footer>581 ··········<footer>
  
582 ··<hr/>582 ··<hr/>
  
583 ··<div·role="contentinfo">583 ··<div·role="contentinfo">
584 ····<p>584 ····<p>
585 ········&#169;·Copyright·2008–2024,·Scrapy·developers.585 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
586 ······<span·class="lastupdated">586 ······<span·class="lastupdated">
587 ········Last·updated·on·May·20,·2022.587 ········Last·updated·on·May·20,·2022.
588 ······</span>588 ······</span>
  
589 ····</p>589 ····</p>
590 ··</div>590 ··</div>
591 ····591 ····
⊟
598 B
html2text {}
    
Offset 456, 9 lines modifiedOffset 456, 9 lines modified
456 ················if·policy_referrer·!=·request_referrer:456 ················if·policy_referrer·!=·request_referrer:
457 ····················if·policy_referrer·is·None:457 ····················if·policy_referrer·is·None:
458 ························request.headers.pop('Referer')458 ························request.headers.pop('Referer')
459 ····················else:459 ····················else:
460 ························request.headers['Referer']·=·policy_referrer460 ························request.headers['Referer']·=·policy_referrer
  
461 ===============================================================================461 ===============================================================================
462 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.462 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
463 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.463 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
987 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/spidermiddlewares/urllength.html
    
Offset 256, 15 lines modifiedOffset 256, 15 lines modified
256 ··········</div>256 ··········</div>
257 ··········<footer>257 ··········<footer>
  
258 ··<hr/>258 ··<hr/>
  
259 ··<div·role="contentinfo">259 ··<div·role="contentinfo">
260 ····<p>260 ····<p>
261 ········&#169;·Copyright·2008–2024,·Scrapy·developers.261 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
262 ······<span·class="lastupdated">262 ······<span·class="lastupdated">
263 ········Last·updated·on·May·20,·2022.263 ········Last·updated·on·May·20,·2022.
264 ······</span>264 ······</span>
  
265 ····</p>265 ····</p>
266 ··</div>266 ··</div>
267 ····267 ····
⊟
475 B
html2text {}
    
Offset 101, 9 lines modifiedOffset 101, 9 lines modified
101 ················return·False101 ················return·False
102 ············else:102 ············else:
103 ················return·True103 ················return·True
  
104 ········return·(r·for·r·in·result·or·()·if·_filter(r))104 ········return·(r·for·r·in·result·or·()·if·_filter(r))
  
105 ===============================================================================105 ===============================================================================
106 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.106 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
107 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.107 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.03 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/spiders/crawl.html
    
Offset 360, 15 lines modifiedOffset 360, 15 lines modified
360 ··········</div>360 ··········</div>
361 ··········<footer>361 ··········<footer>
  
362 ··<hr/>362 ··<hr/>
  
363 ··<div·role="contentinfo">363 ··<div·role="contentinfo">
364 ····<p>364 ····<p>
365 ········&#169;·Copyright·2008–2024,·Scrapy·developers.365 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
366 ······<span·class="lastupdated">366 ······<span·class="lastupdated">
367 ········Last·updated·on·May·20,·2022.367 ········Last·updated·on·May·20,·2022.
368 ······</span>368 ······</span>
  
369 ····</p>369 ····</p>
370 ··</div>370 ··</div>
371 ····371 ····
⊟
574 B
html2text {}
    
Offset 206, 9 lines modifiedOffset 206, 9 lines modified
206 ····def·from_crawler(cls,·crawler,·*args,·**kwargs):206 ····def·from_crawler(cls,·crawler,·*args,·**kwargs):
207 ········spider·=·super().from_crawler(crawler,·*args,·**kwargs)207 ········spider·=·super().from_crawler(crawler,·*args,·**kwargs)
208 ········spider._follow_links·=·crawler.settings.getbool208 ········spider._follow_links·=·crawler.settings.getbool
209 ('CRAWLSPIDER_FOLLOW_LINKS',·True)209 ('CRAWLSPIDER_FOLLOW_LINKS',·True)
210 ········return·spider210 ········return·spider
  
211 ===============================================================================211 ===============================================================================
212 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.212 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
213 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.213 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.04 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/spiders/feed.html
    
Offset 356, 15 lines modifiedOffset 356, 15 lines modified
356 ··········</div>356 ··········</div>
357 ··········<footer>357 ··········<footer>
  
358 ··<hr/>358 ··<hr/>
  
359 ··<div·role="contentinfo">359 ··<div·role="contentinfo">
360 ····<p>360 ····<p>
361 ········&#169;·Copyright·2008–2024,·Scrapy·developers.361 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
362 ······<span·class="lastupdated">362 ······<span·class="lastupdated">
363 ········Last·updated·on·May·20,·2022.363 ········Last·updated·on·May·20,·2022.
364 ······</span>364 ······</span>
  
365 ····</p>365 ····</p>
366 ··</div>366 ··</div>
367 ····367 ····
⊟
578 B
html2text {}
    
Offset 212, 9 lines modifiedOffset 212, 9 lines modified
212 ········if·not·hasattr(self,·'parse_row'):212 ········if·not·hasattr(self,·'parse_row'):
213 ············raise·NotConfigured('You·must·define·parse_row·method·in·order·to213 ············raise·NotConfigured('You·must·define·parse_row·method·in·order·to
214 scrape·this·CSV·feed')214 scrape·this·CSV·feed')
215 ········response·=·self.adapt_response(response)215 ········response·=·self.adapt_response(response)
216 ········return·self.parse_rows(response)216 ········return·self.parse_rows(response)
  
217 ===============================================================================217 ===============================================================================
218 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.218 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
219 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.219 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
999 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/spiders/sitemap.html
    
Offset 320, 15 lines modifiedOffset 320, 15 lines modified
320 ··········</div>320 ··········</div>
321 ··········<footer>321 ··········<footer>
  
322 ··<hr/>322 ··<hr/>
  
323 ··<div·role="contentinfo">323 ··<div·role="contentinfo">
324 ····<p>324 ····<p>
325 ········&#169;·Copyright·2008–2024,·Scrapy·developers.325 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
326 ······<span·class="lastupdated">326 ······<span·class="lastupdated">
327 ········Last·updated·on·May·20,·2022.327 ········Last·updated·on·May·20,·2022.
328 ······</span>328 ······</span>
  
329 ····</p>329 ····</p>
330 ··</div>330 ··</div>
331 ····331 ····
⊟
511 B
html2text {}
    
Offset 165, 9 lines modifiedOffset 165, 9 lines modified
165 ········yield·d['loc']165 ········yield·d['loc']
  
166 ········#·Also·consider·alternate·URLs·(xhtml:link·rel="alternate")166 ········#·Also·consider·alternate·URLs·(xhtml:link·rel="alternate")
167 ········if·alt·and·'alternate'·in·d:167 ········if·alt·and·'alternate'·in·d:
168 ············yield·from·d['alternate']168 ············yield·from·d['alternate']
  
169 ===============================================================================169 ===============================================================================
170 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.170 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
171 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.171 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
998 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/spiders.html
    
Offset 337, 15 lines modifiedOffset 337, 15 lines modified
337 ··········</div>337 ··········</div>
338 ··········<footer>338 ··········<footer>
  
339 ··<hr/>339 ··<hr/>
  
340 ··<div·role="contentinfo">340 ··<div·role="contentinfo">
341 ····<p>341 ····<p>
342 ········&#169;·Copyright·2008–2024,·Scrapy·developers.342 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
343 ······<span·class="lastupdated">343 ······<span·class="lastupdated">
344 ········Last·updated·on·May·20,·2022.344 ········Last·updated·on·May·20,·2022.
345 ······</span>345 ······</span>
  
346 ····</p>346 ····</p>
347 ··</div>347 ··</div>
348 ····348 ····
⊟
526 B
html2text {}
    
Offset 186, 9 lines modifiedOffset 186, 9 lines modified
  
186 #·Top-level·imports186 #·Top-level·imports
187 from·scrapy.spiders.crawl·import·CrawlSpider,·Rule187 from·scrapy.spiders.crawl·import·CrawlSpider,·Rule
188 from·scrapy.spiders.feed·import·XMLFeedSpider,·CSVFeedSpider188 from·scrapy.spiders.feed·import·XMLFeedSpider,·CSVFeedSpider
189 from·scrapy.spiders.sitemap·import·SitemapSpider189 from·scrapy.spiders.sitemap·import·SitemapSpider
  
190 ===============================================================================190 ===============================================================================
191 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.191 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
192 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.192 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
959 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/statscollectors.html
    
Offset 301, 15 lines modifiedOffset 301, 15 lines modified
301 ··········</div>301 ··········</div>
302 ··········<footer>302 ··········<footer>
  
303 ··<hr/>303 ··<hr/>
  
304 ··<div·role="contentinfo">304 ··<div·role="contentinfo">
305 ····<p>305 ····<p>
306 ········&#169;·Copyright·2008–2024,·Scrapy·developers.306 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
307 ······<span·class="lastupdated">307 ······<span·class="lastupdated">
308 ········Last·updated·on·May·20,·2022.308 ········Last·updated·on·May·20,·2022.
309 ······</span>309 ······</span>
  
310 ····</p>310 ····</p>
311 ··</div>311 ··</div>
312 ····312 ····
⊟
471 B
html2text {}
    
Offset 157, 9 lines modifiedOffset 157, 9 lines modified
157 ····def·max_value(self,·key,·value,·spider=None):157 ····def·max_value(self,·key,·value,·spider=None):
158 ········pass158 ········pass
  
159 ····def·min_value(self,·key,·value,·spider=None):159 ····def·min_value(self,·key,·value,·spider=None):
160 ········pass160 ········pass
  
161 ===============================================================================161 ===============================================================================
162 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.162 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
163 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.163 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
998 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/utils/curl.html
    
Offset 319, 15 lines modifiedOffset 319, 15 lines modified
319 ··········</div>319 ··········</div>
320 ··········<footer>320 ··········<footer>
  
321 ··<hr/>321 ··<hr/>
  
322 ··<div·role="contentinfo">322 ··<div·role="contentinfo">
323 ····<p>323 ····<p>
324 ········&#169;·Copyright·2008–2024,·Scrapy·developers.324 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
325 ······<span·class="lastupdated">325 ······<span·class="lastupdated">
326 ········Last·updated·on·May·20,·2022.326 ········Last·updated·on·May·20,·2022.
327 ······</span>327 ······</span>
  
328 ····</p>328 ····</p>
329 ··</div>329 ··</div>
330 ····330 ····
⊟
520 B
html2text {}
    
Offset 162, 9 lines modifiedOffset 162, 9 lines modified
162 ············#·if·the·"data"·is·specified·but·the·"method"·is·not·specified,162 ············#·if·the·"data"·is·specified·but·the·"method"·is·not·specified,
163 ············#·the·default·method·is·'POST'163 ············#·the·default·method·is·'POST'
164 ············result['method']·=·'POST'164 ············result['method']·=·'POST'
  
165 ····return·result165 ····return·result
  
166 ===============================================================================166 ===============================================================================
167 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.167 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
168 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.168 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.01 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/utils/log.html
    
Offset 434, 15 lines modifiedOffset 434, 15 lines modified
434 ··········</div>434 ··········</div>
435 ··········<footer>435 ··········<footer>
  
436 ··<hr/>436 ··<hr/>
  
437 ··<div·role="contentinfo">437 ··<div·role="contentinfo">
438 ····<p>438 ····<p>
439 ········&#169;·Copyright·2008–2024,·Scrapy·developers.439 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
440 ······<span·class="lastupdated">440 ······<span·class="lastupdated">
441 ········Last·updated·on·May·20,·2022.441 ········Last·updated·on·May·20,·2022.
442 ······</span>442 ······</span>
  
443 ····</p>443 ····</p>
444 ··</div>444 ··</div>
445 ····445 ····
⊟
555 B
html2text {}
    
Offset 281, 9 lines modifiedOffset 281, 9 lines modified
281 ····#·NOTE:·This·also·handles·'args'·being·an·empty·dict,·that·case·doesn't281 ····#·NOTE:·This·also·handles·'args'·being·an·empty·dict,·that·case·doesn't
282 ····#·play·well·in·logger.log·calls282 ····#·play·well·in·logger.log·calls
283 ····args·=·logkws·if·not·logkws.get('args')·else·logkws['args']283 ····args·=·logkws·if·not·logkws.get('args')·else·logkws['args']
  
284 ····return·(level,·message,·args)284 ····return·(level,·message,·args)
  
285 ===============================================================================285 ===============================================================================
286 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.286 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
287 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.287 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
977 B
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/utils/reactor.html
    
Offset 309, 15 lines modifiedOffset 309, 15 lines modified
309 ··········</div>309 ··········</div>
310 ··········<footer>310 ··········<footer>
  
311 ··<hr/>311 ··<hr/>
  
312 ··<div·role="contentinfo">312 ··<div·role="contentinfo">
313 ····<p>313 ····<p>
314 ········&#169;·Copyright·2008–2024,·Scrapy·developers.314 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
315 ······<span·class="lastupdated">315 ······<span·class="lastupdated">
316 ········Last·updated·on·May·20,·2022.316 ········Last·updated·on·May·20,·2022.
317 ······</span>317 ······</span>
  
318 ····</p>318 ····</p>
319 ··</div>319 ··</div>
320 ····320 ····
⊟
493 B
html2text {}
    
Offset 153, 9 lines modifiedOffset 153, 9 lines modified
  
  
153 def·is_asyncio_reactor_installed():153 def·is_asyncio_reactor_installed():
154 ····from·twisted.internet·import·reactor154 ····from·twisted.internet·import·reactor
155 ····return·isinstance(reactor,·asyncioreactor.AsyncioSelectorReactor)155 ····return·isinstance(reactor,·asyncioreactor.AsyncioSelectorReactor)
  
156 ===============================================================================156 ===============================================================================
157 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.157 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
158 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.158 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.02 KB
./usr/share/doc/python-scrapy-doc/html/_modules/scrapy/utils/trackref.html
    
Offset 285, 15 lines modifiedOffset 285, 15 lines modified
285 ··········</div>285 ··········</div>
286 ··········<footer>286 ··········<footer>
  
287 ··<hr/>287 ··<hr/>
  
288 ··<div·role="contentinfo">288 ··<div·role="contentinfo">
289 ····<p>289 ····<p>
290 ········&#169;·Copyright·2008–2024,·Scrapy·developers.290 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
291 ······<span·class="lastupdated">291 ······<span·class="lastupdated">
292 ········Last·updated·on·May·20,·2022.292 ········Last·updated·on·May·20,·2022.
293 ······</span>293 ······</span>
  
294 ····</p>294 ····</p>
295 ··</div>295 ··</div>
296 ····296 ····
⊟
559 B
html2text {}
    
Offset 132, 9 lines modifiedOffset 132, 9 lines modified
132 [docs]def·iter_all(class_name):132 [docs]def·iter_all(class_name):
133 ····"""Iterate·over·all·objects·of·the·same·class·by·its·class·name"""133 ····"""Iterate·over·all·objects·of·the·same·class·by·its·class·name"""
134 ····for·cls,·wdict·in·live_refs.items():134 ····for·cls,·wdict·in·live_refs.items():
135 ········if·cls.__name__·==·class_name:135 ········if·cls.__name__·==·class_name:
136 ············return·wdict.keys()136 ············return·wdict.keys()
  
137 ===============================================================================137 ===============================================================================
138 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.138 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
139 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.139 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.12 KB
./usr/share/doc/python-scrapy-doc/html/contributing.html
    
Offset 466, 15 lines modifiedOffset 466, 15 lines modified
466 ········<a·href="news.html"·class="btn·btn-neutral·float-left"·title="Release·notes"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>466 ········<a·href="news.html"·class="btn·btn-neutral·float-left"·title="Release·notes"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
467 ····</div>467 ····</div>
  
468 ··<hr/>468 ··<hr/>
  
469 ··<div·role="contentinfo">469 ··<div·role="contentinfo">
470 ····<p>470 ····<p>
471 ········&#169;·Copyright·2008–2024,·Scrapy·developers.471 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
472 ······<span·class="lastupdated">472 ······<span·class="lastupdated">
473 ········Last·updated·on·May·20,·2022.473 ········Last·updated·on·May·20,·2022.
474 ······</span>474 ······</span>
  
475 ····</p>475 ····</p>
476 ··</div>476 ··</div>
477 ····477 ····
⊟
535 B
html2text {}
    
Offset 243, 9 lines modifiedOffset 243, 9 lines modified
243 name·typically·resembles·the·full·path·of·the·module·they’re·testing.·For243 name·typically·resembles·the·full·path·of·the·module·they’re·testing.·For
244 example,·the·item·loaders·code·is·in:244 example,·the·item·loaders·code·is·in:
245 scrapy.loader245 scrapy.loader
246 And·their·unit-tests·are·in:246 And·their·unit-tests·are·in:
247 tests/test_loader.py247 tests/test_loader.py
248 Next·Previous248 Next·Previous
249 ===============================================================================249 ===============================================================================
250 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.250 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
251 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.251 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.3 KB
./usr/share/doc/python-scrapy-doc/html/faq.html
    
Offset 554, 15 lines modifiedOffset 554, 15 lines modified
554 ········<a·href="topics/webservice.html"·class="btn·btn-neutral·float-left"·title="Web·Service"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>554 ········<a·href="topics/webservice.html"·class="btn·btn-neutral·float-left"·title="Web·Service"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
555 ····</div>555 ····</div>
  
556 ··<hr/>556 ··<hr/>
  
557 ··<div·role="contentinfo">557 ··<div·role="contentinfo">
558 ····<p>558 ····<p>
559 ········&#169;·Copyright·2008–2024,·Scrapy·developers.559 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
560 ······<span·class="lastupdated">560 ······<span·class="lastupdated">
561 ········Last·updated·on·May·20,·2022.561 ········Last·updated·on·May·20,·2022.
562 ······</span>562 ······</span>
  
563 ····</p>563 ····</p>
564 ··</div>564 ··</div>
565 ····565 ····
⊟
723 B
html2text {}
    
Offset 308, 9 lines modifiedOffset 308, 9 lines modified
308 response.·For·instance,·if·you·only·need·the·first·part·of·a·large·response·and308 response.·For·instance,·if·you·only·need·the·first·part·of·a·large·response·and
309 you·would·like·to·save·resources·by·avoiding·the·download·of·the·whole·body.·In309 you·would·like·to·save·resources·by·avoiding·the·download·of·the·whole·body.·In
310 that·case,·you·could·attach·a·handler·to·the·bytes_received·signal·and·raise·a310 that·case,·you·could·attach·a·handler·to·the·bytes_received·signal·and·raise·a
311 StopDownload·exception.·Please·refer·to·the·Stopping_the_download_of_a_Response311 StopDownload·exception.·Please·refer·to·the·Stopping_the_download_of_a_Response
312 topic·for·additional·information·and·examples.312 topic·for·additional·information·and·examples.
313 Next·Previous313 Next·Previous
314 ===============================================================================314 ===============================================================================
315 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.315 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
316 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.316 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.06 KB
./usr/share/doc/python-scrapy-doc/html/genindex.html
    
Offset 3717, 15 lines modifiedOffset 3717, 15 lines modified
3717 ··········</div>3717 ··········</div>
3718 ··········<footer>3718 ··········<footer>
  
3719 ··<hr/>3719 ··<hr/>
  
3720 ··<div·role="contentinfo">3720 ··<div·role="contentinfo">
3721 ····<p>3721 ····<p>
3722 ········&#169;·Copyright·2008–2024,·Scrapy·developers.3722 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
3723 ······<span·class="lastupdated">3723 ······<span·class="lastupdated">
3724 ········Last·updated·on·May·20,·2022.3724 ········Last·updated·on·May·20,·2022.
3725 ······</span>3725 ······</span>
  
3726 ····</p>3726 ····</p>
3727 ··</div>3727 ··</div>
3728 ····3728 ····
⊟
638 B
html2text {}
    
Offset 990, 9 lines modifiedOffset 990, 9 lines modified
990 ······scrapy.spiders)·················method)990 ······scrapy.spiders)·················method)
991 ····*·XmlItemExporter_(class_in···········o·(scrapy.selector.Selector991 ····*·XmlItemExporter_(class_in···········o·(scrapy.selector.Selector
992 ······scrapy.exporters)·····················method)992 ······scrapy.exporters)·····················method)
993 ····*·XmlResponse_(class_in···············o·(scrapy.selector.SelectorList993 ····*·XmlResponse_(class_in···············o·(scrapy.selector.SelectorList
994 ······scrapy.http)··························method)994 ······scrapy.http)··························method)
  
995 ===============================================================================995 ===============================================================================
996 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.996 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
997 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.997 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.15 KB
./usr/share/doc/python-scrapy-doc/html/index.html
    
Offset 378, 15 lines modifiedOffset 378, 15 lines modified
378 ········<a·href="intro/overview.html"·class="btn·btn-neutral·float-right"·title="Scrapy·at·a·glance"·accesskey="n"·rel="next">Next·<span·class="fa·fa-arrow-circle-right"·aria-hidden="true"></span></a>378 ········<a·href="intro/overview.html"·class="btn·btn-neutral·float-right"·title="Scrapy·at·a·glance"·accesskey="n"·rel="next">Next·<span·class="fa·fa-arrow-circle-right"·aria-hidden="true"></span></a>
379 ····</div>379 ····</div>
  
380 ··<hr/>380 ··<hr/>
  
381 ··<div·role="contentinfo">381 ··<div·role="contentinfo">
382 ····<p>382 ····<p>
383 ········&#169;·Copyright·2008–2024,·Scrapy·developers.383 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
384 ······<span·class="lastupdated">384 ······<span·class="lastupdated">
385 ········Last·updated·on·May·20,·2022.385 ········Last·updated·on·May·20,·2022.
386 ······</span>386 ······</span>
  
387 ····</p>387 ····</p>
388 ··</div>388 ··</div>
389 ····389 ····
⊟
565 B
html2text {}
    
Offset 173, 9 lines modifiedOffset 173, 9 lines modified
173 ······See·what·has·changed·in·recent·Scrapy·versions.173 ······See·what·has·changed·in·recent·Scrapy·versions.
174 ··Contributing_to_Scrapy174 ··Contributing_to_Scrapy
175 ······Learn·how·to·contribute·to·the·Scrapy·project.175 ······Learn·how·to·contribute·to·the·Scrapy·project.
176 ··Versioning_and_API_stability176 ··Versioning_and_API_stability
177 ······Understand·Scrapy·versioning·and·API·stability.177 ······Understand·Scrapy·versioning·and·API·stability.
178 Next178 Next
179 ===============================================================================179 ===============================================================================
180 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.180 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
181 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.181 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.23 KB
./usr/share/doc/python-scrapy-doc/html/intro/examples.html
    
Offset 239, 15 lines modifiedOffset 239, 15 lines modified
239 ········<a·href="tutorial.html"·class="btn·btn-neutral·float-left"·title="Scrapy·Tutorial"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>239 ········<a·href="tutorial.html"·class="btn·btn-neutral·float-left"·title="Scrapy·Tutorial"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
240 ····</div>240 ····</div>
  
241 ··<hr/>241 ··<hr/>
  
242 ··<div·role="contentinfo">242 ··<div·role="contentinfo">
243 ····<p>243 ····<p>
244 ········&#169;·Copyright·2008–2024,·Scrapy·developers.244 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
245 ······<span·class="lastupdated">245 ······<span·class="lastupdated">
246 ········Last·updated·on·May·20,·2022.246 ········Last·updated·on·May·20,·2022.
247 ······</span>247 ······</span>
  
248 ····</p>248 ····</p>
249 ··</div>249 ··</div>
250 ····250 ····
⊟
639 B
html2text {}
    
Offset 73, 9 lines modifiedOffset 73, 9 lines modified
73 expressions.73 expressions.
74 The·quotesbot·project·is·available·at:·https://github.com/scrapy/quotesbot.·You74 The·quotesbot·project·is·available·at:·https://github.com/scrapy/quotesbot.·You
75 can·find·more·information·about·it·in·the·project’s·README.75 can·find·more·information·about·it·in·the·project’s·README.
76 If·you’re·familiar·with·git,·you·can·checkout·the·code.·Otherwise·you·can76 If·you’re·familiar·with·git,·you·can·checkout·the·code.·Otherwise·you·can
77 download·the·project·as·a·zip·file·by·clicking·here.77 download·the·project·as·a·zip·file·by·clicking·here.
78 Next·Previous78 Next·Previous
79 ===============================================================================79 ===============================================================================
80 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.80 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
81 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.81 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.2 KB
./usr/share/doc/python-scrapy-doc/html/intro/install.html
    
Offset 461, 15 lines modifiedOffset 461, 15 lines modified
461 ········<a·href="overview.html"·class="btn·btn-neutral·float-left"·title="Scrapy·at·a·glance"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>461 ········<a·href="overview.html"·class="btn·btn-neutral·float-left"·title="Scrapy·at·a·glance"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
462 ····</div>462 ····</div>
  
463 ··<hr/>463 ··<hr/>
  
464 ··<div·role="contentinfo">464 ··<div·role="contentinfo">
465 ····<p>465 ····<p>
466 ········&#169;·Copyright·2008–2024,·Scrapy·developers.466 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
467 ······<span·class="lastupdated">467 ······<span·class="lastupdated">
468 ········Last·updated·on·May·20,·2022.468 ········Last·updated·on·May·20,·2022.
469 ······</span>469 ······</span>
  
470 ····</p>470 ····</p>
471 ··</div>471 ··</div>
472 ····472 ····
⊟
604 B
html2text {}
    
Offset 222, 9 lines modifiedOffset 222, 9 lines modified
222 has·a·version·of·pyOpenSSL·that·your·version·of·Twisted·does·not·support.222 has·a·version·of·pyOpenSSL·that·your·version·of·Twisted·does·not·support.
223 To·install·a·version·of·pyOpenSSL·that·your·version·of·Twisted·supports,223 To·install·a·version·of·pyOpenSSL·that·your·version·of·Twisted·supports,
224 reinstall·Twisted·with·the·tls·extra·option:224 reinstall·Twisted·with·the·tls·extra·option:
225 pip·install·twisted[tls]225 pip·install·twisted[tls]
226 For·details,·see·Issue_#2473.226 For·details,·see·Issue_#2473.
227 Next·Previous227 Next·Previous
228 ===============================================================================228 ===============================================================================
229 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.229 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
230 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.230 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.2 KB
./usr/share/doc/python-scrapy-doc/html/intro/overview.html
    
Offset 361, 15 lines modifiedOffset 361, 15 lines modified
361 ········<a·href="../index.html"·class="btn·btn-neutral·float-left"·title="Scrapy·2.4·documentation"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>361 ········<a·href="../index.html"·class="btn·btn-neutral·float-left"·title="Scrapy·2.4·documentation"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
362 ····</div>362 ····</div>
  
363 ··<hr/>363 ··<hr/>
  
364 ··<div·role="contentinfo">364 ··<div·role="contentinfo">
365 ····<p>365 ····<p>
366 ········&#169;·Copyright·2008–2024,·Scrapy·developers.366 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
367 ······<span·class="lastupdated">367 ······<span·class="lastupdated">
368 ········Last·updated·on·May·20,·2022.368 ········Last·updated·on·May·20,·2022.
369 ······</span>369 ······</span>
  
370 ····</p>370 ····</p>
371 ··</div>371 ··</div>
372 ····372 ····
⊟
592 B
html2text {}
    
Offset 172, 9 lines modifiedOffset 172, 9 lines modified
172 ······resolver,·and·much·more!172 ······resolver,·and·much·more!
173 *****·What’s·next?¶·*****173 *****·What’s·next?¶·*****
174 The·next·steps·for·you·are·to·install_Scrapy,·follow_through_the_tutorial·to174 The·next·steps·for·you·are·to·install_Scrapy,·follow_through_the_tutorial·to
175 learn·how·to·create·a·full-blown·Scrapy·project·and·join_the_community.·Thanks175 learn·how·to·create·a·full-blown·Scrapy·project·and·join_the_community.·Thanks
176 for·your·interest!176 for·your·interest!
177 Next·Previous177 Next·Previous
178 ===============================================================================178 ===============================================================================
179 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.179 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
180 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.180 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.27 KB
./usr/share/doc/python-scrapy-doc/html/intro/tutorial.html
    
Offset 912, 15 lines modifiedOffset 912, 15 lines modified
912 ········<a·href="install.html"·class="btn·btn-neutral·float-left"·title="Installation·guide"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>912 ········<a·href="install.html"·class="btn·btn-neutral·float-left"·title="Installation·guide"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
913 ····</div>913 ····</div>
  
914 ··<hr/>914 ··<hr/>
  
915 ··<div·role="contentinfo">915 ··<div·role="contentinfo">
916 ····<p>916 ····<p>
917 ········&#169;·Copyright·2008–2024,·Scrapy·developers.917 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
918 ······<span·class="lastupdated">918 ······<span·class="lastupdated">
919 ········Last·updated·on·May·20,·2022.919 ········Last·updated·on·May·20,·2022.
920 ······</span>920 ······</span>
  
921 ····</p>921 ····</p>
922 ··</div>922 ··</div>
923 ····923 ····
⊟
678 B
html2text {}
    
Offset 624, 9 lines modifiedOffset 624, 9 lines modified
624 chapter·for·a·quick·overview·of·the·most·important·ones.624 chapter·for·a·quick·overview·of·the·most·important·ones.
625 You·can·continue·from·the·section·Basic_concepts·to·know·more·about·the625 You·can·continue·from·the·section·Basic_concepts·to·know·more·about·the
626 command-line·tool,·spiders,·selectors·and·other·things·the·tutorial·hasn’t626 command-line·tool,·spiders,·selectors·and·other·things·the·tutorial·hasn’t
627 covered·like·modeling·the·scraped·data.·If·you·prefer·to·play·with·an·example627 covered·like·modeling·the·scraped·data.·If·you·prefer·to·play·with·an·example
628 project,·check·the·Examples·section.628 project,·check·the·Examples·section.
629 Next·Previous629 Next·Previous
630 ===============================================================================630 ===============================================================================
631 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.631 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
632 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.632 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.21 KB
./usr/share/doc/python-scrapy-doc/html/news.html
    
Offset 4518, 15 lines modifiedOffset 4518, 15 lines modified
4518 ········<a·href="topics/exporters.html"·class="btn·btn-neutral·float-left"·title="Item·Exporters"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>4518 ········<a·href="topics/exporters.html"·class="btn·btn-neutral·float-left"·title="Item·Exporters"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
4519 ····</div>4519 ····</div>
  
4520 ··<hr/>4520 ··<hr/>
  
4521 ··<div·role="contentinfo">4521 ··<div·role="contentinfo">
4522 ····<p>4522 ····<p>
4523 ········&#169;·Copyright·2008–2024,·Scrapy·developers.4523 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
4524 ······<span·class="lastupdated">4524 ······<span·class="lastupdated">
4525 ········Last·updated·on·May·20,·2022.4525 ········Last·updated·on·May·20,·2022.
4526 ······</span>4526 ······</span>
  
4527 ····</p>4527 ····</p>
4528 ··</div>4528 ··</div>
4529 ····4529 ····
⊟
629 B
html2text {}
    
Offset 3288, 9 lines modifiedOffset 3288, 9 lines modified
3288 ····*·Renamed·extension:·DelayedCloseDomain·to·SpiderCloseDelay·(r1861·|·#121)3288 ····*·Renamed·extension:·DelayedCloseDomain·to·SpiderCloseDelay·(r1861·|·#121)
3289 ····*·Removed·obsolete·scrapy.utils.markup.remove_escape_chars·function·-·use3289 ····*·Removed·obsolete·scrapy.utils.markup.remove_escape_chars·function·-·use
3290 ······scrapy.utils.markup.replace_escape_chars·instead·(r1865)3290 ······scrapy.utils.markup.replace_escape_chars·instead·(r1865)
3291 *****·Scrapy·0.7¶·*****3291 *****·Scrapy·0.7¶·*****
3292 First·release·of·Scrapy.3292 First·release·of·Scrapy.
3293 Next·Previous3293 Next·Previous
3294 ===============================================================================3294 ===============================================================================
3295 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.3295 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
3296 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.3296 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.1 KB
./usr/share/doc/python-scrapy-doc/html/py-modindex.html
    
Offset 502, 15 lines modifiedOffset 502, 15 lines modified
502 ··········</div>502 ··········</div>
503 ··········<footer>503 ··········<footer>
  
504 ··<hr/>504 ··<hr/>
  
505 ··<div·role="contentinfo">505 ··<div·role="contentinfo">
506 ····<p>506 ····<p>
507 ········&#169;·Copyright·2008–2024,·Scrapy·developers.507 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
508 ······<span·class="lastupdated">508 ······<span·class="lastupdated">
509 ········Last·updated·on·May·20,·2022.509 ········Last·updated·on·May·20,·2022.
510 ······</span>510 ······</span>
  
511 ····</p>511 ····</p>
512 ··</div>512 ··</div>
513 ····513 ····
⊟
674 B
html2text {}
    
Offset 128, 9 lines modifiedOffset 128, 9 lines modified
128 ·················································manager·and·spider·middleware128 ·················································manager·and·spider·middleware
129 ····   ·scrapy.statscollectors················Stats·Collectors129 ····   ·scrapy.statscollectors················Stats·Collectors
130 ····   ·scrapy.utils.log······················Logging·utils130 ····   ·scrapy.utils.log······················Logging·utils
131 ····   ·scrapy.utils.trackref·················Track·references·of·live131 ····   ·scrapy.utils.trackref·················Track·references·of·live
132 ·················································objects132 ·················································objects
  
133 ===============================================================================133 ===============================================================================
134 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.134 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
135 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.135 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
956 B
./usr/share/doc/python-scrapy-doc/html/search.html
    
Offset 230, 15 lines modifiedOffset 230, 15 lines modified
230 ··········</div>230 ··········</div>
231 ··········<footer>231 ··········<footer>
  
232 ··<hr/>232 ··<hr/>
  
233 ··<div·role="contentinfo">233 ··<div·role="contentinfo">
234 ····<p>234 ····<p>
235 ········&#169;·Copyright·2008–2024,·Scrapy·developers.235 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
236 ······<span·class="lastupdated">236 ······<span·class="lastupdated">
237 ········Last·updated·on·May·20,·2022.237 ········Last·updated·on·May·20,·2022.
238 ······</span>238 ······</span>
  
239 ····</p>239 ····</p>
240 ··</div>240 ··</div>
241 ····241 ····
⊟
518 B
html2text {}
    
Offset 61, 9 lines modifiedOffset 61, 9 lines modified
61 ····Scrapy61 ····Scrapy
62 ····*··»62 ····*··»
63 ····*·Search63 ····*·Search
64 ===============================================================================64 ===============================================================================
65 Please·activate·JavaScript·to·enable·the·search·functionality.65 Please·activate·JavaScript·to·enable·the·search·functionality.
  
66 ===============================================================================66 ===============================================================================
67 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.67 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
68 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.68 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.2 KB
./usr/share/doc/python-scrapy-doc/html/topics/api.html
    
Offset 994, 15 lines modifiedOffset 994, 15 lines modified
994 ········<a·href="extensions.html"·class="btn·btn-neutral·float-left"·title="Extensions"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>994 ········<a·href="extensions.html"·class="btn·btn-neutral·float-left"·title="Extensions"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
995 ····</div>995 ····</div>
  
996 ··<hr/>996 ··<hr/>
  
997 ··<div·role="contentinfo">997 ··<div·role="contentinfo">
998 ····<p>998 ····<p>
999 ········&#169;·Copyright·2008–2024,·Scrapy·developers.999 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
1000 ······<span·class="lastupdated">1000 ······<span·class="lastupdated">
1001 ········Last·updated·on·May·20,·2022.1001 ········Last·updated·on·May·20,·2022.
1002 ······</span>1002 ······</span>
  
1003 ····</p>1003 ····</p>
1004 ··</div>1004 ··</div>
1005 ····1005 ····
⊟
613 B
html2text {}
    
Offset 473, 9 lines modifiedOffset 473, 9 lines modified
473 ········open_spider(spider)[source]¶473 ········open_spider(spider)[source]¶
474 ············Open·the·given·spider·for·stats·collection.474 ············Open·the·given·spider·for·stats·collection.
475 ········close_spider(spider)[source]¶475 ········close_spider(spider)[source]¶
476 ············Close·the·given·spider.·After·this·is·called,·no·more·specific476 ············Close·the·given·spider.·After·this·is·called,·no·more·specific
477 ············stats·can·be·accessed·or·collected.477 ············stats·can·be·accessed·or·collected.
478 Next·Previous478 Next·Previous
479 ===============================================================================479 ===============================================================================
480 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.480 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
481 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.481 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.16 KB
./usr/share/doc/python-scrapy-doc/html/topics/architecture.html
    
Offset 370, 15 lines modifiedOffset 370, 15 lines modified
370 ········<a·href="asyncio.html"·class="btn·btn-neutral·float-left"·title="asyncio"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>370 ········<a·href="asyncio.html"·class="btn·btn-neutral·float-left"·title="asyncio"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
371 ····</div>371 ····</div>
  
372 ··<hr/>372 ··<hr/>
  
373 ··<div·role="contentinfo">373 ··<div·role="contentinfo">
374 ····<p>374 ····<p>
375 ········&#169;·Copyright·2008–2024,·Scrapy·developers.375 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
376 ······<span·class="lastupdated">376 ······<span·class="lastupdated">
377 ········Last·updated·on·May·20,·2022.377 ········Last·updated·on·May·20,·2022.
378 ······</span>378 ······</span>
  
379 ····</p>379 ····</p>
380 ··</div>380 ··</div>
381 ····381 ····
⊟
559 B
html2text {}
    
Offset 159, 9 lines modifiedOffset 159, 9 lines modified
159 For·more·information·about·asynchronous·programming·and·Twisted·see·these159 For·more·information·about·asynchronous·programming·and·Twisted·see·these
160 links:160 links:
161 ····*·twisted:core/howto/defer-intro161 ····*·twisted:core/howto/defer-intro
162 ····*·Twisted_-_hello,_asynchronous_programming162 ····*·Twisted_-_hello,_asynchronous_programming
163 ····*·Twisted_Introduction_-_Krondo163 ····*·Twisted_Introduction_-_Krondo
164 Next·Previous164 Next·Previous
165 ===============================================================================165 ===============================================================================
166 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.166 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
167 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.167 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.27 KB
./usr/share/doc/python-scrapy-doc/html/topics/asyncio.html
    
Offset 263, 15 lines modifiedOffset 263, 15 lines modified
263 ········<a·href="coroutines.html"·class="btn·btn-neutral·float-left"·title="Coroutines"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>263 ········<a·href="coroutines.html"·class="btn·btn-neutral·float-left"·title="Coroutines"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
264 ····</div>264 ····</div>
  
265 ··<hr/>265 ··<hr/>
  
266 ··<div·role="contentinfo">266 ··<div·role="contentinfo">
267 ····<p>267 ····<p>
268 ········&#169;·Copyright·2008–2024,·Scrapy·developers.268 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
269 ······<span·class="lastupdated">269 ······<span·class="lastupdated">
270 ········Last·updated·on·May·20,·2022.270 ········Last·updated·on·May·20,·2022.
271 ······</span>271 ······</span>
  
272 ····</p>272 ····</p>
273 ··</div>273 ··</div>
274 ····274 ····
⊟
679 B
html2text {}
    
Offset 83, 9 lines modifiedOffset 83, 9 lines modified
83 install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')83 install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')
84 *****·Using·custom·asyncio·loops¶·*****84 *****·Using·custom·asyncio·loops¶·*****
85 You·can·also·use·custom·asyncio·event·loops·with·the·asyncio·reactor.·Set·the85 You·can·also·use·custom·asyncio·event·loops·with·the·asyncio·reactor.·Set·the
86 ASYNCIO_EVENT_LOOP·setting·to·the·import·path·of·the·desired·event·loop·class86 ASYNCIO_EVENT_LOOP·setting·to·the·import·path·of·the·desired·event·loop·class
87 to·use·it·instead·of·the·default·asyncio·event·loop.87 to·use·it·instead·of·the·default·asyncio·event·loop.
88 Next·Previous88 Next·Previous
89 ===============================================================================89 ===============================================================================
90 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.90 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
91 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.91 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.17 KB
./usr/share/doc/python-scrapy-doc/html/topics/autothrottle.html
    
Offset 369, 15 lines modifiedOffset 369, 15 lines modified
369 ········<a·href="deploy.html"·class="btn·btn-neutral·float-left"·title="Deploying·Spiders"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>369 ········<a·href="deploy.html"·class="btn·btn-neutral·float-left"·title="Deploying·Spiders"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
370 ····</div>370 ····</div>
  
371 ··<hr/>371 ··<hr/>
  
372 ··<div·role="contentinfo">372 ··<div·role="contentinfo">
373 ····<p>373 ····<p>
374 ········&#169;·Copyright·2008–2024,·Scrapy·developers.374 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
375 ······<span·class="lastupdated">375 ······<span·class="lastupdated">
376 ········Last·updated·on·May·20,·2022.376 ········Last·updated·on·May·20,·2022.
377 ······</span>377 ······</span>
  
378 ····</p>378 ····</p>
379 ··</div>379 ··</div>
380 ····380 ····
⊟
565 B
html2text {}
    
Offset 166, 9 lines modifiedOffset 166, 9 lines modified
166 ****·AUTOTHROTTLE_DEBUG¶·****166 ****·AUTOTHROTTLE_DEBUG¶·****
167 Default:·False167 Default:·False
168 Enable·AutoThrottle·debug·mode·which·will·display·stats·on·every·response168 Enable·AutoThrottle·debug·mode·which·will·display·stats·on·every·response
169 received,·so·you·can·see·how·the·throttling·parameters·are·being·adjusted·in169 received,·so·you·can·see·how·the·throttling·parameters·are·being·adjusted·in
170 real·time.170 real·time.
171 Next·Previous171 Next·Previous
172 ===============================================================================172 ===============================================================================
173 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.173 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
174 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.174 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.25 KB
./usr/share/doc/python-scrapy-doc/html/topics/benchmarking.html
    
Offset 306, 15 lines modifiedOffset 306, 15 lines modified
306 ········<a·href="autothrottle.html"·class="btn·btn-neutral·float-left"·title="AutoThrottle·extension"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>306 ········<a·href="autothrottle.html"·class="btn·btn-neutral·float-left"·title="AutoThrottle·extension"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
307 ····</div>307 ····</div>
  
308 ··<hr/>308 ··<hr/>
  
309 ··<div·role="contentinfo">309 ··<div·role="contentinfo">
310 ····<p>310 ····<p>
311 ········&#169;·Copyright·2008–2024,·Scrapy·developers.311 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
312 ······<span·class="lastupdated">312 ······<span·class="lastupdated">
313 ········Last·updated·on·May·20,·2022.313 ········Last·updated·on·May·20,·2022.
314 ······</span>314 ······</span>
  
315 ····</p>315 ····</p>
316 ··</div>316 ··</div>
317 ····317 ····
⊟
640 B
html2text {}
    
Offset 154, 9 lines modifiedOffset 154, 9 lines modified
154 follow·links,·any·custom·spider·you·write·will·probably·do·more·stuff·which154 follow·links,·any·custom·spider·you·write·will·probably·do·more·stuff·which
155 results·in·slower·crawl·rates.·How·slower·depends·on·how·much·your·spider·does155 results·in·slower·crawl·rates.·How·slower·depends·on·how·much·your·spider·does
156 and·how·well·it’s·written.156 and·how·well·it’s·written.
157 In·the·future,·more·cases·will·be·added·to·the·benchmarking·suite·to·cover157 In·the·future,·more·cases·will·be·added·to·the·benchmarking·suite·to·cover
158 other·common·scenarios.158 other·common·scenarios.
159 Next·Previous159 Next·Previous
160 ===============================================================================160 ===============================================================================
161 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.161 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
162 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.162 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.3 KB
./usr/share/doc/python-scrapy-doc/html/topics/broad-crawls.html
    
Offset 423, 15 lines modifiedOffset 423, 15 lines modified
423 ········<a·href="practices.html"·class="btn·btn-neutral·float-left"·title="Common·Practices"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>423 ········<a·href="practices.html"·class="btn·btn-neutral·float-left"·title="Common·Practices"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
424 ····</div>424 ····</div>
  
425 ··<hr/>425 ··<hr/>
  
426 ··<div·role="contentinfo">426 ··<div·role="contentinfo">
427 ····<p>427 ····<p>
428 ········&#169;·Copyright·2008–2024,·Scrapy·developers.428 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
429 ······<span·class="lastupdated">429 ······<span·class="lastupdated">
430 ········Last·updated·on·May·20,·2022.430 ········Last·updated·on·May·20,·2022.
431 ······</span>431 ······</span>
  
432 ····</p>432 ····</p>
433 ··</div>433 ··</div>
434 ····434 ····
⊟
700 B
html2text {}
    
Offset 206, 9 lines modifiedOffset 206, 9 lines modified
206 If·your·broad·crawl·shows·a·high·memory·usage,·in·addition·to·crawling_in_BFO206 If·your·broad·crawl·shows·a·high·memory·usage,·in·addition·to·crawling_in_BFO
207 order·and·lowering_concurrency·you·should·debug_your_memory_leaks.207 order·and·lowering_concurrency·you·should·debug_your_memory_leaks.
208 *****·Install·a·specific·Twisted·reactor¶·*****208 *****·Install·a·specific·Twisted·reactor¶·*****
209 If·the·crawl·is·exceeding·the·system’s·capabilities,·you·might·want·to·try209 If·the·crawl·is·exceeding·the·system’s·capabilities,·you·might·want·to·try
210 installing·a·specific·Twisted·reactor,·via·the·TWISTED_REACTOR·setting.210 installing·a·specific·Twisted·reactor,·via·the·TWISTED_REACTOR·setting.
211 Next·Previous211 Next·Previous
212 ===============================================================================212 ===============================================================================
213 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.213 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
214 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.214 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.05 KB
./usr/share/doc/python-scrapy-doc/html/topics/commands.html
    
Offset 777, 15 lines modifiedOffset 777, 15 lines modified
777 ········<a·href="../intro/examples.html"·class="btn·btn-neutral·float-left"·title="Examples"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>777 ········<a·href="../intro/examples.html"·class="btn·btn-neutral·float-left"·title="Examples"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
778 ····</div>778 ····</div>
  
779 ··<hr/>779 ··<hr/>
  
780 ··<div·role="contentinfo">780 ··<div·role="contentinfo">
781 ····<p>781 ····<p>
782 ········&#169;·Copyright·2008–2024,·Scrapy·developers.782 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
783 ······<span·class="lastupdated">783 ······<span·class="lastupdated">
784 ········Last·updated·on·May·20,·2022.784 ········Last·updated·on·May·20,·2022.
785 ······</span>785 ······</span>
  
786 ····</p>786 ····</p>
787 ··</div>787 ··</div>
788 ····788 ····
⊟
453 B
html2text {}
    
Offset 470, 9 lines modifiedOffset 470, 9 lines modified
470 ····'scrapy.commands':·[470 ····'scrapy.commands':·[
471 ······'my_command=my_scrapy_module.commands:MyCommand',471 ······'my_command=my_scrapy_module.commands:MyCommand',
472 ····],472 ····],
473 ··},473 ··},
474 ·)474 ·)
475 Next·Previous475 Next·Previous
476 ===============================================================================476 ===============================================================================
477 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.477 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
478 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.478 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.12 KB
./usr/share/doc/python-scrapy-doc/html/topics/contracts.html
    
Offset 390, 15 lines modifiedOffset 390, 15 lines modified
390 ········<a·href="debug.html"·class="btn·btn-neutral·float-left"·title="Debugging·Spiders"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>390 ········<a·href="debug.html"·class="btn·btn-neutral·float-left"·title="Debugging·Spiders"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
391 ····</div>391 ····</div>
  
392 ··<hr/>392 ··<hr/>
  
393 ··<div·role="contentinfo">393 ··<div·role="contentinfo">
394 ····<p>394 ····<p>
395 ········&#169;·Copyright·2008–2024,·Scrapy·developers.395 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
396 ······<span·class="lastupdated">396 ······<span·class="lastupdated">
397 ········Last·updated·on·May·20,·2022.397 ········Last·updated·on·May·20,·2022.
398 ······</span>398 ······</span>
  
399 ····</p>399 ····</p>
400 ··</div>400 ··</div>
401 ····401 ····
⊟
518 B
html2text {}
    
Offset 161, 9 lines modifiedOffset 161, 9 lines modified
161 ····name·=·'example'161 ····name·=·'example'
  
162 ····def·__init__(self):162 ····def·__init__(self):
163 ········if·os.environ.get('SCRAPY_CHECK'):163 ········if·os.environ.get('SCRAPY_CHECK'):
164 ············pass··#·Do·some·scraper·adjustments·when·a·check·is·running164 ············pass··#·Do·some·scraper·adjustments·when·a·check·is·running
165 Next·Previous165 Next·Previous
166 ===============================================================================166 ===============================================================================
167 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.167 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
168 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.168 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.26 KB
./usr/share/doc/python-scrapy-doc/html/topics/coroutines.html
    
Offset 331, 15 lines modifiedOffset 331, 15 lines modified
331 ········<a·href="jobs.html"·class="btn·btn-neutral·float-left"·title="Jobs:·pausing·and·resuming·crawls"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>331 ········<a·href="jobs.html"·class="btn·btn-neutral·float-left"·title="Jobs:·pausing·and·resuming·crawls"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
332 ····</div>332 ····</div>
  
333 ··<hr/>333 ··<hr/>
  
334 ··<div·role="contentinfo">334 ··<div·role="contentinfo">
335 ····<p>335 ····<p>
336 ········&#169;·Copyright·2008–2024,·Scrapy·developers.336 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
337 ······<span·class="lastupdated">337 ······<span·class="lastupdated">
338 ········Last·updated·on·May·20,·2022.338 ········Last·updated·on·May·20,·2022.
339 ······</span>339 ······</span>
  
340 ····</p>340 ····</p>
341 ··</div>341 ··</div>
342 ····342 ····
⊟
643 B
html2text {}
    
Offset 135, 9 lines modifiedOffset 135, 9 lines modified
135 ····*·storing·data·in·databases·(in·pipelines·and·middlewares);135 ····*·storing·data·in·databases·(in·pipelines·and·middlewares);
136 ····*·delaying·the·spider·initialization·until·some·external·event·(in·the136 ····*·delaying·the·spider·initialization·until·some·external·event·(in·the
137 ······spider_opened·handler);137 ······spider_opened·handler);
138 ····*·calling·asynchronous·Scrapy·methods·like·ExecutionEngine.download·(see138 ····*·calling·asynchronous·Scrapy·methods·like·ExecutionEngine.download·(see
139 ······the_screenshot_pipeline_example).139 ······the_screenshot_pipeline_example).
140 Next·Previous140 Next·Previous
141 ===============================================================================141 ===============================================================================
142 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.142 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
143 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.143 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.14 KB
./usr/share/doc/python-scrapy-doc/html/topics/debug.html
    
Offset 364, 15 lines modifiedOffset 364, 15 lines modified
364 ········<a·href="../faq.html"·class="btn·btn-neutral·float-left"·title="Frequently·Asked·Questions"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>364 ········<a·href="../faq.html"·class="btn·btn-neutral·float-left"·title="Frequently·Asked·Questions"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
365 ····</div>365 ····</div>
  
366 ··<hr/>366 ··<hr/>
  
367 ··<div·role="contentinfo">367 ··<div·role="contentinfo">
368 ····<p>368 ····<p>
369 ········&#169;·Copyright·2008–2024,·Scrapy·developers.369 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
370 ······<span·class="lastupdated">370 ······<span·class="lastupdated">
371 ········Last·updated·on·May·20,·2022.371 ········Last·updated·on·May·20,·2022.
372 ······</span>372 ······</span>
  
373 ····</p>373 ····</p>
374 ··</div>374 ··</div>
375 ····375 ····
⊟
543 B
html2text {}
    
Offset 175, 9 lines modifiedOffset 175, 9 lines modified
175 ········#·populate·more·`item`·fields175 ········#·populate·more·`item`·fields
176 ········return·item176 ········return·item
177 ····else:177 ····else:
178 ········self.logger.warning('No·item·received·for·%s',·response.url)178 ········self.logger.warning('No·item·received·for·%s',·response.url)
179 For·more·information,·check·the·Logging·section.179 For·more·information,·check·the·Logging·section.
180 Next·Previous180 Next·Previous
181 ===============================================================================181 ===============================================================================
182 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.182 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
183 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.183 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.28 KB
./usr/share/doc/python-scrapy-doc/html/topics/deploy.html
    
Offset 266, 15 lines modifiedOffset 266, 15 lines modified
266 ········<a·href="media-pipeline.html"·class="btn·btn-neutral·float-left"·title="Downloading·and·processing·files·and·images"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>266 ········<a·href="media-pipeline.html"·class="btn·btn-neutral·float-left"·title="Downloading·and·processing·files·and·images"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
267 ····</div>267 ····</div>
  
268 ··<hr/>268 ··<hr/>
  
269 ··<div·role="contentinfo">269 ··<div·role="contentinfo">
270 ····<p>270 ····<p>
271 ········&#169;·Copyright·2008–2024,·Scrapy·developers.271 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
272 ······<span·class="lastupdated">272 ······<span·class="lastupdated">
273 ········Last·updated·on·May·20,·2022.273 ········Last·updated·on·May·20,·2022.
274 ······</span>274 ······</span>
  
275 ····</p>275 ····</p>
276 ··</div>276 ··</div>
277 ····277 ····
⊟
660 B
html2text {}
    
Offset 92, 9 lines modifiedOffset 92, 9 lines modified
92 To·deploy·spiders·to·Scrapy·Cloud·you·can·use·the·shub·command·line·tool.92 To·deploy·spiders·to·Scrapy·Cloud·you·can·use·the·shub·command·line·tool.
93 Please·refer·to·the·Scrapy_Cloud_documentation·for·more·information.93 Please·refer·to·the·Scrapy_Cloud_documentation·for·more·information.
94 Scrapy·Cloud·is·compatible·with·Scrapyd·and·one·can·switch·between·them·as94 Scrapy·Cloud·is·compatible·with·Scrapyd·and·one·can·switch·between·them·as
95 needed·-·the·configuration·is·read·from·the·scrapy.cfg·file·just·like·scrapyd-95 needed·-·the·configuration·is·read·from·the·scrapy.cfg·file·just·like·scrapyd-
96 deploy.96 deploy.
97 Next·Previous97 Next·Previous
98 ===============================================================================98 ===============================================================================
99 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.99 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
100 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.100 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.29 KB
./usr/share/doc/python-scrapy-doc/html/topics/developer-tools.html
    
Offset 484, 15 lines modifiedOffset 484, 15 lines modified
484 ········<a·href="broad-crawls.html"·class="btn·btn-neutral·float-left"·title="Broad·Crawls"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>484 ········<a·href="broad-crawls.html"·class="btn·btn-neutral·float-left"·title="Broad·Crawls"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
485 ····</div>485 ····</div>
  
486 ··<hr/>486 ··<hr/>
  
487 ··<div·role="contentinfo">487 ··<div·role="contentinfo">
488 ····<p>488 ····<p>
489 ········&#169;·Copyright·2008–2024,·Scrapy·developers.489 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
490 ······<span·class="lastupdated">490 ······<span·class="lastupdated">
491 ········Last·updated·on·May·20,·2022.491 ········Last·updated·on·May·20,·2022.
492 ······</span>492 ······</span>
  
493 ····</p>493 ····</p>
494 ··</div>494 ··</div>
495 ····495 ····
⊟
686 B
html2text {}
    
Offset 280, 9 lines modifiedOffset 280, 9 lines modified
280 As·you·can·see,·with·a·few·inspections·in·theNetwork-tool·we·were·able·to280 As·you·can·see,·with·a·few·inspections·in·theNetwork-tool·we·were·able·to
281 easily·replicate·the·dynamic·requests·of·the·scrolling·functionality·of·the281 easily·replicate·the·dynamic·requests·of·the·scrolling·functionality·of·the
282 page.·Crawling·dynamic·pages·can·be·quite·daunting·and·pages·can·be·very282 page.·Crawling·dynamic·pages·can·be·quite·daunting·and·pages·can·be·very
283 complex,·but·it·(mostly)·boils·down·to·identifying·the·correct·request·and283 complex,·but·it·(mostly)·boils·down·to·identifying·the·correct·request·and
284 replicating·it·in·your·spider.284 replicating·it·in·your·spider.
285 Next·Previous285 Next·Previous
286 ===============================================================================286 ===============================================================================
287 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.287 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
288 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.288 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.01 KB
./usr/share/doc/python-scrapy-doc/html/topics/djangoitem.html
    
Offset 228, 15 lines modifiedOffset 228, 15 lines modified
228 ··········</div>228 ··········</div>
229 ··········<footer>229 ··········<footer>
  
230 ··<hr/>230 ··<hr/>
  
231 ··<div·role="contentinfo">231 ··<div·role="contentinfo">
232 ····<p>232 ····<p>
233 ········&#169;·Copyright·2008–2024,·Scrapy·developers.233 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
234 ······<span·class="lastupdated">234 ······<span·class="lastupdated">
235 ········Last·updated·on·May·20,·2022.235 ········Last·updated·on·May·20,·2022.
236 ······</span>236 ······</span>
  
237 ····</p>237 ····</p>
238 ··</div>238 ··</div>
239 ····239 ····
⊟
574 B
html2text {}
    
Offset 65, 9 lines modifiedOffset 65, 9 lines modified
65 ===============================================================================65 ===============================================================================
66 ******·DjangoItem¶·******66 ******·DjangoItem¶·******
67 DjangoItem·has·been·moved·into·a·separate·project.67 DjangoItem·has·been·moved·into·a·separate·project.
68 It·is·hosted·at:68 It·is·hosted·at:
69 ·····https://github.com/scrapy-plugins/scrapy-djangoitem69 ·····https://github.com/scrapy-plugins/scrapy-djangoitem
  
70 ===============================================================================70 ===============================================================================
71 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.71 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
72 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.72 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.18 KB
./usr/share/doc/python-scrapy-doc/html/topics/downloader-middleware.html
    
Offset 1270, 15 lines modifiedOffset 1270, 15 lines modified
1270 ········<a·href="architecture.html"·class="btn·btn-neutral·float-left"·title="Architecture·overview"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>1270 ········<a·href="architecture.html"·class="btn·btn-neutral·float-left"·title="Architecture·overview"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
1271 ····</div>1271 ····</div>
  
1272 ··<hr/>1272 ··<hr/>
  
1273 ··<div·role="contentinfo">1273 ··<div·role="contentinfo">
1274 ····<p>1274 ····<p>
1275 ········&#169;·Copyright·2008–2024,·Scrapy·developers.1275 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
1276 ······<span·class="lastupdated">1276 ······<span·class="lastupdated">
1277 ········Last·updated·on·May·20,·2022.1277 ········Last·updated·on·May·20,·2022.
1278 ······</span>1278 ······</span>
  
1279 ····</p>1279 ····</p>
1280 ··</div>1280 ··</div>
1281 ····1281 ····
⊟
542 B
html2text {}
    
Offset 743, 9 lines modifiedOffset 743, 9 lines modified
743 Default:·True743 Default:·True
744 Whether·or·not·to·enable·the·HttpProxyMiddleware.744 Whether·or·not·to·enable·the·HttpProxyMiddleware.
745 **·HTTPPROXY_AUTH_ENCODING¶·**745 **·HTTPPROXY_AUTH_ENCODING¶·**
746 Default:·"latin-1"746 Default:·"latin-1"
747 The·default·encoding·for·proxy·authentication·on·HttpProxyMiddleware.747 The·default·encoding·for·proxy·authentication·on·HttpProxyMiddleware.
748 Next·Previous748 Next·Previous
749 ===============================================================================749 ===============================================================================
750 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.750 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
751 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.751 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.25 KB
./usr/share/doc/python-scrapy-doc/html/topics/dynamic-content.html
    
Offset 433, 15 lines modifiedOffset 433, 15 lines modified
433 ········<a·href="developer-tools.html"·class="btn·btn-neutral·float-left"·title="Using·your·browser’s·Developer·Tools·for·scraping"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>433 ········<a·href="developer-tools.html"·class="btn·btn-neutral·float-left"·title="Using·your·browser’s·Developer·Tools·for·scraping"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
434 ····</div>434 ····</div>
  
435 ··<hr/>435 ··<hr/>
  
436 ··<div·role="contentinfo">436 ··<div·role="contentinfo">
437 ····<p>437 ····<p>
438 ········&#169;·Copyright·2008–2024,·Scrapy·developers.438 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
439 ······<span·class="lastupdated">439 ······<span·class="lastupdated">
440 ········Last·updated·on·May·20,·2022.440 ········Last·updated·on·May·20,·2022.
441 ······</span>441 ······</span>
  
442 ····</p>442 ····</p>
443 ··</div>443 ··</div>
444 ····444 ····
⊟
604 B
html2text {}
    
Offset 214, 9 lines modifiedOffset 214, 9 lines modified
214 *****·Using·a·headless·browser¶·*****214 *****·Using·a·headless·browser¶·*****
215 A·headless_browser·is·a·special·web·browser·that·provides·an·API·for215 A·headless_browser·is·a·special·web·browser·that·provides·an·API·for
216 automation.216 automation.
217 The·easiest·way·to·use·a·headless·browser·with·Scrapy·is·to·use·Selenium,·along217 The·easiest·way·to·use·a·headless·browser·with·Scrapy·is·to·use·Selenium,·along
218 with·scrapy-selenium·for·seamless·integration.218 with·scrapy-selenium·for·seamless·integration.
219 Next·Previous219 Next·Previous
220 ===============================================================================220 ===============================================================================
221 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.221 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
222 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.222 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.17 KB
./usr/share/doc/python-scrapy-doc/html/topics/email.html
    
Offset 371, 15 lines modifiedOffset 371, 15 lines modified
371 ········<a·href="stats.html"·class="btn·btn-neutral·float-left"·title="Stats·Collection"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>371 ········<a·href="stats.html"·class="btn·btn-neutral·float-left"·title="Stats·Collection"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
372 ····</div>372 ····</div>
  
373 ··<hr/>373 ··<hr/>
  
374 ··<div·role="contentinfo">374 ··<div·role="contentinfo">
375 ····<p>375 ····<p>
376 ········&#169;·Copyright·2008–2024,·Scrapy·developers.376 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
377 ······<span·class="lastupdated">377 ······<span·class="lastupdated">
378 ········Last·updated·on·May·20,·2022.378 ········Last·updated·on·May·20,·2022.
379 ······</span>379 ······</span>
  
380 ····</p>380 ····</p>
381 ··</div>381 ··</div>
382 ····382 ····
⊟
582 B
html2text {}
    
Offset 160, 9 lines modifiedOffset 160, 9 lines modified
160 Enforce·using·STARTTLS.·STARTTLS·is·a·way·to·take·an·existing·insecure160 Enforce·using·STARTTLS.·STARTTLS·is·a·way·to·take·an·existing·insecure
161 connection,·and·upgrade·it·to·a·secure·connection·using·SSL/TLS.161 connection,·and·upgrade·it·to·a·secure·connection·using·SSL/TLS.
162 ****·MAIL_SSL¶·****162 ****·MAIL_SSL¶·****
163 Default:·False163 Default:·False
164 Enforce·connecting·using·an·SSL·encrypted·connection164 Enforce·connecting·using·an·SSL·encrypted·connection
165 Next·Previous165 Next·Previous
166 ===============================================================================166 ===============================================================================
167 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.167 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
168 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.168 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.21 KB
./usr/share/doc/python-scrapy-doc/html/topics/exceptions.html
    
Offset 361, 15 lines modifiedOffset 361, 15 lines modified
361 ········<a·href="settings.html"·class="btn·btn-neutral·float-left"·title="Settings"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>361 ········<a·href="settings.html"·class="btn·btn-neutral·float-left"·title="Settings"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
362 ····</div>362 ····</div>
  
363 ··<hr/>363 ··<hr/>
  
364 ··<div·role="contentinfo">364 ··<div·role="contentinfo">
365 ····<p>365 ····<p>
366 ········&#169;·Copyright·2008–2024,·Scrapy·developers.366 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
367 ······<span·class="lastupdated">367 ······<span·class="lastupdated">
368 ········Last·updated·on·May·20,·2022.368 ········Last·updated·on·May·20,·2022.
369 ······</span>369 ······</span>
  
370 ····</p>370 ····</p>
371 ··</div>371 ··</div>
372 ····372 ····
⊟
618 B
html2text {}
    
Offset 131, 9 lines modifiedOffset 131, 9 lines modified
131 Note131 Note
132 fail·is·a·keyword-only·parameter,·i.e.·raising·StopDownload(False)·or132 fail·is·a·keyword-only·parameter,·i.e.·raising·StopDownload(False)·or
133 StopDownload(True)·will·raise·a·TypeError.133 StopDownload(True)·will·raise·a·TypeError.
134 See·the·documentation·for·the·bytes_received·signal·and·the·Stopping_the134 See·the·documentation·for·the·bytes_received·signal·and·the·Stopping_the
135 download_of_a_Response·topic·for·additional·information·and·examples.135 download_of_a_Response·topic·for·additional·information·and·examples.
136 Next·Previous136 Next·Previous
137 ===============================================================================137 ===============================================================================
138 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.138 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
139 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.139 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.23 KB
./usr/share/doc/python-scrapy-doc/html/topics/exporters.html
    
Offset 689, 15 lines modifiedOffset 689, 15 lines modified
689 ········<a·href="signals.html"·class="btn·btn-neutral·float-left"·title="Signals"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>689 ········<a·href="signals.html"·class="btn·btn-neutral·float-left"·title="Signals"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
690 ····</div>690 ····</div>
  
691 ··<hr/>691 ··<hr/>
  
692 ··<div·role="contentinfo">692 ··<div·role="contentinfo">
693 ····<p>693 ····<p>
694 ········&#169;·Copyright·2008–2024,·Scrapy·developers.694 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
695 ······<span·class="lastupdated">695 ······<span·class="lastupdated">
696 ········Last·updated·on·May·20,·2022.696 ········Last·updated·on·May·20,·2022.
697 ······</span>697 ······</span>
  
698 ····</p>698 ····</p>
699 ··</div>699 ··</div>
700 ····700 ····
⊟
639 B
html2text {}
    
Offset 379, 9 lines modifiedOffset 379, 9 lines modified
379 ······Exports·items·in·a·Python-specific·binary·format·(see·marshal).379 ······Exports·items·in·a·Python-specific·binary·format·(see·marshal).
380 ········Parameters380 ········Parameters
381 ············file·â€“·The·file-like·object·to·use·for·exporting·the·data.·Its381 ············file·â€“·The·file-like·object·to·use·for·exporting·the·data.·Its
382 ············write·method·should·accept·bytes·(a·disk·file·opened·in·binary382 ············write·method·should·accept·bytes·(a·disk·file·opened·in·binary
383 ············mode,·a·BytesIO·object,·etc)383 ············mode,·a·BytesIO·object,·etc)
384 Next·Previous384 Next·Previous
385 ===============================================================================385 ===============================================================================
386 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.386 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
387 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.387 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.25 KB
./usr/share/doc/python-scrapy-doc/html/topics/extensions.html
    
Offset 578, 15 lines modifiedOffset 578, 15 lines modified
578 ········<a·href="spider-middleware.html"·class="btn·btn-neutral·float-left"·title="Spider·Middleware"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>578 ········<a·href="spider-middleware.html"·class="btn·btn-neutral·float-left"·title="Spider·Middleware"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
579 ····</div>579 ····</div>
  
580 ··<hr/>580 ··<hr/>
  
581 ··<div·role="contentinfo">581 ··<div·role="contentinfo">
582 ····<p>582 ····<p>
583 ········&#169;·Copyright·2008–2024,·Scrapy·developers.583 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
584 ······<span·class="lastupdated">584 ······<span·class="lastupdated">
585 ········Last·updated·on·May·20,·2022.585 ········Last·updated·on·May·20,·2022.
586 ······</span>586 ······</span>
  
587 ····</p>587 ····</p>
588 ··</div>588 ··</div>
589 ····589 ····
⊟
641 B
html2text {}
    
Offset 292, 9 lines modifiedOffset 292, 9 lines modified
292 Invokes·a·Python·debugger·inside·a·running·Scrapy·process·when·a·SIGUSR2·signal292 Invokes·a·Python·debugger·inside·a·running·Scrapy·process·when·a·SIGUSR2·signal
293 is·received.·After·the·debugger·is·exited,·the·Scrapy·process·continues·running293 is·received.·After·the·debugger·is·exited,·the·Scrapy·process·continues·running
294 normally.294 normally.
295 For·more·info·see·Debugging_in_Python.295 For·more·info·see·Debugging_in_Python.
296 This·extension·only·works·on·POSIX-compliant·platforms·(i.e.·not·Windows).296 This·extension·only·works·on·POSIX-compliant·platforms·(i.e.·not·Windows).
297 Next·Previous297 Next·Previous
298 ===============================================================================298 ===============================================================================
299 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.299 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
300 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.300 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.21 KB
./usr/share/doc/python-scrapy-doc/html/topics/feed-exports.html
    
Offset 822, 15 lines modifiedOffset 822, 15 lines modified
822 ········<a·href="item-pipeline.html"·class="btn·btn-neutral·float-left"·title="Item·Pipeline"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>822 ········<a·href="item-pipeline.html"·class="btn·btn-neutral·float-left"·title="Item·Pipeline"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
823 ····</div>823 ····</div>
  
824 ··<hr/>824 ··<hr/>
  
825 ··<div·role="contentinfo">825 ··<div·role="contentinfo">
826 ····<p>826 ····<p>
827 ········&#169;·Copyright·2008–2024,·Scrapy·developers.827 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
828 ······<span·class="lastupdated">828 ······<span·class="lastupdated">
829 ········Last·updated·on·May·20,·2022.829 ········Last·updated·on·May·20,·2022.
830 ······</span>830 ······</span>
  
831 ····</p>831 ····</p>
832 ··</div>832 ··</div>
833 ····833 ····
⊟
604 B
html2text {}
    
Offset 421, 9 lines modifiedOffset 421, 9 lines modified
421 ···2.·Point·FEED_URI_PARAMS·to·that·function·in·your·settings:421 ···2.·Point·FEED_URI_PARAMS·to·that·function·in·your·settings:
422 ······#·myproject/settings.py422 ······#·myproject/settings.py
423 ······FEED_URI_PARAMS·=·'myproject.utils.uri_params'423 ······FEED_URI_PARAMS·=·'myproject.utils.uri_params'
424 ···3.·Use·%(spider_name)s·in·your·feed·URI:424 ···3.·Use·%(spider_name)s·in·your·feed·URI:
425 ······scrapy·crawl·<spider_name>·-o·"%(spider_name)s.jl"425 ······scrapy·crawl·<spider_name>·-o·"%(spider_name)s.jl"
426 Next·Previous426 Next·Previous
427 ===============================================================================427 ===============================================================================
428 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.428 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
429 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.429 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.23 KB
./usr/share/doc/python-scrapy-doc/html/topics/item-pipeline.html
    
Offset 491, 15 lines modifiedOffset 491, 15 lines modified
491 ········<a·href="shell.html"·class="btn·btn-neutral·float-left"·title="Scrapy·shell"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>491 ········<a·href="shell.html"·class="btn·btn-neutral·float-left"·title="Scrapy·shell"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
492 ····</div>492 ····</div>
  
493 ··<hr/>493 ··<hr/>
  
494 ··<div·role="contentinfo">494 ··<div·role="contentinfo">
495 ····<p>495 ····<p>
496 ········&#169;·Copyright·2008–2024,·Scrapy·developers.496 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
497 ······<span·class="lastupdated">497 ······<span·class="lastupdated">
498 ········Last·updated·on·May·20,·2022.498 ········Last·updated·on·May·20,·2022.
499 ······</span>499 ······</span>
  
500 ····</p>500 ····</p>
501 ··</div>501 ··</div>
502 ····502 ····
⊟
629 B
html2text {}
    
Offset 257, 9 lines modifiedOffset 257, 9 lines modified
257 ····'myproject.pipelines.JsonWriterPipeline':·800,257 ····'myproject.pipelines.JsonWriterPipeline':·800,
258 }258 }
259 The·integer·values·you·assign·to·classes·in·this·setting·determine·the·order·in259 The·integer·values·you·assign·to·classes·in·this·setting·determine·the·order·in
260 which·they·run:·items·go·through·from·lower·valued·to·higher·valued·classes.260 which·they·run:·items·go·through·from·lower·valued·to·higher·valued·classes.
261 It’s·customary·to·define·these·numbers·in·the·0-1000·range.261 It’s·customary·to·define·these·numbers·in·the·0-1000·range.
262 Next·Previous262 Next·Previous
263 ===============================================================================263 ===============================================================================
264 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.264 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
265 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.265 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.17 KB
./usr/share/doc/python-scrapy-doc/html/topics/items.html
    
Offset 614, 15 lines modifiedOffset 614, 15 lines modified
614 ········<a·href="selectors.html"·class="btn·btn-neutral·float-left"·title="Selectors"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>614 ········<a·href="selectors.html"·class="btn·btn-neutral·float-left"·title="Selectors"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
615 ····</div>615 ····</div>
  
616 ··<hr/>616 ··<hr/>
  
617 ··<div·role="contentinfo">617 ··<div·role="contentinfo">
618 ····<p>618 ····<p>
619 ········&#169;·Copyright·2008–2024,·Scrapy·developers.619 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
620 ······<span·class="lastupdated">620 ······<span·class="lastupdated">
621 ········Last·updated·on·May·20,·2022.621 ········Last·updated·on·May·20,·2022.
622 ······</span>622 ······</span>
  
623 ····</p>623 ····</p>
624 ··</div>624 ··</div>
625 ····625 ····
⊟
581 B
html2text {}
    
Offset 304, 9 lines modifiedOffset 304, 9 lines modified
304 ······False·otherwise.304 ······False·otherwise.
305 ······Alias·for·ItemAdapter.is_item305 ······Alias·for·ItemAdapter.is_item
306 *****·Other·classes·related·to·items¶·*****306 *****·Other·classes·related·to·items¶·*****
307 ··classscrapy.item.ItemMeta(class_name,·bases,·attrs)[source]¶307 ··classscrapy.item.ItemMeta(class_name,·bases,·attrs)[source]¶
308 ······Metaclass·of·Item·that·handles·field·definitions.308 ······Metaclass·of·Item·that·handles·field·definitions.
309 Next·Previous309 Next·Previous
310 ===============================================================================310 ===============================================================================
311 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.311 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
312 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.312 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.24 KB
./usr/share/doc/python-scrapy-doc/html/topics/jobs.html
    
Offset 306, 15 lines modifiedOffset 306, 15 lines modified
306 ········<a·href="benchmarking.html"·class="btn·btn-neutral·float-left"·title="Benchmarking"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>306 ········<a·href="benchmarking.html"·class="btn·btn-neutral·float-left"·title="Benchmarking"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
307 ····</div>307 ····</div>
  
308 ··<hr/>308 ··<hr/>
  
309 ··<div·role="contentinfo">309 ··<div·role="contentinfo">
310 ····<p>310 ····<p>
311 ········&#169;·Copyright·2008–2024,·Scrapy·developers.311 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
312 ······<span·class="lastupdated">312 ······<span·class="lastupdated">
313 ········Last·updated·on·May·20,·2022.313 ········Last·updated·on·May·20,·2022.
314 ······</span>314 ······</span>
  
315 ····</p>315 ····</p>
316 ··</div>316 ··</div>
317 ····317 ····
⊟
653 B
html2text {}
    
Offset 116, 9 lines modifiedOffset 116, 9 lines modified
116 except·for·the·callback·and·errback·values·passed·to·their·__init__·method,116 except·for·the·callback·and·errback·values·passed·to·their·__init__·method,
117 which·must·be·methods·of·the·running·Spider·class.117 which·must·be·methods·of·the·running·Spider·class.
118 If·you·wish·to·log·the·requests·that·couldn’t·be·serialized,·you·can·set·the118 If·you·wish·to·log·the·requests·that·couldn’t·be·serialized,·you·can·set·the
119 SCHEDULER_DEBUG·setting·to·True·in·the·project’s·settings·page.·It·is·False119 SCHEDULER_DEBUG·setting·to·True·in·the·project’s·settings·page.·It·is·False
120 by·default.120 by·default.
121 Next·Previous121 Next·Previous
122 ===============================================================================122 ===============================================================================
123 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.123 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
124 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.124 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.26 KB
./usr/share/doc/python-scrapy-doc/html/topics/leaks.html
    
Offset 493, 15 lines modifiedOffset 493, 15 lines modified
493 ········<a·href="dynamic-content.html"·class="btn·btn-neutral·float-left"·title="Selecting·dynamically-loaded·content"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>493 ········<a·href="dynamic-content.html"·class="btn·btn-neutral·float-left"·title="Selecting·dynamically-loaded·content"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
494 ····</div>494 ····</div>
  
495 ··<hr/>495 ··<hr/>
  
496 ··<div·role="contentinfo">496 ··<div·role="contentinfo">
497 ····<p>497 ····<p>
498 ········&#169;·Copyright·2008–2024,·Scrapy·developers.498 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
499 ······<span·class="lastupdated">499 ······<span·class="lastupdated">
500 ········Last·updated·on·May·20,·2022.500 ········Last·updated·on·May·20,·2022.
501 ······</span>501 ······</span>
  
502 ····</p>502 ····</p>
503 ··</div>503 ··</div>
504 ····504 ····
⊟
643 B
html2text {}
    
Offset 258, 9 lines modifiedOffset 258, 9 lines modified
258 ·····collector,·which·is·able·to·move·objects·in·memory.·This·would258 ·····collector,·which·is·able·to·move·objects·in·memory.·This·would
259 ·····require·significant·changes·to·the·Python·interpreter.259 ·····require·significant·changes·to·the·Python·interpreter.
260 To·keep·memory·consumption·reasonable·you·can·split·the·job·into·several260 To·keep·memory·consumption·reasonable·you·can·split·the·job·into·several
261 smaller·jobs·or·enable·persistent_job_queue·and·stop/start·spider·from·time·to261 smaller·jobs·or·enable·persistent_job_queue·and·stop/start·spider·from·time·to
262 time.262 time.
263 Next·Previous263 Next·Previous
264 ===============================================================================264 ===============================================================================
265 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.265 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
266 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.266 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.29 KB
./usr/share/doc/python-scrapy-doc/html/topics/link-extractors.html
    
Offset 381, 15 lines modifiedOffset 381, 15 lines modified
381 ········<a·href="request-response.html"·class="btn·btn-neutral·float-left"·title="Requests·and·Responses"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>381 ········<a·href="request-response.html"·class="btn·btn-neutral·float-left"·title="Requests·and·Responses"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
382 ····</div>382 ····</div>
  
383 ··<hr/>383 ··<hr/>
  
384 ··<div·role="contentinfo">384 ··<div·role="contentinfo">
385 ····<p>385 ····<p>
386 ········&#169;·Copyright·2008–2024,·Scrapy·developers.386 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
387 ······<span·class="lastupdated">387 ······<span·class="lastupdated">
388 ········Last·updated·on·May·20,·2022.388 ········Last·updated·on·May·20,·2022.
389 ······</span>389 ······</span>
  
390 ····</p>390 ····</p>
391 ··</div>391 ··</div>
392 ····392 ····
⊟
669 B
html2text {}
    
Offset 109, 9 lines modifiedOffset 109, 9 lines modified
109 ··················Dont·follow·this·one.109 ··················Dont·follow·this·one.
110 ················*·fragment·â€“·the·part·of·the·url·after·the·hash·symbol.·From110 ················*·fragment·â€“·the·part·of·the·url·after·the·hash·symbol.·From
111 ··················the·sample,·this·is·foo.111 ··················the·sample,·this·is·foo.
112 ················*·nofollow·â€“·an·indication·of·the·presence·or·absence·of·a112 ················*·nofollow·â€“·an·indication·of·the·presence·or·absence·of·a
113 ··················nofollow·value·in·the·rel·attribute·of·the·anchor·tag.113 ··················nofollow·value·in·the·rel·attribute·of·the·anchor·tag.
114 Next·Previous114 Next·Previous
115 ===============================================================================115 ===============================================================================
116 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.116 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
117 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.117 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.26 KB
./usr/share/doc/python-scrapy-doc/html/topics/loaders.html
    
Offset 873, 15 lines modifiedOffset 873, 15 lines modified
873 ········<a·href="items.html"·class="btn·btn-neutral·float-left"·title="Items"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>873 ········<a·href="items.html"·class="btn·btn-neutral·float-left"·title="Items"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
874 ····</div>874 ····</div>
  
875 ··<hr/>875 ··<hr/>
  
876 ··<div·role="contentinfo">876 ··<div·role="contentinfo">
877 ····<p>877 ····<p>
878 ········&#169;·Copyright·2008–2024,·Scrapy·developers.878 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
879 ······<span·class="lastupdated">879 ······<span·class="lastupdated">
880 ········Last·updated·on·May·20,·2022.880 ········Last·updated·on·May·20,·2022.
881 ······</span>881 ······</span>
  
882 ····</p>882 ····</p>
883 ··</div>883 ··</div>
884 ····884 ····
⊟
679 B
html2text {}
    
Offset 513, 9 lines modifiedOffset 513, 9 lines modified
513 There·are·many·other·possible·ways·to·extend,·inherit·and·override·your·Item513 There·are·many·other·possible·ways·to·extend,·inherit·and·override·your·Item
514 Loaders,·and·different·Item·Loaders·hierarchies·may·fit·better·for·different514 Loaders,·and·different·Item·Loaders·hierarchies·may·fit·better·for·different
515 projects.·Scrapy·only·provides·the·mechanism;·it·doesn’t·impose·any·specific515 projects.·Scrapy·only·provides·the·mechanism;·it·doesn’t·impose·any·specific
516 organization·of·your·Loaders·collection·-·that’s·up·to·you·and·your516 organization·of·your·Loaders·collection·-·that’s·up·to·you·and·your
517 project’s·needs.517 project’s·needs.
518 Next·Previous518 Next·Previous
519 ===============================================================================519 ===============================================================================
520 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.520 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
521 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.521 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.13 KB
./usr/share/doc/python-scrapy-doc/html/topics/logging.html
    
Offset 599, 15 lines modifiedOffset 599, 15 lines modified
599 ········<a·href="exceptions.html"·class="btn·btn-neutral·float-left"·title="Exceptions"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>599 ········<a·href="exceptions.html"·class="btn·btn-neutral·float-left"·title="Exceptions"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
600 ····</div>600 ····</div>
  
601 ··<hr/>601 ··<hr/>
  
602 ··<div·role="contentinfo">602 ··<div·role="contentinfo">
603 ····<p>603 ····<p>
604 ········&#169;·Copyright·2008–2024,·Scrapy·developers.604 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
605 ······<span·class="lastupdated">605 ······<span·class="lastupdated">
606 ········Last·updated·on·May·20,·2022.606 ········Last·updated·on·May·20,·2022.
607 ······</span>607 ······</span>
  
608 ····</p>608 ····</p>
609 ··</div>609 ··</div>
610 ····610 ····
⊟
533 B
html2text {}
    
Offset 319, 9 lines modifiedOffset 319, 9 lines modified
319 ··········format='%(levelname)s:·%(message)s',319 ··········format='%(levelname)s:·%(message)s',
320 ··········level=logging.INFO320 ··········level=logging.INFO
321 ······)321 ······)
322 ······Refer·to·Run_Scrapy_from_a_script·for·more·details·about·using·Scrapy322 ······Refer·to·Run_Scrapy_from_a_script·for·more·details·about·using·Scrapy
323 ······this·way.323 ······this·way.
324 Next·Previous324 Next·Previous
325 ===============================================================================325 ===============================================================================
326 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.326 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
327 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.327 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.18 KB
./usr/share/doc/python-scrapy-doc/html/topics/media-pipeline.html
    
Offset 790, 15 lines modifiedOffset 790, 15 lines modified
790 ········<a·href="leaks.html"·class="btn·btn-neutral·float-left"·title="Debugging·memory·leaks"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>790 ········<a·href="leaks.html"·class="btn·btn-neutral·float-left"·title="Debugging·memory·leaks"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
791 ····</div>791 ····</div>
  
792 ··<hr/>792 ··<hr/>
  
793 ··<div·role="contentinfo">793 ··<div·role="contentinfo">
794 ····<p>794 ····<p>
795 ········&#169;·Copyright·2008–2024,·Scrapy·developers.795 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
796 ······<span·class="lastupdated">796 ······<span·class="lastupdated">
797 ········Last·updated·on·May·20,·2022.797 ········Last·updated·on·May·20,·2022.
798 ······</span>798 ······</span>
  
799 ····</p>799 ····</p>
800 ··</div>800 ··</div>
801 ····801 ····
⊟
570 B
html2text {}
    
Offset 461, 9 lines modifiedOffset 461, 9 lines modified
461 To·enable·your·custom·media·pipeline·component·you·must·add·its·class·import461 To·enable·your·custom·media·pipeline·component·you·must·add·its·class·import
462 path·to·the·ITEM_PIPELINES·setting,·like·in·the·following·example:462 path·to·the·ITEM_PIPELINES·setting,·like·in·the·following·example:
463 ITEM_PIPELINES·=·{463 ITEM_PIPELINES·=·{
464 ····'myproject.pipelines.MyImagesPipeline':·300464 ····'myproject.pipelines.MyImagesPipeline':·300
465 }465 }
466 Next·Previous466 Next·Previous
467 ===============================================================================467 ===============================================================================
468 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.468 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
469 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.469 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.24 KB
./usr/share/doc/python-scrapy-doc/html/topics/practices.html
    
Offset 448, 15 lines modifiedOffset 448, 15 lines modified
448 ········<a·href="contracts.html"·class="btn·btn-neutral·float-left"·title="Spiders·Contracts"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>448 ········<a·href="contracts.html"·class="btn·btn-neutral·float-left"·title="Spiders·Contracts"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
449 ····</div>449 ····</div>
  
450 ··<hr/>450 ··<hr/>
  
451 ··<div·role="contentinfo">451 ··<div·role="contentinfo">
452 ····<p>452 ····<p>
453 ········&#169;·Copyright·2008–2024,·Scrapy·developers.453 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
454 ······<span·class="lastupdated">454 ······<span·class="lastupdated">
455 ········Last·updated·on·May·20,·2022.455 ········Last·updated·on·May·20,·2022.
456 ······</span>456 ······</span>
  
457 ····</p>457 ····</p>
458 ··</div>458 ··</div>
459 ····459 ····
⊟
635 B
html2text {}
    
Offset 254, 9 lines modifiedOffset 254, 9 lines modified
254 ····*·use·a·highly·distributed·downloader·that·circumvents·bans·internally,·so254 ····*·use·a·highly·distributed·downloader·that·circumvents·bans·internally,·so
255 ······you·can·just·focus·on·parsing·clean·pages.·One·example·of·such255 ······you·can·just·focus·on·parsing·clean·pages.·One·example·of·such
256 ······downloaders·is·Crawlera256 ······downloaders·is·Crawlera
257 If·you·are·still·unable·to·prevent·your·bot·getting·banned,·consider·contacting257 If·you·are·still·unable·to·prevent·your·bot·getting·banned,·consider·contacting
258 commercial_support.258 commercial_support.
259 Next·Previous259 Next·Previous
260 ===============================================================================260 ===============================================================================
261 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.261 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
262 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.262 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.24 KB
./usr/share/doc/python-scrapy-doc/html/topics/request-response.html
    
Offset 1262, 15 lines modifiedOffset 1262, 15 lines modified
1262 ········<a·href="feed-exports.html"·class="btn·btn-neutral·float-left"·title="Feed·exports"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>1262 ········<a·href="feed-exports.html"·class="btn·btn-neutral·float-left"·title="Feed·exports"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
1263 ····</div>1263 ····</div>
  
1264 ··<hr/>1264 ··<hr/>
  
1265 ··<div·role="contentinfo">1265 ··<div·role="contentinfo">
1266 ····<p>1266 ····<p>
1267 ········&#169;·Copyright·2008–2024,·Scrapy·developers.1267 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
1268 ······<span·class="lastupdated">1268 ······<span·class="lastupdated">
1269 ········Last·updated·on·May·20,·2022.1269 ········Last·updated·on·May·20,·2022.
1270 ······</span>1270 ······</span>
  
1271 ····</p>1271 ····</p>
1272 ··</div>1272 ··</div>
1273 ····1273 ····
⊟
626 B
html2text {}
    
Offset 715, 9 lines modifiedOffset 715, 9 lines modified
715 ****·XmlResponse·objects¶·****715 ****·XmlResponse·objects¶·****
716 ··classscrapy.http.XmlResponse(url[,·...])[source]¶716 ··classscrapy.http.XmlResponse(url[,·...])[source]¶
717 ······The·XmlResponse·class·is·a·subclass·of·TextResponse·which·adds·encoding717 ······The·XmlResponse·class·is·a·subclass·of·TextResponse·which·adds·encoding
718 ······auto-discovering·support·by·looking·into·the·XML·declaration·line.·See718 ······auto-discovering·support·by·looking·into·the·XML·declaration·line.·See
719 ······TextResponse.encoding.719 ······TextResponse.encoding.
720 Next·Previous720 Next·Previous
721 ===============================================================================721 ===============================================================================
722 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.722 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
723 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.723 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.01 KB
./usr/share/doc/python-scrapy-doc/html/topics/scrapyd.html
    
Offset 228, 15 lines modifiedOffset 228, 15 lines modified
228 ··········</div>228 ··········</div>
229 ··········<footer>229 ··········<footer>
  
230 ··<hr/>230 ··<hr/>
  
231 ··<div·role="contentinfo">231 ··<div·role="contentinfo">
232 ····<p>232 ····<p>
233 ········&#169;·Copyright·2008–2024,·Scrapy·developers.233 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
234 ······<span·class="lastupdated">234 ······<span·class="lastupdated">
235 ········Last·updated·on·May·20,·2022.235 ········Last·updated·on·May·20,·2022.
236 ······</span>236 ······</span>
  
237 ····</p>237 ····</p>
238 ··</div>238 ··</div>
239 ····239 ····
⊟
577 B
html2text {}
    
Offset 65, 9 lines modifiedOffset 65, 9 lines modified
65 ===============================================================================65 ===============================================================================
66 ******·Scrapyd¶·******66 ******·Scrapyd¶·******
67 Scrapyd·has·been·moved·into·a·separate·project.67 Scrapyd·has·been·moved·into·a·separate·project.
68 Its·documentation·is·now·hosted·at:68 Its·documentation·is·now·hosted·at:
69 ·····https://scrapyd.readthedocs.io/en/latest/69 ·····https://scrapyd.readthedocs.io/en/latest/
  
70 ===============================================================================70 ===============================================================================
71 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.71 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
72 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.72 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.18 KB
./usr/share/doc/python-scrapy-doc/html/topics/selectors.html
    
Offset 1403, 15 lines modifiedOffset 1403, 15 lines modified
1403 ········<a·href="spiders.html"·class="btn·btn-neutral·float-left"·title="Spiders"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>1403 ········<a·href="spiders.html"·class="btn·btn-neutral·float-left"·title="Spiders"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
1404 ····</div>1404 ····</div>
  
1405 ··<hr/>1405 ··<hr/>
  
1406 ··<div·role="contentinfo">1406 ··<div·role="contentinfo">
1407 ····<p>1407 ····<p>
1408 ········&#169;·Copyright·2008–2024,·Scrapy·developers.1408 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
1409 ······<span·class="lastupdated">1409 ······<span·class="lastupdated">
1410 ········Last·updated·on·May·20,·2022.1410 ········Last·updated·on·May·20,·2022.
1411 ······</span>1411 ······</span>
  
1412 ····</p>1412 ····</p>
1413 ··</div>1413 ··</div>
1414 ····1414 ····
⊟
590 B
html2text {}
    
Offset 889, 9 lines modifiedOffset 889, 9 lines modified
889 ······sel.xpath("//product")889 ······sel.xpath("//product")
890 ···2.·Extract·all·prices·from·a·Google_Base_XML_feed·which·requires·registering890 ···2.·Extract·all·prices·from·a·Google_Base_XML_feed·which·requires·registering
891 ······a·namespace:891 ······a·namespace:
892 ······sel.register_namespace("g",·"http://base.google.com/ns/1.0")892 ······sel.register_namespace("g",·"http://base.google.com/ns/1.0")
893 ······sel.xpath("//g:price").getall()893 ······sel.xpath("//g:price").getall()
894 Next·Previous894 Next·Previous
895 ===============================================================================895 ===============================================================================
896 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.896 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
897 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.897 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.09 KB
./usr/share/doc/python-scrapy-doc/html/topics/settings.html
    
Offset 1578, 15 lines modifiedOffset 1578, 15 lines modified
1578 ········<a·href="link-extractors.html"·class="btn·btn-neutral·float-left"·title="Link·Extractors"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>1578 ········<a·href="link-extractors.html"·class="btn·btn-neutral·float-left"·title="Link·Extractors"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
1579 ····</div>1579 ····</div>
  
1580 ··<hr/>1580 ··<hr/>
  
1581 ··<div·role="contentinfo">1581 ··<div·role="contentinfo">
1582 ····<p>1582 ····<p>
1583 ········&#169;·Copyright·2008–2024,·Scrapy·developers.1583 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
1584 ······<span·class="lastupdated">1584 ······<span·class="lastupdated">
1585 ········Last·updated·on·May·20,·2022.1585 ········Last·updated·on·May·20,·2022.
1586 ······</span>1586 ······</span>
  
1587 ····</p>1587 ····</p>
1588 ··</div>1588 ··</div>
1589 ····1589 ····
⊟
485 B
html2text {}
    
Offset 1065, 9 lines modifiedOffset 1065, 9 lines modified
1065 ····*·RETRY_TIMES1065 ····*·RETRY_TIMES
1066 ····*·TELNETCONSOLE_HOST1066 ····*·TELNETCONSOLE_HOST
1067 ····*·TELNETCONSOLE_PASSWORD1067 ····*·TELNETCONSOLE_PASSWORD
1068 ····*·TELNETCONSOLE_PORT1068 ····*·TELNETCONSOLE_PORT
1069 ····*·TELNETCONSOLE_USERNAME1069 ····*·TELNETCONSOLE_USERNAME
1070 Next·Previous1070 Next·Previous
1071 ===============================================================================1071 ===============================================================================
1072 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.1072 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
1073 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.1073 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.18 KB
./usr/share/doc/python-scrapy-doc/html/topics/shell.html
    
Offset 511, 15 lines modifiedOffset 511, 15 lines modified
511 ········<a·href="loaders.html"·class="btn·btn-neutral·float-left"·title="Item·Loaders"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>511 ········<a·href="loaders.html"·class="btn·btn-neutral·float-left"·title="Item·Loaders"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
512 ····</div>512 ····</div>
  
513 ··<hr/>513 ··<hr/>
  
514 ··<div·role="contentinfo">514 ··<div·role="contentinfo">
515 ····<p>515 ····<p>
516 ········&#169;·Copyright·2008–2024,·Scrapy·developers.516 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
517 ······<span·class="lastupdated">517 ······<span·class="lastupdated">
518 ········Last·updated·on·May·20,·2022.518 ········Last·updated·on·May·20,·2022.
519 ······</span>519 ······</span>
  
520 ····</p>520 ····</p>
521 ··</div>521 ··</div>
522 ····522 ····
⊟
593 B
html2text {}
    
Offset 281, 9 lines modifiedOffset 281, 9 lines modified
281 example.net>·(referer:·None)281 example.net>·(referer:·None)
282 ...282 ...
283 Note·that·you·can’t·use·the·fetch·shortcut·here·since·the·Scrapy·engine·is283 Note·that·you·can’t·use·the·fetch·shortcut·here·since·the·Scrapy·engine·is
284 blocked·by·the·shell.·However,·after·you·leave·the·shell,·the·spider·will284 blocked·by·the·shell.·However,·after·you·leave·the·shell,·the·spider·will
285 continue·crawling·where·it·stopped,·as·shown·above.285 continue·crawling·where·it·stopped,·as·shown·above.
286 Next·Previous286 Next·Previous
287 ===============================================================================287 ===============================================================================
288 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.288 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
289 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.289 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.23 KB
./usr/share/doc/python-scrapy-doc/html/topics/signals.html
    
Offset 717, 15 lines modifiedOffset 717, 15 lines modified
717 ········<a·href="api.html"·class="btn·btn-neutral·float-left"·title="Core·API"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>717 ········<a·href="api.html"·class="btn·btn-neutral·float-left"·title="Core·API"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
718 ····</div>718 ····</div>
  
719 ··<hr/>719 ··<hr/>
  
720 ··<div·role="contentinfo">720 ··<div·role="contentinfo">
721 ····<p>721 ····<p>
722 ········&#169;·Copyright·2008–2024,·Scrapy·developers.722 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
723 ······<span·class="lastupdated">723 ······<span·class="lastupdated">
724 ········Last·updated·on·May·20,·2022.724 ········Last·updated·on·May·20,·2022.
725 ······</span>725 ······</span>
  
726 ····</p>726 ····</p>
727 ··</div>727 ··</div>
728 ····728 ····
⊟
644 B
html2text {}
    
Offset 361, 9 lines modifiedOffset 361, 9 lines modified
361 ················*·response·(Response·object)·â€“·the·response·downloaded361 ················*·response·(Response·object)·â€“·the·response·downloaded
362 ················*·request·(Request·object)·â€“·the·request·that·generated·the362 ················*·request·(Request·object)·â€“·the·request·that·generated·the
363 ··················response363 ··················response
364 ················*·spider·(Spider·object)·â€“·the·spider·for·which·the·response364 ················*·spider·(Spider·object)·â€“·the·spider·for·which·the·response
365 ··················is·intended365 ··················is·intended
366 Next·Previous366 Next·Previous
367 ===============================================================================367 ===============================================================================
368 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.368 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
369 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.369 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.29 KB
./usr/share/doc/python-scrapy-doc/html/topics/spider-middleware.html
    
Offset 746, 15 lines modifiedOffset 746, 15 lines modified
746 ········<a·href="downloader-middleware.html"·class="btn·btn-neutral·float-left"·title="Downloader·Middleware"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>746 ········<a·href="downloader-middleware.html"·class="btn·btn-neutral·float-left"·title="Downloader·Middleware"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
747 ····</div>747 ····</div>
  
748 ··<hr/>748 ··<hr/>
  
749 ··<div·role="contentinfo">749 ··<div·role="contentinfo">
750 ····<p>750 ····<p>
751 ········&#169;·Copyright·2008–2024,·Scrapy·developers.751 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
752 ······<span·class="lastupdated">752 ······<span·class="lastupdated">
753 ········Last·updated·on·May·20,·2022.753 ········Last·updated·on·May·20,·2022.
754 ······</span>754 ······</span>
  
755 ····</p>755 ····</p>
756 ··</div>756 ··</div>
757 ····757 ····
⊟
657 B
html2text {}
    
Offset 400, 9 lines modifiedOffset 400, 9 lines modified
400 ······Filters·out·requests·with·URLs·longer·than·URLLENGTH_LIMIT400 ······Filters·out·requests·with·URLs·longer·than·URLLENGTH_LIMIT
401 ······The·UrlLengthMiddleware·can·be·configured·through·the·following·settings401 ······The·UrlLengthMiddleware·can·be·configured·through·the·following·settings
402 ······(see·the·settings·documentation·for·more·info):402 ······(see·the·settings·documentation·for·more·info):
403 ···············*·URLLENGTH_LIMIT·-·The·maximum·URL·length·to·allow·for403 ···············*·URLLENGTH_LIMIT·-·The·maximum·URL·length·to·allow·for
404 ·················crawled·URLs.404 ·················crawled·URLs.
405 Next·Previous405 Next·Previous
406 ===============================================================================406 ===============================================================================
407 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.407 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
408 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.408 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.11 KB
./usr/share/doc/python-scrapy-doc/html/topics/spiders.html
    
Offset 1087, 15 lines modifiedOffset 1087, 15 lines modified
1087 ········<a·href="commands.html"·class="btn·btn-neutral·float-left"·title="Command·line·tool"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>1087 ········<a·href="commands.html"·class="btn·btn-neutral·float-left"·title="Command·line·tool"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
1088 ····</div>1088 ····</div>
  
1089 ··<hr/>1089 ··<hr/>
  
1090 ··<div·role="contentinfo">1090 ··<div·role="contentinfo">
1091 ····<p>1091 ····<p>
1092 ········&#169;·Copyright·2008–2024,·Scrapy·developers.1092 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
1093 ······<span·class="lastupdated">1093 ······<span·class="lastupdated">
1094 ········Last·updated·on·May·20,·2022.1094 ········Last·updated·on·May·20,·2022.
1095 ······</span>1095 ······</span>
  
1096 ····</p>1096 ····</p>
1097 ··</div>1097 ··</div>
1098 ····1098 ····
⊟
512 B
html2text {}
    
Offset 695, 9 lines modifiedOffset 695, 9 lines modified
695 ····def·parse_shop(self,·response):695 ····def·parse_shop(self,·response):
696 ········pass·#·...·scrape·shop·here·...696 ········pass·#·...·scrape·shop·here·...
  
697 ····def·parse_other(self,·response):697 ····def·parse_other(self,·response):
698 ········pass·#·...·scrape·other·here·...698 ········pass·#·...·scrape·other·here·...
699 Next·Previous699 Next·Previous
700 ===============================================================================700 ===============================================================================
701 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.701 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
702 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.702 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.3 KB
./usr/share/doc/python-scrapy-doc/html/topics/stats.html
    
Offset 334, 15 lines modifiedOffset 334, 15 lines modified
334 ········<a·href="logging.html"·class="btn·btn-neutral·float-left"·title="Logging"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>334 ········<a·href="logging.html"·class="btn·btn-neutral·float-left"·title="Logging"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
335 ····</div>335 ····</div>
  
336 ··<hr/>336 ··<hr/>
  
337 ··<div·role="contentinfo">337 ··<div·role="contentinfo">
338 ····<p>338 ····<p>
339 ········&#169;·Copyright·2008–2024,·Scrapy·developers.339 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
340 ······<span·class="lastupdated">340 ······<span·class="lastupdated">
341 ········Last·updated·on·May·20,·2022.341 ········Last·updated·on·May·20,·2022.
342 ······</span>342 ······</span>
  
343 ····</p>343 ····</p>
344 ··</div>344 ··</div>
345 ····345 ····
⊟
719 B
html2text {}
    
Offset 132, 9 lines modifiedOffset 132, 9 lines modified
132 ······A·Stats·collector·which·does·nothing·but·is·very·efficient·(because·it132 ······A·Stats·collector·which·does·nothing·but·is·very·efficient·(because·it
133 ······does·nothing).·This·stats·collector·can·be·set·via·the·STATS_CLASS133 ······does·nothing).·This·stats·collector·can·be·set·via·the·STATS_CLASS
134 ······setting,·to·disable·stats·collect·in·order·to·improve·performance.134 ······setting,·to·disable·stats·collect·in·order·to·improve·performance.
135 ······However,·the·performance·penalty·of·stats·collection·is·usually·marginal135 ······However,·the·performance·penalty·of·stats·collection·is·usually·marginal
136 ······compared·to·other·Scrapy·workload·like·parsing·pages.136 ······compared·to·other·Scrapy·workload·like·parsing·pages.
137 Next·Previous137 Next·Previous
138 ===============================================================================138 ===============================================================================
139 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.139 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
140 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.140 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.14 KB
./usr/share/doc/python-scrapy-doc/html/topics/telnetconsole.html
    
Offset 440, 15 lines modifiedOffset 440, 15 lines modified
440 ········<a·href="email.html"·class="btn·btn-neutral·float-left"·title="Sending·e-mail"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>440 ········<a·href="email.html"·class="btn·btn-neutral·float-left"·title="Sending·e-mail"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
441 ····</div>441 ····</div>
  
442 ··<hr/>442 ··<hr/>
  
443 ··<div·role="contentinfo">443 ··<div·role="contentinfo">
444 ····<p>444 ····<p>
445 ········&#169;·Copyright·2008–2024,·Scrapy·developers.445 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
446 ······<span·class="lastupdated">446 ······<span·class="lastupdated">
447 ········Last·updated·on·May·20,·2022.447 ········Last·updated·on·May·20,·2022.
448 ······</span>448 ······</span>
  
449 ····</p>449 ····</p>
450 ··</div>450 ··</div>
451 ····451 ····
⊟
535 B
html2text {}
    
Offset 189, 9 lines modifiedOffset 189, 9 lines modified
189 The·username·used·for·the·telnet·console189 The·username·used·for·the·telnet·console
190 ****·TELNETCONSOLE_PASSWORD¶·****190 ****·TELNETCONSOLE_PASSWORD¶·****
191 Default:·None191 Default:·None
192 The·password·used·for·the·telnet·console,·default·behaviour·is·to·have·it192 The·password·used·for·the·telnet·console,·default·behaviour·is·to·have·it
193 autogenerated193 autogenerated
194 Next·Previous194 Next·Previous
195 ===============================================================================195 ===============================================================================
196 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.196 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
197 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.197 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.19 KB
./usr/share/doc/python-scrapy-doc/html/topics/webservice.html
    
Offset 234, 15 lines modifiedOffset 234, 15 lines modified
234 ········<a·href="telnetconsole.html"·class="btn·btn-neutral·float-left"·title="Telnet·Console"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>234 ········<a·href="telnetconsole.html"·class="btn·btn-neutral·float-left"·title="Telnet·Console"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
235 ····</div>235 ····</div>
  
236 ··<hr/>236 ··<hr/>
  
237 ··<div·role="contentinfo">237 ··<div·role="contentinfo">
238 ····<p>238 ····<p>
239 ········&#169;·Copyright·2008–2024,·Scrapy·developers.239 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
240 ······<span·class="lastupdated">240 ······<span·class="lastupdated">
241 ········Last·updated·on·May·20,·2022.241 ········Last·updated·on·May·20,·2022.
242 ······</span>242 ······</span>
  
243 ····</p>243 ····</p>
244 ··</div>244 ··</div>
245 ····245 ····
⊟
585 B
html2text {}
    
Offset 67, 9 lines modifiedOffset 67, 9 lines modified
67 ===============================================================================67 ===============================================================================
68 ******·Web·Service¶·******68 ******·Web·Service¶·******
69 webservice·has·been·moved·into·a·separate·project.69 webservice·has·been·moved·into·a·separate·project.
70 It·is·hosted·at:70 It·is·hosted·at:
71 ·····https://github.com/scrapy-plugins/scrapy-jsonrpc71 ·····https://github.com/scrapy-plugins/scrapy-jsonrpc
72 Next·Previous72 Next·Previous
73 ===============================================================================73 ===============================================================================
74 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.74 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
75 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.75 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
⊟
1.16 KB
./usr/share/doc/python-scrapy-doc/html/versioning.html
    
Offset 279, 15 lines modifiedOffset 279, 15 lines modified
279 ········<a·href="contributing.html"·class="btn·btn-neutral·float-left"·title="Contributing·to·Scrapy"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>279 ········<a·href="contributing.html"·class="btn·btn-neutral·float-left"·title="Contributing·to·Scrapy"·accesskey="p"·rel="prev"><span·class="fa·fa-arrow-circle-left"·aria-hidden="true"></span>·Previous</a>
280 ····</div>280 ····</div>
  
281 ··<hr/>281 ··<hr/>
  
282 ··<div·role="contentinfo">282 ··<div·role="contentinfo">
283 ····<p>283 ····<p>
284 ········&#169;·Copyright·2008–2024,·Scrapy·developers.284 ········&#169;·Copyright·2008–2025,·Scrapy·developers.
285 ······<span·class="lastupdated">285 ······<span·class="lastupdated">
286 ········Last·updated·on·May·20,·2022.286 ········Last·updated·on·May·20,·2022.
287 ······</span>287 ······</span>
  
288 ····</p>288 ····</p>
289 ··</div>289 ··</div>
290 ····290 ····
⊟
561 B
html2text {}
    
Offset 101, 9 lines modifiedOffset 101, 9 lines modified
101 14th·2021·or·before·that.101 14th·2021·or·before·that.
102 Any·new·Scrapy·release·after·a·year·may·remove·support·for·that·deprecated102 Any·new·Scrapy·release·after·a·year·may·remove·support·for·that·deprecated
103 feature.103 feature.
104 All·deprecated·features·removed·in·a·Scrapy·release·are·explicitly·mentioned·in104 All·deprecated·features·removed·in·a·Scrapy·release·are·explicitly·mentioned·in
105 the·release_notes.105 the·release_notes.
106 Previous106 Previous
107 ===============================================================================107 ===============================================================================
108 ©·Copyright·2008–2024,·Scrapy·developers.··Last·updated·on·May·20,·2022.108 ©·Copyright·2008–2025,·Scrapy·developers.··Last·updated·on·May·20,·2022.
109 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.109 Built·with·Sphinx·using·a·theme·provided·by·Read_the_Docs.
Generated by diffoscope 240