Archit Pandey 61a32cc596 Using --robots=off downloads robots.txt ...

* src/wget.c (add_url_to_queue): remove config.robots check,
  (add_url): add config.robots check when disallowing URLs
* tests/test-robots-off.c: New file to test --robots=off behavior
* tests/Makefile.am: Added test-robots-off
* tests/test-iri-percent.c: Add check for robots.txt
* docs/wget2.md: Describe --robots=off behaviour

2019-10-22 13:42:28 +02:00

..

development

Add Building with alternate GnuTLS version doc

2018-09-19 12:33:08 +02:00

DoxygenLayout.xml

Update cfg.mk. Add comments for future

2019-09-10 23:02:05 +02:00

faq.md

Update documentation about how to add a new command line option

2018-09-04 14:59:11 +02:00

gen-logo.sh

Add logo to HTML docs

2016-01-18 12:31:46 +01:00

libwget.doxy.in

Update cfg.mk. Add comments for future

2019-09-10 23:02:05 +02:00

Makefile.am

* docs/Makefile.am: Add README.md to DISTCLEANFILES unconditionally

2019-07-11 15:17:13 +02:00

wget2_md2man.sh.in

* docs/wget2_md2man.sh.in: Remove set -o pipefail (incompatible)

2018-01-13 11:49:41 +01:00

wget2.md

Using --robots=off downloads robots.txt

2019-10-22 13:42:28 +02:00

wget-logo.png

Add logo to HTML docs

2016-01-18 12:31:46 +01:00