209 Commits

Author SHA1 Message Date
7a945d31ae Disable TCP Fast Open by default
* docs/wget2.md: Amended description of --tcp-fastopen.
* src/options.c (struct config config): Disabled TFO.
2024-06-30 19:33:23 +02:00
35986bd093 Disable explicit OCSP requests by default
* docs/wget2.md: Document --ocsp default value as 'off'.
* src/options.c (struct config): Disable .ocsp by default.

OCSP validation of the server certificate implies privacy issues:
  - The OCSP request tells the CA which web service the client tries to reach.
  - The OCSP requests are sent via unencrypted HTTP, so every "listener in the
    middle" can see which web service the client tries to connect.
Additionally, the OCSP requests slow down operation and may cause unexpected
network traffic, which may trigger security alarms unnecessarily.

Due to these issues we explicitly disable OCSP by default.
2024-05-18 14:44:53 +02:00
188989e067 Update copyright year 2024-01-07 12:41:02 +01:00
0415d036b3 Fix minor typos in comments 2023-11-04 14:36:43 +01:00
7449d63e17 announce: Remove trailing spaces
Trailing spaces are causing the first stage of the pipeline to fail.
2023-09-11 10:26:38 +02:00
b5a3348997 Fix typos
* NEWS: HTMP -> HTML.
* docs/announce_2.1.0.txt: Likewise.
2023-08-31 16:17:47 +02:00
657e3d22c4 Release v2.1.0 2023-08-31 14:46:39 +02:00
15ba1db248 Update copyrights 2023-08-31 12:57:30 +02:00
fe842f6ee9 * docs/wget2.md: Wrap short options with backtick
Copyright-paperwork-exempt: Yes
2023-08-05 13:31:55 +02:00
f827dbaaae * docs/wget2.md: Wrap long options with backtick
Copyright-paperwork-exempt: Yes
2023-08-05 13:31:51 +02:00
de7cb1fb07 * docs/announce_template.txt: Fix sc_trailing_blank 2023-07-08 13:03:54 +02:00
2c878d8b10 Either CA or DANE verification is required 2023-04-23 19:50:33 +02:00
84d0220e22 New option --dane
* configure.ac: Enable DANE if libgnutls-dane is installed,
  add option --without-libdane.
* docs/wget2.md: Add docs for option --dane.
* include/wget/wget.h: Add WGET_SSL_DANE.
* libwget/ssl_gnutls.c: Include include <gnutls/dane.h> if requested,
  (struct config): New member 'dane',
  (wget_ssl_set_config_int): Set value for dane via WGET_SSL_DANE,
  (wget_ssl_open): Call dane_verify_session_crt().
* src/options.c (options): Add entry for --dane,
  (init): Set config.dane via wget_ssl_set_config_int().
* src/wget_options.h (struct config): Add member 'dane'.
2023-04-23 19:49:12 +02:00
b68b1179fe Add new option --follow-sitemaps
* docs/wget2.md: Add section for --follow-sitemaps.
* src/options.c (struct config config): Set follow_sitemaps = true,
  (options[]): Add "follow-sitemaps".
* src/wget.c (process_response): Only parse sitemaps if config.follow_sitemaps is set.
2022-12-03 19:46:52 +01:00
89f7b8fde6 * docs/wget2.md (--progress): Fix the docs for --progress 2022-06-12 20:03:59 +02:00
c355efb7c8 * docs/wget2.md: Fix .wgetrc to .wget2rc 2022-06-12 14:39:48 +02:00
032734ab8f * docs/announce_template.txt: New file [skip ci] 2022-05-27 19:49:57 +02:00
3d25515253 Release v2.0.1 2022-05-27 11:48:57 +02:00
215facd4e2 Fix storage files in help text and in the docs
* docs/wget2.md: Fix ~/.wget-* to $XDG_DATA_HOME/wget/.wget-*
* src/options.c: Likewise.
2022-02-25 19:02:44 +01:00
3dc7f71098 Update copyright year 2022-02-25 17:46:43 +01:00
97afd2138d Allow spaces and \ escaping in passwords in .netrc
This is Wget1.x compatible.

* docs/wget2.md: Document the new behavior.
* libwget/netrc.c (wget_netrc_db_load): Amended the code to allow \ and "".
* unit-tests/test.c (test_netrc): Add tests.
2022-02-13 19:27:42 +01:00
08ebd8eb5d * docs/announce_2.0.0.txt: Add list of noteworthy changes 2021-09-12 14:02:52 +02:00
b63496ab51 * docs/announce_2.0.0.txt: New file 2021-09-12 13:15:22 +02:00
9e56d9ed50 * docs/wget2.md: Mention that -P creates the prefix directory 2021-05-30 19:23:07 +02:00
22162f82b2 Update copyright year 2021-01-22 21:58:38 +01:00
1948557a4c Implement --download-attr=[strippath|usepath]
* docs/wget2.md: Amend docs.
* src/options.c: Add parse_download_attr(),
  amend option --download-attr.
* src/wget.c (queue_url_from_remote): Strip path if
  DOWNLOAD_ATTR_STRIPPATH is set.
* src/wget_options.h: Add enum download_attr_mode.
* tests/test-download-attr.c: Test strippath and usepath.
2020-09-13 17:16:55 +02:00
c3ab2df4eb Add new option --download-attr
* src/options.c (options): Add new option --download-attr.
* src/wget.c (html_parse): Check for config.download_attr.
* src/wget_options.h (struct config): Add new member download_attr.
2020-03-17 16:49:35 +01:00
91bba3f697 Add option --bind-interface
* libwget/net.h (wget_tcp_st): New variable
  const char *bind_interface.
* libwget/net.c: Add function wget_tcp_bind_interface().
  (set_socket_options): Add parameter wget_tcp *tcp and
  implement binding of tcp to bind_interface.
* libwget/init.c (wget_global_init): Call wget_tcp_bind_interface().
* include/wget/wget.h: Add wget_tcp_bind_interface() and
  WGET_BIND_INTERFACE.
* src/options.c: Add --bind-interface handling.
  (init): Call wget_tcp_bind_interface().
* src/wget2_options.h (config): Add const char *bind_interface.
* docs/wget2.md: Add documentation for --bind-interface.

This finishes task #430
2020-02-24 12:28:00 +01:00
84fd6c1f58 Amend docs regarding -q -O- [skip ci] 2020-02-21 13:00:21 +01:00
0a2ded6ab3 Add --background support for Windows
* src/wget.c: Add function make_section_name(),
  fake_fork_child(), fake_fork(), fork_to_background().
  Add struct fake_fork_info.
* docs/wget2.md: Remove [Not Supported on Windows yet]
  from --background.

Closes issue #457

Copyright-paperwork-exempt: Yes
2020-01-28 10:40:55 +01:00
59d9ecd9c0 * Update copyright year to 2020 2020-01-10 00:33:02 +01:00
9ab3c2a80d Support terminal hyperlinks in output
Some terminal emulators now allow printing hyperlinks through escape
sequences. See
https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda for
more information.

Hyperlink support cannot be gauged accurately. And while most terminals
just ignore the extra escape sequences, older versions of vte actually
print the URI along with some garbage. So, in this patch, the hyperlink
support is disabled by default and used only if the user explicitly
requests it. Once there is a way to query for feature support in
terminals, this can be intelligently enabled automatically.

* bootstrap.conf: Add new module xgethostname
* src/wget_options.h: Add a new option hyperlink and a global variable
  hostname
* src/options.c: Add a new command line option --hyperlink. This is the
  same option name as `ls` from coreutils for consistency across
  programs
  (init): If the user requests hyperlink support, get the hostname and
  store it. This is to prevent having to get the hostname for each
  downloaded file
* src/wget.c (prepare_file): Print the filename with a hyperlink if the
  user requests it.
* docs/wget2.md: Add documentation entry for hyperlinks
2019-11-06 15:09:44 +01:00
e5f33ae07a Added --convert-file-only handling
* src/options.c: Add --convert-file-only handling
* src/wget_options.h (config): Add bool convert_file_only
* src/wget.c (html_parse): conversions initialized when
convert_file_only is set
(main): Call convert_links when config.convert_file_only
(convert_links): move conversion logic to convert_link_whole and
convert_link_file_only
New function convert_link_whole: convert whole URL
New function convert_link_file_only: convert filename part of URLs
* tests/test-convert-file-only.c: testcase for --convert-file-only
with -E
* tests/Makefile.am: add test-convert-file-only object
* docs/wget2.md: Removed [unimplemented-option] from
--convert-file-only
2019-11-06 15:06:29 +01:00
5c42026ee4 Add --ignore-length handling
* include/wget/wget.h (wget_http_request): Add bool
response_ignorelength
(wget_http_response): Add bool length_inconsistent
* src/options.c: Add --ignore-length handling
* src/wget_options.h (config): Add bool ignore_length
(exit_status_e): Add WGET_EXIT_INCONSISTENT_LENGTH
* libwget/http.c (wget_http_response_cb): Set length_inconsistent
when mismatch with Content-Length
* src/wget.c (wget_send_request): Set ignore-length in request
* tests/test-ignore-length.c: test --ignore-length
* tests/Makefile.am: add test-ignore-length
* docs/wget2.md: Removed [unimplemented-option] from
--ignore-length
2019-11-05 21:43:17 +05:30
a1f3f7bcc5 src/libwget.doxy.in: Update doxygen file 2019-11-02 16:23:38 +01:00
c34d426d5b Explicitly set the FILE_PATTERNS for doxygen
* docs/libwget.doxy.in: Set FILE_PATTERNS to workaround bug in doxygen
  1.8.16

Fix-by: Tim Ruehsen <rockdaboot@gmail.com>
2019-11-02 16:23:38 +01:00
3a56a2f392 Added options --method, --body-data, --body-file
* src/wget_options.h (config): Added method, body_data, body_file
* src/options.c (options): Added --method, --body-data, --body-file
handling
* src/wget_job.h (JOB): New variable bool redirect_get
* src/wget.c (http_create_request): Added support for user
provided HTTP method and request body
(process_response_header): Set redirect_get to TRUE for 30x
response codes
* tests/test-redirection.c: Added test for redirection with
--method=POST
* docs/wget2.md: Removed [unimplemented-option] for --method,
--body-data, --body-file
2019-10-27 19:27:32 +01:00
61a32cc596 Using --robots=off downloads robots.txt
* src/wget.c (add_url_to_queue): remove config.robots check,
  (add_url): add config.robots check when disallowing URLs
* tests/test-robots-off.c: New file to test --robots=off behavior
* tests/Makefile.am: Added test-robots-off
* tests/test-iri-percent.c: Add check for robots.txt
* docs/wget2.md: Describe --robots=off behaviour
2019-10-22 13:42:28 +02:00
998b74c18c Rename --retry-on-http-status to --retry-on-http-error
* docs/wget2.md: Likewise
* libwget/http.c: Likewise
* src/options.c: Likewise
* src/wget.c: Likewise
* src/wget_options.h: Likewise

Needed for backward compatibility with Wget 1.x.
2019-10-04 17:47:37 +02:00
3bd8e6aa97 Implement Accept-Encoding: lzip including decompressor
* README.md: Mention lzip
* configure.ac: Detect lzip
* docs/wget2.md: Document 'lzip'
* include/wget/wget.h: Add wget_content_encoding_lzip
* libwget/decompressor.c: Implement lzip decompression
* libwget/http_parse.c (wget_http_parse_content_encoding): Add "lzip"
* src/options.c: Add code to support lzip
* src/wget.c (http_create_request): Add lzip to Accept-Encoding: if available
* tests/test-compression.c: Add test vector for lzip decompression
2019-09-19 12:58:09 +02:00
a4dac96cf3 Update cfg.mk. Add comments for future
* cfg.mk: Multiple changes.
  (sc_GPL_version): Don't exclude all files when only one has a parsing
  issue
  (sc_po_check): Tests should always output English. Their output is for
  developers, not users
  (sc_prohibit_magic_number_exit): tests/test-plugin.c has a parsing
  issue that has been reported to gnulib. Others should be checked.
  (sc_trailing_blank): Fixed the relevant files instead of ignoring them
  (sc_two_space_separator_in_usage): Doesn't seem to trigger on removal
  (sc_prohibit_empty_lines_at_EOF): Same
  (sc_prohibit_sprintf): Same
  (sc_prohibit_printf): Re-arrange within file
  (sc_prohibit_free): Same
  (sc_prohibit_alloc): Same
  (sc_gettext_printf): Same
* docs/DoxygenLayout.xml: Remove trailing whitespaces
* docs/libwget.doxy.in: Same
* tests/libtest.c: Don't internationalize test suite strings
* tests/test-*.c: Use EXIT_FAILURE and EXIT_SUCCESS instead of 1 and 0
2019-09-10 23:02:05 +02:00
6fdea3400c Consider files as 'application/octet-stream' if MIME is not present
* src/wget.c: Likewise, following RFC 7231
* tests/test--filter-mime-type.c: Added tests to ensure this behavior is present
* docs/wget2.md: Added this new behavior to the docs
2019-08-23 10:17:38 +00:00
91bd09aec2 Implement --no-if-modified-since
* src/options.c: Added new option --if-modified-since, and prevented certain combination of flags
* src/wget.c: Added -N --no-if-modified-since implementation itself
* src/wget_job.h: Added bool 'recursive_send_head' to JOB structure
* src/wget_options.h: Added char 'if_modified_since' to config structure
* tests/test--accept.c: Added tests that make use of -N --no-if-modified-since
* tests/test--filter-mime-type.c: Added tests that make use of -N --no-if-modified-since
* tests/test-include-and-exclude-directories.c: Added tests that make use of -N --no-if-modified-since
* tests/test-timestamping.c: Added tests for -N --no-if-modified-since
* docs/wget2.md: Unmarked --no-if-modified-since as "Not Implemented Yet"
2019-08-22 16:20:01 +00:00
603987be5d Improvements on the progress bar
* docs/wget2.md: There is no default progress value
* src/options.c: If no progress type specifier is given return an error
* src/wget.c: Set global file size and already downloaded amount when continuing a download
2019-08-17 17:16:34 +00:00
325b2b126b Implement --start-pos option
* src/wget.c: Added --start-pos implementation
* src/options.c: Added --start-pos option and resolved flag combination with --continue
* src/wget_options.h: Added start_pos to the config structure
* docs/wget2.md: Removed "Not Implemented Yet" label from --start-pos documentation
2019-08-14 10:30:50 +02:00
79801c6dc1 Background support for fork() compatible OS
* src/wget.c: Added fork_to_background(), in order to implement --background
* src/options.c: Added "--background" to the options list
* src/wget_options.h: Added background member to structure config
* docs/wget2.md: Background option marked as "Not supported on Windows"
2019-08-05 07:52:40 +00:00
d00ce9edcc Added option --ocsp-date
* docs/wget2.md: Added documentation for --ocsp-date
* include/wget/wget.h: Added macro WGET_SSL_OCSP_DATE
* libwget/ssl_gnutls.c: Modified to use option
* src/options.c: Added option handler
* src/wget_options.h: Declared bool for option
2019-07-23 22:50:14 +05:30
f294764740 Added option --ocsp-nonce
* docs/wget2.md: Added documentation for --ocsp-nonce
* include/wget/wget.h: Added macro WGET_SSL_OCSP_NONCE
* libwget/ssl_gnutls.c: Modified to use option
* src/options.c: Added option handler
* src/wget_options.h: Declared bool for option
2019-07-23 22:50:14 +05:30
9275125b8d Added option --ocsp-server
* docs/wget2.md: Added documentation for --ocsp-server
* libwget/ssl_gnutls.c: Modified to use value from option parameter
* src/options.c: Added option handler
* src/wget_options.h: Declared storage for option parameter
2019-07-23 22:50:14 +05:30
b9a80ee16f * docs/Makefile.am: Add README.md to DISTCLEANFILES unconditionally 2019-07-11 15:17:13 +02:00