537 Commits

Author SHA1 Message Date
07d10bd090 Drop iri->uri_allocated
* include/wget/wget.h (struct wget_iri_st): Remove uri_allocated.
* libwget/iri.c (wget_iri_free_content): Do not free iri->uri,
  (wget_iri_clone): Simplify code and do not heap-allocate iri->uri.
2025-02-02 19:19:47 +01:00
dc8966d906 Don't log URI userinfo to logs
* include/wget/wget.h (struct wget_iri_st): Add safe_uri member.
* libwget/iri.c: New function create_safe_uri(),
  (wget_iri_unescape_url_inline): Check iri->userinfo,
  (wget_iri_parse): Likewise,
  (wget_iri_clone): Likewise,
  (wget_iri_relative_to_abs): Remove debug logs,
  (wget_iri_set_scheme): Check iri->userinfo.
* src/blacklist.c: Log safe_uri instead of uri.
* src/host.c: Likewise.
* src/options.c: Likewise.
* src/stats_site.c: Likewise.
* src/wget.c: Likewise,
  (queue_url_from_remote): Move redirection check further down.
* unit-tests/test.c (test_iri_parse): Add checks for safe_uri,
  (test_iri_parse): Add safe_uri to IRI comparison,
  Print safe_uri and password.

Co-authored-by: Tim Rühsen <tim.ruehsen@gmx.de>
2024-10-03 17:25:38 +02:00
3359e3e182 set debug_skip_body for OCSP requests in openssl tls provider 2024-09-16 15:34:19 +00:00
ec27488fea Fix downloading multiple files via HTTP/2
* include/wget/wget.h: New function declaration wget_http_connection_receive_only().
* libwget/http.c: New function definition wget_http_connection_receive_only().
* libwget/http.h (struct wget_http_connection_st): Add member goaway.
* libwget/http2.c (struct http2_stream_context): Add member conn,
  (on_frame_recv_callback): Handle NGHTTP2_GOAWAY,
  (wget_http2_send_request): Initialize conn member.
* src/host.c (_release_job): Don't release parts if in 'done' state.
* src/wget.c (process_response_header): Remove handling of LINK headers,
  (process_response): Add handling of LINK headers,
  (downloader_thread): Handle pending responses correctly.

This patch fixes two related issues:
1. With HTTP/2, servers sometimes limit the number of requests per connection.
   Beforethis patch, the connection was closed too early after GOAWAY and pending
   responses weren't received.
2. In _release_job(), already downloaded parts of a file were erroneous released.
   This caused successful metalink and chunked downloads to fail in combination with
   a GOAWAY or a remotely closed connection.
2024-09-11 17:45:09 +02:00
4b2692120b * libwget/bar.c: Add new function wget_bar_write_line_ext() 2024-06-01 18:11:50 +02:00
188989e067 Update copyright year 2024-01-07 12:41:02 +01:00
15ba1db248 Update copyrights 2023-08-31 12:57:30 +02:00
3595963416 Add better default cert/bundle paths for Windows
* include/wget/wget.h: New function wget_ssl_default_cert_dir,
  new function wget_ssl_default_ca_bundle_path.
* libwget/Makefile.am: Add ssl.c.
* libwget/ssl.c: New file for common ssl/tls code.
* libwget/ssl_gnutls.c (tls_init): Initialize cert paths.
* libwget/ssl_openssl.c (tls_init): Likewise.
* libwget/ssl_wolfssl.c (tls_init): Likewise.
* src/options.c (struct config): Init ca_cert,
  (init): Allocate / dup config.ca_cert.

Co-authored-by: Tim Rühsen <tim.ruehsen@gmx.d
2023-08-18 12:40:27 +02:00
0f1abaa11c Fix alternate port requests to proxies
* include/wget/wget.h (wget_http_request_to_buffer): Add param 'port'.
* libwget/http.c (wget_http_request_to_buffer): Likewise,
  (wget_http_send_request): Call wget_http_request_to_buffer with
  connection port.

Copyright-paperwork-exempt: Yes
2023-07-09 12:28:39 +02:00
68a11b7287 Add CIDR support in 'no_proxy' for IPv6
* include/wget/wget.h: Fix http_get_no_proxy -> wget_http_get_no_proxy.
* libwget/http.c: New function cidr_v6_match(),
  check for ipv6 CIDRs in wget_http_match_no_proxy().
* unit-tests/test.c (test_match_no_proxy): Add IPv6 CIDR tests.
2023-06-17 23:50:53 +02:00
6ab47257f0 CIDR support added for no_proxy
* libwget/http.c: Add function wget_http_cidr_match to support
  CIDR IP method for no_proxy
* unit-tests/test.c: Add testcases to verify the functionality
  of the above function

Fixes #615
2023-05-22 17:46:02 +00:00
84d0220e22 New option --dane
* configure.ac: Enable DANE if libgnutls-dane is installed,
  add option --without-libdane.
* docs/wget2.md: Add docs for option --dane.
* include/wget/wget.h: Add WGET_SSL_DANE.
* libwget/ssl_gnutls.c: Include include <gnutls/dane.h> if requested,
  (struct config): New member 'dane',
  (wget_ssl_set_config_int): Set value for dane via WGET_SSL_DANE,
  (wget_ssl_open): Call dane_verify_session_crt().
* src/options.c (options): Add entry for --dane,
  (init): Set config.dane via wget_ssl_set_config_int().
* src/wget_options.h (struct config): Add member 'dane'.
2023-04-23 19:49:12 +02:00
7ab64e25af Change arguments of wget_iri_compare to const
* include/wget/wget.hi (wget_iri_compare): Add 'const' to arguments.
* libwget/iri.c (wget_iri_compare): Likewise.
2022-10-15 19:41:12 +02:00
b89af6d0b8 Introduce XML function to decode entities
* include/wget/wget.h: Add wget_xml_decode_entities_inline().
* libwget/iri.c (iri_unescape_inline): Remove decoding of XML entities.
* libwget/xml.c: Add wget_xml_decode_entities_inline().
* src/wget.c (normalize_uri): Call wget_xml_decode_entities_inline().
2022-09-02 19:51:52 +02:00
2661ab6d47 Support compilation of wget.h from C++
* include/wget/wget.h: Replace restrict with __restrict

Copyright-paperwork-exempt: Yes
2022-08-14 15:07:58 +02:00
9395dd4b79 Implement --check-certificate=quiet
* include/wget/wget.h: Add WGET_SSL_REPORT_INVALID_CERT macro
* libwget/ssl_gnutls.c: Change certificate related error_printf() calls, for error_printf_check()
* libwget/ssl_openssl.c: Consider the case WGET_SSL_REPORT_INVALID_CERT in wget_ssl_set_config_int()
* libwget/ssl_wolfssl.c: Change certificate related error_printf() calls, for error_printf_check()
* src/options.c: Admit quiet as a valid value on --check-certificate
* src/wget_options.h: Move check_certificate from bool to an enum type (with enabled, disabled and log_disabled as values)
2022-07-23 16:28:22 +02:00
3dc7f71098 Update copyright year 2022-02-25 17:46:43 +01:00
20a6046fca Fix issue reported by scan-build
* include/wget/wget.h: /// -> //.
* libwget/netrc.c (unescape_password): Check wget_malloc() return against NULL.
* libwget/xml.c (wget_xml_parse_file): Likewise.
2022-02-19 14:35:40 +01:00
01a6fe02e3 Fix request path escaping
* include/wget/wget.h: Remove function wget_iri_isunreserved_path().
* libwget/iri.c: Remove function wget_iri_isunreserved_path().
  (iri_ctype): Extend array with unreserved characters.
  (wget_iri_isunreserved): Simplify code.
  (wget_iri_escape): Use macro iri_isunreserved instead of wget_iri_isunreserved().
  (wget_iri_escape_path): Add RFC links to function comment,
  Fix check whether char needs percent-encoding or not,
  Use macro iri_isunreserved instead of wget_iri_isunreserved().
2021-05-16 14:29:26 +02:00
22162f82b2 Update copyright year 2021-01-22 21:58:38 +01:00
f270997432 * include/wget/wget.h (wget_hsts_host_match_fn): Fix argument names (cppcheck) 2020-03-17 16:49:35 +01:00
ef40fb5221 Parse HTML download attribute
* examples/print_html_urls.c: Print download attribute.
* include/wget/wget.h (wget_html_parsed_url): Add download member.
* libwget/html_url.c (html_get_url): Parse download attribute.
2020-03-15 11:53:09 +01:00
ddab633a5d Add new function wget_strmemcpy_a
* include/wget/wget.h: Add prototype of wget_strmemcpy_a.
* src/mem.c: Implement wget_strmemcpy_a.
* libwget/http_parse.c (wget_http_parse_header_line): Use wget_strmemcpy_a.
* src/utils.c (mkdir_path): Likewise.
2020-02-24 16:31:01 +01:00
473a4da2bc Let wget_strmemcpy return the number of copied bytes
* include/wget/wget.h: Set return type of wget_strmemcpy to size_t.
* libwget/mem.c (wget_strmemcpy): Return length of destination string.
* unit-tests/test.c: Amend tests of wget_strmemcpy.
2020-02-24 16:30:02 +01:00
91bba3f697 Add option --bind-interface
* libwget/net.h (wget_tcp_st): New variable
  const char *bind_interface.
* libwget/net.c: Add function wget_tcp_bind_interface().
  (set_socket_options): Add parameter wget_tcp *tcp and
  implement binding of tcp to bind_interface.
* libwget/init.c (wget_global_init): Call wget_tcp_bind_interface().
* include/wget/wget.h: Add wget_tcp_bind_interface() and
  WGET_BIND_INTERFACE.
* src/options.c: Add --bind-interface handling.
  (init): Call wget_tcp_bind_interface().
* src/wget2_options.h (config): Add const char *bind_interface.
* docs/wget2.md: Add documentation for --bind-interface.

This finishes task #430
2020-02-24 12:28:00 +01:00
50c8e3a0f1 * include/wget/wget.h: Make more use of 'restrict' keyword 2020-02-10 11:55:46 +01:00
59d9ecd9c0 * Update copyright year to 2020 2020-01-10 00:33:02 +01:00
792c35b780 * include/wget/wget.h: Fix old types in comments [skip ci] 2019-12-08 18:29:18 +01:00
6d47c6b0db Fix --cut-file-get-vars creating dirs on single file download
* tests/test-cut-get-vars.c: Amend to reproduce issue #490.
* src/blacklist.c (get_local_filename_real): Generate basename
  instead of filename, without query.
* include/wget/wget.h: Add flag WGET_IRI_WITH_QUERY.
  Add param 'flags' to wget_iri_get_basename.
* libwget/iri.c: Add param 'flags' to wget_iri_get_basename.
  Implement flag WGET_IRI_WITH_QUERY.
* fuzz/libwget_iri_fuzzer.c (test): Use WGET_IRI_WITH_QUERY.

Closes #490
2019-12-07 17:29:48 +01:00
f3e1d61a25 Rename wget_iri_get_filename -> wget_iri_get_basename
* fuzz/libwget_iri_fuzzer.c: Likewise.
* include/wget/wget.h: Likewise.
* libwget/iri.c: Likewise.
* src/blacklist.c: Likewise.
2019-12-07 15:40:53 +01:00
5c42026ee4 Add --ignore-length handling
* include/wget/wget.h (wget_http_request): Add bool
response_ignorelength
(wget_http_response): Add bool length_inconsistent
* src/options.c: Add --ignore-length handling
* src/wget_options.h (config): Add bool ignore_length
(exit_status_e): Add WGET_EXIT_INCONSISTENT_LENGTH
* libwget/http.c (wget_http_response_cb): Set length_inconsistent
when mismatch with Content-Length
* src/wget.c (wget_send_request): Set ignore-length in request
* tests/test-ignore-length.c: test --ignore-length
* tests/Makefile.am: add test-ignore-length
* docs/wget2.md: Removed [unimplemented-option] from
--ignore-length
2019-11-05 21:43:17 +05:30
003afe3b3b * include/wget/wget.h (wget_bar_slot_begin): Set 'filename' not NONNULL 2019-10-29 23:44:57 +01:00
149a811543 Fix type of 2nd param of wget_stringmap_iterator_next() to void **
* include/wget/wget.h: Likewise
* unit-tests/test.c: Likewise
2019-10-26 18:48:33 +02:00
53e8b14764 Make metalink->iri 'const'
* include/wget/wget.h: Make metalink->iri 'const'
* libwget/metalink.c (add_mirror): Code cleanup
* src/wget.c (establish_connection): Skip HTTP mirrors if HTTPS requested,
  (queue_url_from_local): Lock mutex a bit later
2019-10-26 18:48:33 +02:00
2ec7c7e290 Deduplicate code for creating robots.txt jobs
* src/wget.c (queue_url_from_local): Remove duplicated code,
  (queue_url_from_remote): Likewise,
  (set_file_metadata): Amend calling wget_iri_get_connection_part()
* include/wget/wget.h (wget_iri_parse_base): Add 'const' to 'base' param,
  (wget_iri_get_connection_part): Add 'const' to 'iri' param,
  add second param 'buf'
* fuzz/libwget_iri_fuzzer.c (test): Amend calling wget_iri_get_connection_part()
* libwget/iri.c (wget_iri_get_connection_part): Add param 'buf',
  add 'const' to 'iri' param, amend docs,
  (wget_iri_relative_to_abs): Add 'const' to 'base' param,
  amend calling wget_iri_get_connection_part(),
  (wget_iri_parse_base): Add 'const' to 'iri' param
* src/host.c (host_add_robotstxt_job): Add code removed in wget.c
* src/wget_host.h: Don't include wget_blacklist.h,
  amend signature of host_add_robotstxt_job()
* src/wget_job.h (struct JOB): Add 'const' to 'blacklist_entry'
2019-10-26 18:48:33 +02:00
d930fe3bc1 Moved WGET_E_GPG_* definitions out of libwget
* libwget/error.c (wget_strerror): Remove WGET_E_GPG_* symbols
* include/wget/wget.h (wget_error): Remove WGET_E_GPG_* symbol definitions
* src/wget_gpgme.h: New enum wget_gpg_error to hold WGET_E_GPG_* symbol definitions
* AUHOTRS: Add archpndy [at] gmail.com

Copyright-paperwork-exempt: Yes
2019-10-21 18:33:55 +05:30
193efa7556 * include/wget/wget.h: Add WGET_GCC_UNUSED_RESULT to wget_hashmap_get and wget_stringmap_get 2019-10-10 15:27:13 +02:00
77c3c33ec8 Use int64_t for time_t to fix 32bit time_t limitations
* include/wget/wget.h: Use int64_t instead of time_t
* libwget/hpkp.c: Likewise
* libwget/hsts.c: Likewise
* libwget/http_parse.c: Likewise
* libwget/ocsp.c: Likewise
* libwget/ssl_gnutls.c: Likewise
* libwget/tls_session.c: Likewise
* src/options.c: Likewise
* src/stats_site.c: Likewise
* src/wget.c: Likewise
* tests/libtest.c: Likewise
* tests/libtest.h: Likewise
* tests/test-plugin-dummy.c: Likewise
* unit-tests/test.c: Likewise
2019-09-23 21:17:38 +02:00
803589baff Fix http date functions to use int64_t instead of time_t
* include/wget/wget.h: Change time_t -> int64_t for wget_http_print_date()
  and for wget_http_parse_full_date()
* libwget/http_parse.c: Likewise
2019-09-23 15:34:03 +02:00
9329563387 Modify wget_decompress() 2nd arg to const
* include/wget/wget.h: Likewise
* libwget/decompressor.c: Likewise
2019-09-21 19:35:06 +02:00
3bd8e6aa97 Implement Accept-Encoding: lzip including decompressor
* README.md: Mention lzip
* configure.ac: Detect lzip
* docs/wget2.md: Document 'lzip'
* include/wget/wget.h: Add wget_content_encoding_lzip
* libwget/decompressor.c: Implement lzip decompression
* libwget/http_parse.c (wget_http_parse_content_encoding): Add "lzip"
* src/options.c: Add code to support lzip
* src/wget.c (http_create_request): Add lzip to Accept-Encoding: if available
* tests/test-compression.c: Add test vector for lzip decompression
2019-09-19 12:58:09 +02:00
44f3411f73 Change type to bool for wget_tcp_set_ssl/wget_tcp_get_ssl
* include/wget/wget.h: Likewise
* libwget/net.c: Likewise
2019-09-18 13:16:18 +02:00
7e41468e17 Use '!= 0' instead of 'git commit --amend' in libwget/ssl_gnutls.c
* include/wget/wget.h (wget_tls_stats_data): Change type of tfo and false_start to bool
* libwget/ssl_gnutls.c: Use bool in _session_context,
  remove use of !!
2019-09-18 13:12:16 +02:00
8d2acb05e4 Change return type of wget_tcp_get_tls_false_start() to bool
* include/wget/wget.h: Likewise
* libwget/net.c: Likewise
2019-09-18 13:07:16 +02:00
93245a5838 Change second param type of wget_tcp_set_tls_false_start() to bool
* include/wget/wget.h: Likewise
* libwget/net.c: Likewise
* src/wget_options.h (struct config): Change type of tls_false_start to bool
2019-09-18 13:05:50 +02:00
2013bd2168 Change return type of wget_tcp_get_tcp_fastopen() to bool
* include/wget/wget.h: Likewise
* libwget/net.c: Likewise
2019-09-18 13:02:46 +02:00
29d6777b57 Change second param type of wget_tcp_set_tcp_fastopen() to bool
* include/wget/wget.h: Likewise
* libwget/net.c: Likewise
* libwget/init.c (wget_global_init): Fix param type to wget_tcp_set_tcp_fastopen()
* src/wget_options.h (struct config): Change type of tcp_fastopen to bool
2019-09-18 12:59:43 +02:00
08d445e3f6 Use bool instead of char in HSTS code
* include/wget/wget.h: Change param to bool in wget_hsts_db_add_fn(),
  (struct wget_http_response_st): Use bool for hsts_include_subdomains and keep_alive,
  change param to bool in wget_http_parse_strict_transport_security(),
  change param to bool in wget_http_parse_connection()
* libwget/hsts.c: change param to bool in new_hsts(),
  change param to bool in wget_hsts_db_add()
* libwget/http_parse.c: change param to bool in wget_http_parse_strict_transport_security(),
  change param to bool in wget_http_parse_connection()
* tests/test-plugin-dummy.c: change param to bool in test_hsts_db_add()
* unit-tests/test.c (test_hsts): Change type of include_subdomains to bool
2019-09-18 12:44:26 +02:00
b8a2a8274b Change return type of wget_str_needs_encoding() to bool
* include/wget/wget.h: Likewise
* libwget/encoding.c: Likewise
2019-09-18 12:29:35 +02:00
5b32f680ee Use bool param for wget_cookie_set_keep_session_cookies()
* include/wget/wget.h: Change second param of wget_cookie_set_keep_session_cookies() to bool
* libwget/cookie.c: Likewise
* libwget/init.c (struct _CONFIG): Use bool for cookies_enabled, keep_session_cookies
2019-09-18 12:23:47 +02:00