Commit Graph

102 Commits

Author SHA1 Message Date
5687254219 Amend --http-proxy and --https-proxy behavior
* libwget/http.c (_parse_proxies): Alloc vector on demand
* src/options.c (wget_http_set_http_proxy): Return # of proxies
  successfully parsed,
  (wget_http_set_https_proxy). Likewise.
* src/options.c (init): Empty and undefined proxy variable is the same.
* tests/test.c (test_set_proxy): Amend tests.
2017-05-02 10:53:03 +02:00
569a646876 Remove unnecessary while loop & add test
* libwget/http.c (_parse_proxies): Remove non-required `while`
* tests/test.c (test_set_proxy): Add test with leading whitespaces
2017-05-02 10:39:57 +02:00
88cc3ead0f Add tests for wget_http_set_http(s)_proxy() functions
* tests/test.c (test_set_proxy): Add tests for wget_http_set_http_proxy()
* tests/test.c (test_set_proxy): Add tests for wget_http_set_https_proxy()
* libwget/http.c (wget_http_set_http_proxy): Return length of http_proxies if no parse error
* libwget/http.c (wget_http_set_https_proxy): Return length of https_proxies if no parse error
2017-05-02 10:36:11 +02:00
c6ee3d79ad Fix syntax-check 'sc_prohibit_have_config_h'
* cfg.mk: Remove sc_prohibit_have_config_h from local-checks-to-skip
* libwget/*.c: Include <config.h> unconditionally
* src/*.c: Likewise
* tests/*.c: Likewise
2017-04-30 22:01:34 +02:00
56efdab1a7 Remove trailing whitespaces (syntax-check) 2017-04-29 20:51:32 +02:00
bc06128232 Fix space-tab sequences (syntax-check) 2017-04-29 20:40:14 +02:00
8f0aa96537 Hide wget_cookie_st
* include/wget/wget.h: Remove struct wget_cookie_st declaration,
  add wget_cookie_to_setcookie(),
  add wget_cookie_parse_setcookie(),
  change wget_http_parse_setcookie() cookie param.
* libwget/cookie.c: Add struct wget_cookie_st declaration,
  (wget_cookie_init): Use xcalloc(),
  (wget_cookie_to_setcookie): New function,
  (wget_cookie_parse_setcookie): New function,
  (_wget_cookie_normalize_cookie): Fix host_only flag.
* libwget/http.c (wget_http_parse_setcookie): Code moved to
  wget_http_parse_setcookie(),
  (wget_http_parse_response_header): Amend cookie handling.
* tests/test.c (test_cookies): Amend test code and test data.
2017-04-20 16:02:18 +02:00
89a5245cfc * tests/test.c: Add wget_robots_parse unit test 2017-04-04 12:29:45 +02:00
7d2eb91b1c Skip IDNA tests when IDNA not available
* libwget/encoding.c: Fix #ifdef
* tests/test-idn-cmd.c: Skip if IDNA not available
* tests/test-idn-meta.c: Likewise
* tests/test-idn-robots.c: Likewise
* tests/test.c (test_iri_parse): Skip IDN test
2017-03-27 21:51:42 +02:00
0c6cf2a630 Fix .netrc parsing
* include/wget/wget.h (_wget_netrc_db_st): Rename .key to .host
* libwget/netrc.c (wget_netrc_db_add): Fix parsing
* tests/test.c: New function test_netrc()
2017-03-07 13:17:46 +01:00
7d8539eca0 * tests/test.c: Remove call of wget_hpkp_set_port() 2017-03-04 18:59:28 +01:00
e01926d2e2 Allow new 'expires' format for Set-Cookie header
* libwget/http.c (wget_http_parse_full_date): Allow format
  '1 Mar 2027 09:23:12 GMT'
* tests/test.c (test_cookies): Test new format
2017-03-03 10:45:01 +01:00
ec396c577f Fix URLs to HTTPS where possible 2017-02-28 15:31:30 +01:00
364ebfd4b2 Remove HPKP entries with zero PINs
* libwget/hpkp.c (wget_hpkp_db_check_pubkey): Fix subdomain default,
  (wget_hpkp_db_add): Remove entry on zero PINs
* src/wget.c: Remove comment line
* tests/test.c (test_hpkp): Test this feature
2017-02-22 13:16:40 +01:00
2b448ffa92 Fix memleak in HPKP
* include/wget/wget.h: Changed declaration of wget_hpkp_free()
* libwget/hpkp.c: Use wget_hpkp_free() as destructor function
* libwget/http.c (wget_http_free_hpkp_entries): Amended
* tests/test.c: Fix typos
2017-02-22 13:16:40 +01:00
b991db695f * tests/test.c: Add unit testing for HPKP 2017-02-22 13:16:40 +01:00
295fbdaf80 * tests/test.c (test_buffer): Add more tests 2017-02-21 17:03:26 +01:00
0a9ee00db9 Fix possible file name truncation
* tests/test-parse-html.c: Increase buffer for filename
* tests/test.c: Likewise
2017-02-01 11:19:52 +01:00
bfcd65c12b Use typedefs for function pointer arguments
* include/wget/wget.h: Add typedefs
* libwget/cookie.c: Use typedefs
* libwget/css.c: Likewise
* libwget/css_url.c: Likewise
* libwget/decompressor.c: Likewise
* libwget/hashmap.c: Likewise
* libwget/hsts.c: Likewise
* libwget/http.c: Likewise
* libwget/init.c: Likewise
* libwget/io.c: Likewise
* libwget/list.c: Likewise
* libwget/logger.c: Likewise
* libwget/metalink.c: Likewise
* libwget/net.c: Likewise
* libwget/netrc.c: Likewise
* libwget/ocsp.c: Likewise
* libwget/private.h: Likewise
* libwget/robots.c: Likewise
* libwget/stringmap.c: Likewise
* libwget/tls_session.c: Likewise
* libwget/vector.c: Likewise
* libwget/xml.c: Likewise
* src/blacklist.c: Likewise
* src/host.c: Likewise
* src/options.c: Likewise
* src/wget.c: Likewise
* tests/stringmap_perf.c: Likewise
* tests/test.c: Likewise
2017-01-23 14:43:17 +01:00
8b1a701c21 Use wget_strdup() instead of strdup()
* examples/getstream.c: Use wget_strdup().
* libwget/cookie.c: Likewise.
* libwget/encoding.c: Likewise.
* libwget/http.c: Likewise.
* libwget/iri.c: Likewise.
* libwget/netrc.c: Likewise.
* src/options.c: Likewise.
* src/wget.c: Likewise.
* tests/test.c: Likewise.
2016-11-03 09:58:09 +01:00
b049eb6d3a Fix gcc/clang warnings from the new manywarnings feature
* configure.ac: Suppress -Wunsuffixed-float-constants for gcc,
  add -Wno-disabled-macro-expansion to clang flags
* examples/getstream.c: Removed 'unused' attributes from argc and argv.
* include/wget/wget.h: Fix slot to int for bar functions.
* libwget/bar.c: Fix slot to int for bar functions, cast to int,
  change type of bitfield 'redraw' to unsigned.
* libwget/hsts.c (wget_hsts_new): Fix int to bitfield conversion.
* libwget/http.c: Cast printf %p format to (void *).
* src/wget.c: Cast printf %p format to (void *),
  remove ACTION_DONE from enum actions.
* src/wget_host.h: Fix double declaration of host_remove_job()
* tests/libtest.c (wget_test_check_filesystem): Fix type of 'rc'.
* tests/test-idn-robots.c: Fix comma to semikolon at end of line.
* tests/test.c (test_strcasecmp_ascii): Fix type of loop variable.
2016-11-02 22:19:41 +01:00
e10cf1d1f3 Move window size detection to libwget
* include/wget/wget.h: Declare new function wget_determine_screen_width.
Also change prototype of wget_bar_init to not accept a max_width
parameter
* src/utils.c: Move function determine_screen_width from here ...
* libwget/utils.c: ... to here.
* src/utils.{c,h}: Delete empty file
* src/Makefile.am: Remove source files utils.{c,h}
* src/bar.c: Move detection of screen width from here ...
* libwget/bar.c: ... to here
* tests/test.c: wget_bar_init() no longer takes a max_width parameter
2016-10-04 20:29:30 +02:00
9568b0c87e Rename include/libwget.h to include/wget/wget.h+wgetver.h 2016-09-30 09:47:32 +02:00
4ac70bf7c3 * tests/test.c: Add wget_bar_* unit test 2016-09-29 11:53:04 +02:00
e3a314d2ab Fix unescaping of '+' in URL
* libwget/iri.c (wget_iri_unescape_inline): Do not touch '+'.
  (wget_iri_parse): Replace '+' by ' ' in query part.
* tests/test.c (test_iri_parse): Add more tests
2016-09-27 11:32:34 +02:00
10d2091795 * tests/test.c: Fix cookie tests for when libpsl not available 2016-08-28 12:30:34 +02:00
667127e193 Support cookie prefixes __Secure- and __Host-
* libwget/cookie.c (_wget_cookie_normalize_cookie): Add checks
  for __Secure- and __Host- prefixes.
* tests/test.c (test_cookies): Add test cases for the prefixes.

This change implements proposals from draft-ietf-httpbis-cookie-prefixes-00
as modern browsers do.
2016-08-28 00:01:19 +02:00
9ca39a7dbd Make compiler happy about using uninitialized vars
* tests/test.c(test_striconv): Explcitly initialize variables to
    NULL to make compiler happy.
2016-07-28 20:57:48 +02:00
79dd277c12 Refactoring to separate send, receive and HTTP state machine
* examples/websequencediagram.c (main): Call wget_http_request_set_body()
  and wget_http_send_request() instead of wget_http_send_request_with_body().
* include/libwget.h.in: Add WGET_HTTP_USER_DATA, wget_get_timemillis(),
  extend wget_thread_cond_wait(), add body, user_data, body_length to
  wget_http_request_t, remove wget_http_send_request_with_body(),
  add wget_http_request_get_int(), wget_http_request_set_ptr(),
  wget_http_request_get_ptr(), wget_http_request_set_body().
* libwget/http.c: Add wget_http_request_get_int(),
  wget_http_request_set_ptr(), wget_http_request_get_ptr(),
  wget_http_request_set_body(),
  remove wget_http_send_request_with_body(),
  (wget_http_request_to_buffer): add body to request buffer.
* libwget/http_highlevel.c (wget_http_get): Replace
  wget_http_send_request_with_body()
* libwget/iri.c: Use c-ctype.h instead of ctype.h
  (wget_iri_parse): Allow any numbers of / after scheme:
  (wget_iri_parse): Catch URIs without /after scheme:
* libwget/list.c (wget_list_browse): Small code rearrangement
* libwget/metalink.c (_add_mirror): Check mirror.iri for NULL
* libwget/ssl_gnutls.c (send_ocsp_request): Replace
  wget_http_send_request_with_body()
* libwget/thread.c (wget_thread_cond_signal): Add timeout param
* libwget/utils.c: New function wget_get_timemillis()
* src/blacklist.c: Include wget.h instead of log.h
* src/blacklist.h: Fix indentation
* src/host.c: Add queueing stuff
* src/host.h: Reflect changes in host.c
* src/job.c: Remove queueing stuff
* src/job.h: Reflect changes in job.c
* src/log.c: Sync stdout/stderr to correct output order
* src/log.h: Remove shortcuts of print functions
* src/wget.c: Remove download_part() and http_get().
  Add http_send_request(), http_receive_response(), try_connection(),
  establish_connection(), add_statistics(), process_response_header().
  Amend downloader_thread() to reflect the changes.
* src/wget.h: Add shortcut defines for print functions.
* tests/libtest.c (_http_server_thread): Fix compiler warning,
  fix debug message.
  New function _write_msg() to print server messages yellow.
  (wget_test) Add -d to wget command line.
* tests/test-metalink.c (main): Add tests for V3 and V4 metalink
  files read from command line (-i, --force-metalink)
* tests/test.c (test_iri_parse): Add test for slash-less mailto: URI
2016-07-11 14:53:36 +02:00
9cff7dd096 Fix printf format warnings
* tests/buffer_printf_perf.c: Fix printf format warnings
* tests/stringmap_perf.c: Likewise
* tests/test.c: Likewise
2016-06-14 15:55:53 +02:00
483c304aef Support parsing of UTF-16 HTML documents
* examples/print_html_urls.c (html_parse_localfile):
  Detect BOM (Byte Order Mark), convert UTF-16 to UTF-8 before parsing.
* include/libwget.h.in: Remove wget_charset_transcode(),
  add wget_memiconv() and wget_striconv().
* libwget/encoding.c: Implement wget_memiconv() and wget_striconv()
* src/wget.c (html_parse): Add 'html_len' param,
  convert UTF-16 to UTF-8 before parsing.
* tests/test.c: New test for wget_memiconv().
2016-04-24 15:08:20 +02:00
385e0d52b3 Allow options and URLs in any order
* src/options.c (parse_command_line): Reorder URLs after options
* src/options.h: Fix init() prototype
* src/wget.c, tests/test-parse-html.c, tests/test.c:
  Fix main() argv argument
2016-03-28 21:58:10 +02:00
dd3c2f63b2 Updated copyright year for all relevant files 2016-01-25 13:06:21 +01:00
0ecbd6e69a Removed redundant buffer print routines
Removed wget_buffer_vprintf_append, wget_buffer_printf_append,
wget_buffer_vprintf2 and wget_buffer_printf2.
Renamed their *2 counterpart to their old names.
2016-01-13 15:01:43 +01:00
9232f787fc Do not print responses in tests
* tests/test.c (check_cookie): Do not print responses. We want
  this test to expose memory leaks, but printing the response of
  wget_cookie_create_request_header isn't needed.
2016-01-08 11:24:04 +01:00
d2f20fd787 Fix hashing with GnuTLS
* libwget/ssl_gnutls.c: Do not skip automatic library initialization
* libwget/md5.c (wget_md5_printf_hex): Print message on error
* tests/test.c: New tests for hashing functionality

GnuTLS automatically initializes on application startup.
This wastes CPU cycles when not using HTTPS.
Switched off, simple functionality like MD5 fail (needed for metalink and
authorization). We have to implement hashing via -lcrypt or -lgcrypt
later - and than we could  use GNUTLS_SKIP_GLOBAL_INIT again.
2016-01-06 12:53:56 +01:00
954136e2df Do not percent unescape query
* tests/test.c: Test query parsing in API
* tests/test-base.c: Test query handling of wget2
* include/libwget.h: Add flags to wget_iri_t,
  add function wget_iri_unescape_inline()
* libwget/iri.c: Add function wget_iri_unescape_inline(),
  fix wget_iri_parse() to not percent unescape query
  do not add fragment in wget_iri_get_escaped_resource()
2016-01-06 11:27:07 +01:00
0eb71df322 Fix tests/test.c for non-GNU standard C libraries
* tests/test.c (test_buffer_printf): Skip left zero padding strings
2015-11-23 10:25:39 +01:00
97496182e9 Adjust HSTS unit tests to latest changes
* tests/test.c (test_hsts): Adjust port 80 test, new port 8080 test
2015-11-06 20:58:30 +01:00
ff9db5a03f Exit immediately if cannot set http_proxy or https_proxy
* src/wget.c (main): Exit immediately on an "init()" error.
* src/options.c (init): Raise an error if cannot set http_proxy or https_proxy.
* tests/test-parse-html.c (main): Check for "init" errors.
* tests/test.c (main): Check for "init" errors.
2015-10-23 20:22:41 +02:00
0f8e49128a Transfer copyright to Free Software Foundation, Inc. 2015-09-22 11:50:06 +02:00
c6b0e461a1 Transform Mget into Wget 2015-09-19 22:54:38 +02:00
abf38ba748 Fixed cookie test routine 2015-09-13 21:38:19 +02:00
1d45d9a704 changes in hashmap handling 2015-03-08 21:44:01 +01:00
6cdd208444 fix several issues found by Coverity 2015-03-07 22:55:17 +01:00
e2e12530ee add comment regarding Solaris sprintf behaviour 2015-02-27 09:31:07 +01:00
42bc281945 fixed Mget return value for HTTPS failures 2015-02-24 10:52:22 +01:00
9b980fbdf7 added more tests for parsing challenges 2014-12-21 23:03:02 +01:00
7ae1e53c0a added test for WWW-Authenticate challenge parsing 2014-12-18 21:46:16 +01:00
a11ceba71e changed valgrind testing 2014-10-10 15:20:06 +02:00