Commit Graph

220 Commits

Author SHA1 Message Date
6246345ef7 Remove nonnull attribute from some functions
* include/wget/wget.h: Remove nonnull attribute from some functions
* libwget/encoding.c: Add explicit checks for NULL
* libwget/io.c: Likewise
* libwget/pipe.c: Likewise
* libwget/strlcpy.c: Likewise
* libwget/utils.c: Likewise
2017-05-15 10:28:03 +02:00
e0b3dc1d26 Make wget_http_connection_t an abstract type
* include/wget/wget.h: Convert wget_http_connection_t to forward declaration
 * libwget/http.h: Move wget_http_connection_t definition here
 * libwget/Makefile.am: Add new file http.h
 * libwget/http.c: Include http.h and add accessor functions
 * libwget/http_highlevel.c: Include http.h
 * src/wget.c: Replace access to wget_http_connection_t with accessor function calls
2017-05-03 15:37:50 +02:00
7c1bebf9a0 Add new function wget_free()
* examples/check_url_types.c: Use wget_xfree() instead of free()
* examples/http_get2.c: Likewise
* examples/print_css_urls2.c: Likewise
* examples/print_html_urls.c: Likewise
* include/wget/wget.h: Add wget_free() and define wget_xfree()
* libwget/xalloc.c: Implement wget_free()

* include/wget/wget.h: Add declaration for wget_free()
* libwget/xalloc.c: Implement wget_free()
2017-05-03 14:47:04 +02:00
8a93319b47 Rename strlcpy to wget_strlcpy
* include/wget/wget.h: Rename and remove gnulib deps.
* libwget/strlcpy.c: Rename function and fall back to strlcpy.
* */*.c: Rename strlcpy to wget_strlcpy.
2017-05-02 11:36:51 +02:00
978036a931 Remove useless parens (syntax-check)
* include/wget/wget.h: Remove useless parens
* libwget/ssl_gnutls.c: Likewise
2017-04-30 21:41:25 +02:00
b3aede0e57 Remove gnulib dependencies from examples and wget.h
* cfg.mk: Add examples as excludes for config.h
* examples/Makefile.am: Remove -I to srcdir and builddir
* examples/*.c: Remove #include <config.h>
* include/wget/wget.h: Remove gettext include and defines
* libwget/private.h: Add gettext include and defines
* src/wget_main.h: Likewise
* tests/libtest.h: Likewise
* src/log.c: Add #include <string.h>
2017-04-30 21:37:39 +02:00
59f163cd7b Remove useless cpp parens (syntax-check) 2017-04-29 20:56:57 +02:00
2d7fe3e9fc Prevent tiny memory allocations when globbing
* libwget/utils.c: Rename wget_strnglob to wget_strglob. It now calls
glob on the entire string.
* include/wget/wget.h: Change protoype of wget_strnglob
* src/options.c: Change calls to wget_strnglob to use new wget_strglob
calls
2017-04-27 18:31:15 +02:00
d46ecad443 Add error handler for decompression
* include/wget/wget.h: New functions wget_decompress_set_error_handler()
  and wget_decompress_get_context().
* libwget/decompressor.c: New functions wget_decompress_set_error_handler()
  and wget_decompress_get_context().
* libwget/http.c: Implement and set decompressor error handler.
2017-04-26 12:21:25 +02:00
8f0aa96537 Hide wget_cookie_st
* include/wget/wget.h: Remove struct wget_cookie_st declaration,
  add wget_cookie_to_setcookie(),
  add wget_cookie_parse_setcookie(),
  change wget_http_parse_setcookie() cookie param.
* libwget/cookie.c: Add struct wget_cookie_st declaration,
  (wget_cookie_init): Use xcalloc(),
  (wget_cookie_to_setcookie): New function,
  (wget_cookie_parse_setcookie): New function,
  (_wget_cookie_normalize_cookie): Fix host_only flag.
* libwget/http.c (wget_http_parse_setcookie): Code moved to
  wget_http_parse_setcookie(),
  (wget_http_parse_response_header): Amend cookie handling.
* tests/test.c (test_cookies): Amend test code and test data.
2017-04-20 16:02:18 +02:00
9f2dd3a199 Fix spelling issues found by ka7/misspell_fixer
* include/wget/wget.h: Fix typo
* libwget/base64.c: Likewise
* libwget/http.c: Likewise
* src/wget.c: Likewise
2017-04-19 11:29:55 +02:00
6214291e3b Fixed crash in convert-links code introduced with html css parsing
* include/wget/wget.h: Removed unused variable.
* libwget/html_url.c: Updated code to always return pointer inside "html" variable.
* test/test-k.c: Updated test to cover the bug.
2017-04-13 19:58:10 +02:00
31762cc633 Added support for html css style tag and attribute
* include/wget/wget.h: Add 'size_t len' to wget_css_parse_buffer()
  and wget_css_get_urls()
* libwget/css.c (wget_css_parse_buffer): Use yy_scan_bytes() instead
  of yy_scan_string()
* libwget/css_url.c: Add param to wget_css_get_urls() and
  wget_css_parse_buffer()
* libwget/html_url.c: Add _css_parse_encoding() and _css_parse_uri(),
  (_html_get_url): Deal with 'style' attribute,
  (wget_html_free_urls_inline): Free url if needed
* libwget/test_linking.c: Add param to wget_css_parse_buffer()
* libwget/xml.c (parseXML): Add check for 'style'
* src/wget.c (css_parse): Add param 'size_t len'
* tests/Makefile.am: Add new test test-parse-html-css
* tests/test-parse-html-css.c: New file
2017-04-13 10:06:25 +02:00
bc4010fa39 Fix progress bar when off_t is 32 bits
* include/wget/wget.h: Amend prototype of wget_human_readable()
* libwget/bar.c: Use uint64_t instead of off_t
* libwget/utils.c (wget_human_readable): Change 3rd param off_t -> uint64_t
2017-04-12 11:48:15 +02:00
1804929290 * include/wget/wget.h: Silence -Wundef from gnulib's gettext.h 2017-04-06 10:53:01 +02:00
d31de86454 Add support for 'no_proxy' environment variable
* include/wget/wget.h: Add flag 'proxied' to wget_http_connection_t,
  add arg 'proxied' to wget_http_request_to_buffer(),
  add new function declaration wget_http_set_no_proxy(),
  add new function declaration wget_http_match_no_proxy().
* libwget/http.c: Add variable 'no_proxies',
  (wget_http_open): Check host against 'no_proxy' and set 'conn->proxied'
  when appropriate,
  (wget_http_send_request): Call wget_http_request_to_buffer() with
   conn->proxied,
  (wget_http_request_to_buffer): Check 'proxied',
  (_parse_proxies): Cleanup,
  (_parse_no_proxies): New function,
  (wget_http_set_no_proxy): New function,
  (wget_http_match_no_proxy): New function
* libwget/init.c (wget_global_deinit): Free no_proxy memory
* libwget/ip.c: Add code in comments (for later use)
* libwget/utils.c (wget_match_tail, wget_match_tail_nocase): Cleanup
* src/options.c (init): Handle 'no_proxy' env variable
* src/wget_options.h (struct config): Add no_proxy member
2017-04-05 11:37:33 +02:00
bf5671c514 Use time_t instead of long
* include/wget/wget.h : Use time_t instead of long

Reported-by : Gisle Vanem
2017-04-04 14:25:58 +05:30
4d7238e282 Fix vpath builds
* examples/Makefile.am: Add -DWGETVER_FILE to AM_CPPFLAGS
* libwget/Makefile.am: Likewise
* src/Makefile.am: Likewise
* tests/Makefile.am: Likewise
* include/wget/wget.h: Include WGETVER_FILE if defined
2017-03-25 20:49:41 +01:00
13371152f8 Fix console colors for Windows
* include/wget/wget.h: Add WGET_CONSOLE_COLOR_MAGENTA
* libwget/console.c: Add WGET_CONSOLE_COLOR_MAGENTA
* src/log.c: Add colors to write functions

Reported-by: Gisle Vanem
2017-03-21 12:54:54 +01:00
a6900631fe Check NULL value param in wget_iri_escape()
* include/wget/wget.h: Remove G_GNUC_WGET_NONNULL_ALL from wget_iri_escape
* libwget/iri.c (wget_iri_escape): Check 'src' for NULL value
2017-03-10 17:12:25 +01:00
0c6cf2a630 Fix .netrc parsing
* include/wget/wget.h (_wget_netrc_db_st): Rename .key to .host
* libwget/netrc.c (wget_netrc_db_add): Fix parsing
* tests/test.c: New function test_netrc()
2017-03-07 13:17:46 +01:00
79667f113d Remove port from HPKP code
* include/wget/wget.h: Remove wget_hpkp_set_port()
* libwget/hpkp.c (struct _wget_hpkp_st): Remove 'port'
* src/wget.c (process_response_header): Do not call wget_hpkp_set_port()
2017-03-04 18:32:22 +01:00
7fd917e94e Exclude some URLs from recursive download with -p.
* include/wget/wget.h: Add link_inline to WGET_HTML_PARSED_URL
* libwget/html_url.c: Add link_inline to _html_context_t,
  (_html_get_url): Set ctx->link_inline
* src/wget.c (html_parse): Exclude action and formaction URLs,
  exclude certain 'link' URLs from download
2017-02-28 11:35:09 +01:00
5b2888ac9c Add brotli decompression, Accept-Encoding 'br' 2017-02-24 12:59:55 +01:00
c545263d54 Support tilde expansion of filenames from wgetrc
* src/options.c (_shell_expand): New function to perform tilde
expansion on a given string
(parse_filename): Parse the input string and perform shell expansion on
it using _shell_expand
(parse_string): Set the parser functions for various options that may be
set from a config file to use parse_filename
(wget_strnglob): Run glob() on only the specified part of the string
2017-02-22 16:28:10 +01:00
2b448ffa92 Fix memleak in HPKP
* include/wget/wget.h: Changed declaration of wget_hpkp_free()
* libwget/hpkp.c: Use wget_hpkp_free() as destructor function
* libwget/http.c (wget_http_free_hpkp_entries): Amended
* tests/test.c: Fix typos
2017-02-22 13:16:40 +01:00
68b8f02fab Implement HPKP checking
* include/wget/wget.h: Add function wget_hpkp_db_check_pubkey(),
  add WGET_SSL_HPKP_CACHE
* libwget/hpkp.c: Implement wget_hpkp_db_check_pubkey()
* libwget/ssl_gnutls.c: Implement and use _cert_verify_hpkp()
* src/options.c (init): Set WGET_SSL_HPKP_CACHE
2017-02-22 13:16:40 +01:00
afc4d5a1af HPKP API redesign 2017-02-22 13:16:40 +01:00
e6a73ffa99 * include/wget/wget.h: Fix HPKP function declarations 2017-02-22 13:16:39 +01:00
0cd52c7cce * include/wget/wget.h: Use WGETAPI for hpkp function declarations 2017-02-22 13:16:34 +01:00
89fed7e4db HPKP: Fix memory leaks
* src/wget.c (main, process_response_header): add debug traces
 * libwget/http.c (wget_http_parse_public_key_pins): fix memory leaks
 * libwget/hpkp.c (__wget_hpkp_free, wget_hpkp_db_load): fix memory
   leaks
2017-02-22 13:13:54 +01:00
86cb1180b6 HPKP: Define return values. 2017-02-22 13:13:54 +01:00
4d263eb628 HPKP: use vector rather than linked list for b64 pins 2017-02-22 13:13:54 +01:00
5a3a418705 HPKP: hpkp complete 2017-02-22 13:13:54 +01:00
33e01c5dfa HPKP: fix wget_hpkp_db_save()
* src/wget.c (main): call wget_hpkp_db_save() at the end (params
   inverted).
 * include/wget/wget.h: export wget_hpkp_db_save().
 * libwget/hpkp.c (__hashtable_browse_cb): fix fprintf format string.
   (wget_hpkp_db_save): check filename is not an empty string.
   (wget_hpkp_db_load): likewise.
2017-02-22 13:13:54 +01:00
584b15e67e Prototype HPKP functions
* include/wget/wget.h: new public function wget_hpkp_db_init().
 * libwget/hpkp.c: new file.
2017-02-22 12:46:01 +01:00
5ce41960e5 Merge from upstream/master
Conflicts: libwget/Makefile.am
2017-02-22 12:46:01 +01:00
57b259e868 HPKP boilerplate
* libwget/Makefile.am: new file hpkp.c.
 * src/options.c (print_help, struct options): new command line option '--hpkp'.
   (init): initialize HPKP database with wget_hpkp_db_init().
 * src/wget_options.h (struct config): new handle 'hpkp' to the HPKP
   database.
2017-02-22 12:46:01 +01:00
874a209fb4 HPKP: initial commit 2017-02-22 12:46:01 +01:00
0fe5f9282d New function wget_logger_is_active()
* include/wget/wget.h: Add prototype
* libwget/Makefile.am: Add logger.h
* libwget/log.c: Include logger.h
* libwget/logger.c: Include logger.h, add wget_logger_is_active()
* libwget/net.c: Use wget_logger_is_active()
* libwget/private.h: Remove struct _wget_logger_st
2017-01-23 16:55:52 +01:00
bfcd65c12b Use typedefs for function pointer arguments
* include/wget/wget.h: Add typedefs
* libwget/cookie.c: Use typedefs
* libwget/css.c: Likewise
* libwget/css_url.c: Likewise
* libwget/decompressor.c: Likewise
* libwget/hashmap.c: Likewise
* libwget/hsts.c: Likewise
* libwget/http.c: Likewise
* libwget/init.c: Likewise
* libwget/io.c: Likewise
* libwget/list.c: Likewise
* libwget/logger.c: Likewise
* libwget/metalink.c: Likewise
* libwget/net.c: Likewise
* libwget/netrc.c: Likewise
* libwget/ocsp.c: Likewise
* libwget/private.h: Likewise
* libwget/robots.c: Likewise
* libwget/stringmap.c: Likewise
* libwget/tls_session.c: Likewise
* libwget/vector.c: Likewise
* libwget/xml.c: Likewise
* src/blacklist.c: Likewise
* src/host.c: Likewise
* src/options.c: Likewise
* src/wget.c: Likewise
* tests/stringmap_perf.c: Likewise
* tests/test.c: Likewise
2017-01-23 14:43:17 +01:00
315c8f7516 base46: Use size_t as string length param 2017-01-20 16:23:02 +01:00
dfc4b53eae Add typedef for wget_vector_browse() callback function
* include/wget/wget.h: Add typedef wget_list_browse_cb_t
* libwget/list.c: Use wget_list_browse_cb_t
2017-01-19 17:11:58 +01:00
566e9f7c70 Add typedef for wget_update() callback functions
* include/wget/wget.h: Add typedef wget_update_cb_t
* libwget/cookie.c: Use wget_update_cb_t
* libwget/hsts.c: Likewise
* libwget/tls_session.c: Likewise
2017-01-19 17:11:58 +01:00
8248bfecd1 Add console color abstraction
* docs/Makefile.am: Add libwget-console.3
* include/wget/wget.h: Add console routines
* libwget/Makefile.am: Add libwget/console.c
* libwget/console.c: New file with console functions
* src/log.c: Add WIN32 and console function calls
2017-01-06 14:23:37 +01:00
1f39794852 Add function wget_truncate()
* bootstrap.conf: Ad gnulib module ftruncate
* include/wget/wget.h: Add wget_truncate
* libwget/io.c: Add wget_truncate
* src/job.c: Call wget_truncate instead of truncate

The truncate() function is not covered by gnulib.
wget_truncate() is a wrapper around ftruncate(), which is covered
by gnulib.
2017-01-06 12:14:40 +01:00
042065d44f Let MSVC compile libwget/init.c (wget_global_init)
Reported-by: Gisle Vanem
2017-01-05 22:13:46 +01:00
0354295e03 Add network init function to bootstrap sockets on Windows
* bootstrap.conf: Add gnulib module 'sockets'
* examples/http_get2.c: Call wget_net_init()
* include/wget/wget.h: Add wget_net_init() and wget_net_deinit()
* libwget/init.c (wget_global_init): Call wget_net_init(),
  (wget_global_deinit): Call wget_net_deinit()
* libwget/net.c: Add wget_net_init() and wget_net_deinit()
2017-01-05 12:09:25 +01:00
43f30f309e Rename wget_str_[v]asprintf to wget_[v]aprintf
* include/wget/wget.h: Rename wget_str_[v]asprintf to wget_[v]aprintf
* libwget/http.c: Likewise
* libwget/iri.c: Likewise
* libwget/printf.c: Likewise
* libwget/vector.c: Likewise
* src/options.c: Likewise
* tests/test-metalink.c: Likewise
2016-12-21 11:54:36 +01:00
909afd75fc Make the visibiity attribute preceed the prototype
* include/wget/wget.h: Make library symbol export attribute
((__visibility__)) preceed the function prototype to work on Windows
with MSVC
* tests/libtest.h: Same

Reported-by: Gisle Vanem
2016-11-16 12:03:27 +01:00