Commit Graph

208 Commits

Author SHA1 Message Date
31762cc633 Added support for html css style tag and attribute
* include/wget/wget.h: Add 'size_t len' to wget_css_parse_buffer()
  and wget_css_get_urls()
* libwget/css.c (wget_css_parse_buffer): Use yy_scan_bytes() instead
  of yy_scan_string()
* libwget/css_url.c: Add param to wget_css_get_urls() and
  wget_css_parse_buffer()
* libwget/html_url.c: Add _css_parse_encoding() and _css_parse_uri(),
  (_html_get_url): Deal with 'style' attribute,
  (wget_html_free_urls_inline): Free url if needed
* libwget/test_linking.c: Add param to wget_css_parse_buffer()
* libwget/xml.c (parseXML): Add check for 'style'
* src/wget.c (css_parse): Add param 'size_t len'
* tests/Makefile.am: Add new test test-parse-html-css
* tests/test-parse-html-css.c: New file
2017-04-13 10:06:25 +02:00
bc4010fa39 Fix progress bar when off_t is 32 bits
* include/wget/wget.h: Amend prototype of wget_human_readable()
* libwget/bar.c: Use uint64_t instead of off_t
* libwget/utils.c (wget_human_readable): Change 3rd param off_t -> uint64_t
2017-04-12 11:48:15 +02:00
1804929290 * include/wget/wget.h: Silence -Wundef from gnulib's gettext.h 2017-04-06 10:53:01 +02:00
d31de86454 Add support for 'no_proxy' environment variable
* include/wget/wget.h: Add flag 'proxied' to wget_http_connection_t,
  add arg 'proxied' to wget_http_request_to_buffer(),
  add new function declaration wget_http_set_no_proxy(),
  add new function declaration wget_http_match_no_proxy().
* libwget/http.c: Add variable 'no_proxies',
  (wget_http_open): Check host against 'no_proxy' and set 'conn->proxied'
  when appropriate,
  (wget_http_send_request): Call wget_http_request_to_buffer() with
   conn->proxied,
  (wget_http_request_to_buffer): Check 'proxied',
  (_parse_proxies): Cleanup,
  (_parse_no_proxies): New function,
  (wget_http_set_no_proxy): New function,
  (wget_http_match_no_proxy): New function
* libwget/init.c (wget_global_deinit): Free no_proxy memory
* libwget/ip.c: Add code in comments (for later use)
* libwget/utils.c (wget_match_tail, wget_match_tail_nocase): Cleanup
* src/options.c (init): Handle 'no_proxy' env variable
* src/wget_options.h (struct config): Add no_proxy member
2017-04-05 11:37:33 +02:00
bf5671c514 Use time_t instead of long
* include/wget/wget.h : Use time_t instead of long

Reported-by : Gisle Vanem
2017-04-04 14:25:58 +05:30
4d7238e282 Fix vpath builds
* examples/Makefile.am: Add -DWGETVER_FILE to AM_CPPFLAGS
* libwget/Makefile.am: Likewise
* src/Makefile.am: Likewise
* tests/Makefile.am: Likewise
* include/wget/wget.h: Include WGETVER_FILE if defined
2017-03-25 20:49:41 +01:00
13371152f8 Fix console colors for Windows
* include/wget/wget.h: Add WGET_CONSOLE_COLOR_MAGENTA
* libwget/console.c: Add WGET_CONSOLE_COLOR_MAGENTA
* src/log.c: Add colors to write functions

Reported-by: Gisle Vanem
2017-03-21 12:54:54 +01:00
a6900631fe Check NULL value param in wget_iri_escape()
* include/wget/wget.h: Remove G_GNUC_WGET_NONNULL_ALL from wget_iri_escape
* libwget/iri.c (wget_iri_escape): Check 'src' for NULL value
2017-03-10 17:12:25 +01:00
0c6cf2a630 Fix .netrc parsing
* include/wget/wget.h (_wget_netrc_db_st): Rename .key to .host
* libwget/netrc.c (wget_netrc_db_add): Fix parsing
* tests/test.c: New function test_netrc()
2017-03-07 13:17:46 +01:00
79667f113d Remove port from HPKP code
* include/wget/wget.h: Remove wget_hpkp_set_port()
* libwget/hpkp.c (struct _wget_hpkp_st): Remove 'port'
* src/wget.c (process_response_header): Do not call wget_hpkp_set_port()
2017-03-04 18:32:22 +01:00
7fd917e94e Exclude some URLs from recursive download with -p.
* include/wget/wget.h: Add link_inline to WGET_HTML_PARSED_URL
* libwget/html_url.c: Add link_inline to _html_context_t,
  (_html_get_url): Set ctx->link_inline
* src/wget.c (html_parse): Exclude action and formaction URLs,
  exclude certain 'link' URLs from download
2017-02-28 11:35:09 +01:00
5b2888ac9c Add brotli decompression, Accept-Encoding 'br' 2017-02-24 12:59:55 +01:00
c545263d54 Support tilde expansion of filenames from wgetrc
* src/options.c (_shell_expand): New function to perform tilde
expansion on a given string
(parse_filename): Parse the input string and perform shell expansion on
it using _shell_expand
(parse_string): Set the parser functions for various options that may be
set from a config file to use parse_filename
(wget_strnglob): Run glob() on only the specified part of the string
2017-02-22 16:28:10 +01:00
2b448ffa92 Fix memleak in HPKP
* include/wget/wget.h: Changed declaration of wget_hpkp_free()
* libwget/hpkp.c: Use wget_hpkp_free() as destructor function
* libwget/http.c (wget_http_free_hpkp_entries): Amended
* tests/test.c: Fix typos
2017-02-22 13:16:40 +01:00
68b8f02fab Implement HPKP checking
* include/wget/wget.h: Add function wget_hpkp_db_check_pubkey(),
  add WGET_SSL_HPKP_CACHE
* libwget/hpkp.c: Implement wget_hpkp_db_check_pubkey()
* libwget/ssl_gnutls.c: Implement and use _cert_verify_hpkp()
* src/options.c (init): Set WGET_SSL_HPKP_CACHE
2017-02-22 13:16:40 +01:00
afc4d5a1af HPKP API redesign 2017-02-22 13:16:40 +01:00
e6a73ffa99 * include/wget/wget.h: Fix HPKP function declarations 2017-02-22 13:16:39 +01:00
0cd52c7cce * include/wget/wget.h: Use WGETAPI for hpkp function declarations 2017-02-22 13:16:34 +01:00
89fed7e4db HPKP: Fix memory leaks
* src/wget.c (main, process_response_header): add debug traces
 * libwget/http.c (wget_http_parse_public_key_pins): fix memory leaks
 * libwget/hpkp.c (__wget_hpkp_free, wget_hpkp_db_load): fix memory
   leaks
2017-02-22 13:13:54 +01:00
86cb1180b6 HPKP: Define return values. 2017-02-22 13:13:54 +01:00
4d263eb628 HPKP: use vector rather than linked list for b64 pins 2017-02-22 13:13:54 +01:00
5a3a418705 HPKP: hpkp complete 2017-02-22 13:13:54 +01:00
33e01c5dfa HPKP: fix wget_hpkp_db_save()
* src/wget.c (main): call wget_hpkp_db_save() at the end (params
   inverted).
 * include/wget/wget.h: export wget_hpkp_db_save().
 * libwget/hpkp.c (__hashtable_browse_cb): fix fprintf format string.
   (wget_hpkp_db_save): check filename is not an empty string.
   (wget_hpkp_db_load): likewise.
2017-02-22 13:13:54 +01:00
584b15e67e Prototype HPKP functions
* include/wget/wget.h: new public function wget_hpkp_db_init().
 * libwget/hpkp.c: new file.
2017-02-22 12:46:01 +01:00
5ce41960e5 Merge from upstream/master
Conflicts: libwget/Makefile.am
2017-02-22 12:46:01 +01:00
57b259e868 HPKP boilerplate
* libwget/Makefile.am: new file hpkp.c.
 * src/options.c (print_help, struct options): new command line option '--hpkp'.
   (init): initialize HPKP database with wget_hpkp_db_init().
 * src/wget_options.h (struct config): new handle 'hpkp' to the HPKP
   database.
2017-02-22 12:46:01 +01:00
874a209fb4 HPKP: initial commit 2017-02-22 12:46:01 +01:00
0fe5f9282d New function wget_logger_is_active()
* include/wget/wget.h: Add prototype
* libwget/Makefile.am: Add logger.h
* libwget/log.c: Include logger.h
* libwget/logger.c: Include logger.h, add wget_logger_is_active()
* libwget/net.c: Use wget_logger_is_active()
* libwget/private.h: Remove struct _wget_logger_st
2017-01-23 16:55:52 +01:00
bfcd65c12b Use typedefs for function pointer arguments
* include/wget/wget.h: Add typedefs
* libwget/cookie.c: Use typedefs
* libwget/css.c: Likewise
* libwget/css_url.c: Likewise
* libwget/decompressor.c: Likewise
* libwget/hashmap.c: Likewise
* libwget/hsts.c: Likewise
* libwget/http.c: Likewise
* libwget/init.c: Likewise
* libwget/io.c: Likewise
* libwget/list.c: Likewise
* libwget/logger.c: Likewise
* libwget/metalink.c: Likewise
* libwget/net.c: Likewise
* libwget/netrc.c: Likewise
* libwget/ocsp.c: Likewise
* libwget/private.h: Likewise
* libwget/robots.c: Likewise
* libwget/stringmap.c: Likewise
* libwget/tls_session.c: Likewise
* libwget/vector.c: Likewise
* libwget/xml.c: Likewise
* src/blacklist.c: Likewise
* src/host.c: Likewise
* src/options.c: Likewise
* src/wget.c: Likewise
* tests/stringmap_perf.c: Likewise
* tests/test.c: Likewise
2017-01-23 14:43:17 +01:00
315c8f7516 base46: Use size_t as string length param 2017-01-20 16:23:02 +01:00
dfc4b53eae Add typedef for wget_vector_browse() callback function
* include/wget/wget.h: Add typedef wget_list_browse_cb_t
* libwget/list.c: Use wget_list_browse_cb_t
2017-01-19 17:11:58 +01:00
566e9f7c70 Add typedef for wget_update() callback functions
* include/wget/wget.h: Add typedef wget_update_cb_t
* libwget/cookie.c: Use wget_update_cb_t
* libwget/hsts.c: Likewise
* libwget/tls_session.c: Likewise
2017-01-19 17:11:58 +01:00
8248bfecd1 Add console color abstraction
* docs/Makefile.am: Add libwget-console.3
* include/wget/wget.h: Add console routines
* libwget/Makefile.am: Add libwget/console.c
* libwget/console.c: New file with console functions
* src/log.c: Add WIN32 and console function calls
2017-01-06 14:23:37 +01:00
1f39794852 Add function wget_truncate()
* bootstrap.conf: Ad gnulib module ftruncate
* include/wget/wget.h: Add wget_truncate
* libwget/io.c: Add wget_truncate
* src/job.c: Call wget_truncate instead of truncate

The truncate() function is not covered by gnulib.
wget_truncate() is a wrapper around ftruncate(), which is covered
by gnulib.
2017-01-06 12:14:40 +01:00
042065d44f Let MSVC compile libwget/init.c (wget_global_init)
Reported-by: Gisle Vanem
2017-01-05 22:13:46 +01:00
0354295e03 Add network init function to bootstrap sockets on Windows
* bootstrap.conf: Add gnulib module 'sockets'
* examples/http_get2.c: Call wget_net_init()
* include/wget/wget.h: Add wget_net_init() and wget_net_deinit()
* libwget/init.c (wget_global_init): Call wget_net_init(),
  (wget_global_deinit): Call wget_net_deinit()
* libwget/net.c: Add wget_net_init() and wget_net_deinit()
2017-01-05 12:09:25 +01:00
43f30f309e Rename wget_str_[v]asprintf to wget_[v]aprintf
* include/wget/wget.h: Rename wget_str_[v]asprintf to wget_[v]aprintf
* libwget/http.c: Likewise
* libwget/iri.c: Likewise
* libwget/printf.c: Likewise
* libwget/vector.c: Likewise
* src/options.c: Likewise
* tests/test-metalink.c: Likewise
2016-12-21 11:54:36 +01:00
909afd75fc Make the visibiity attribute preceed the prototype
* include/wget/wget.h: Make library symbol export attribute
((__visibility__)) preceed the function prototype to work on Windows
with MSVC
* tests/libtest.h: Same

Reported-by: Gisle Vanem
2016-11-16 12:03:27 +01:00
ecbfea77c0 Rename LIBWGET_EXPORT to WGETAPI
* include/wget/wget.t: Rename LIBWGET_EXPORT to WGETAPI
* tests/libtest.h: Same
2016-11-16 11:45:46 +01:00
fe967f4817 Export symbols only from wget.h
* libwget/iri.c: Move exporting of symbol wget_iri_schemes from here
* include/wget/wget.h: To here
2016-11-16 11:16:58 +01:00
ba365efd0d Replace x{malloc,calloc} calls with wget_* methods
* include/wget/wget.h: Remove preprocessor definitions for xmalloc,
xcalloc and xrealloc
* src/job.c(job_validate_file): Replace xcalloc with wget_calloc
* src/wget.c(main): Same
(process_head_response): Same
(_prepare_file): Same
(http_send_request): Same
2016-11-16 11:06:25 +01:00
f6c3382f30 * include/wget.h: G_GNUC_WGET_PRINT_FORMAT -> G_GNUC_WGET_PRINTF_FORMAT 2016-11-08 21:40:08 +01:00
58d18ad60c New function to check for literal IPv4/IPv6 addresses
* bootstrap.conf: Add gnulib module 'inet_pton'
* docs/Makefile.am: Add man page libwget-ip.3
* wget/wget.h (struct wget_iri_st): New field 'is_ip_address'.
* libwget/Makefile.am: Add new file ip.c
* libwget/ip.c: New file defining new function wget_ip_is_family()
* libwget/iri.c (wget_iri_parse): Check for literal IP address
* src/wget.c: Do not add to HSTS if literal IP address
2016-11-08 16:22:08 +01:00
f7d74af712 * include/wget/wget.h: Fix compilation error 2016-11-07 17:30:31 +01:00
2e1b1ad2fc * include/wget/wget.h: Define empty LIBWGET_EXPORT if LIBWGET_STATIC is defined 2016-11-07 16:18:18 +01:00
c270cda231 New inline function to compute length of base64 decoded string
* include/wget/wget.h (wget_base64_get_decoded_length): new function
 * libwget/tls_session.c (_tls_session_db_load): use
   wget_base64_get_decoded_length().
2016-11-06 17:10:18 +01:00
b049eb6d3a Fix gcc/clang warnings from the new manywarnings feature
* configure.ac: Suppress -Wunsuffixed-float-constants for gcc,
  add -Wno-disabled-macro-expansion to clang flags
* examples/getstream.c: Removed 'unused' attributes from argc and argv.
* include/wget/wget.h: Fix slot to int for bar functions.
* libwget/bar.c: Fix slot to int for bar functions, cast to int,
  change type of bitfield 'redraw' to unsigned.
* libwget/hsts.c (wget_hsts_new): Fix int to bitfield conversion.
* libwget/http.c: Cast printf %p format to (void *).
* src/wget.c: Cast printf %p format to (void *),
  remove ACTION_DONE from enum actions.
* src/wget_host.h: Fix double declaration of host_remove_job()
* tests/libtest.c (wget_test_check_filesystem): Fix type of 'rc'.
* tests/test-idn-robots.c: Fix comma to semikolon at end of line.
* tests/test.c (test_strcasecmp_ascii): Fix type of loop variable.
2016-11-02 22:19:41 +01:00
861385a6c8 Separate progress bar logic into library and application parts
* include/wget/wget.h: Remove struct _wget_bar_ctx and wget_bat_ctx_t,
  amend prototypes of wget_bar functions,
  new function wget_bar_write_line()
* libwget/bar.c: Bunch of cleanups and code rearrangements
* src/wget_progress.h: Remove
* src/Makefile.am: Remove src/wget_progress.h
* src/bar.c: Remove mutex usage,
  (_error_write) call wget_bar_write_line
* src/log.c: Remove redundant param checks
* src/wget.c: Add struct _body_callback_context,
  always call bar_slot_begin() on begin of new download,
  use downloader->id as progress slot
* src/wget_bar.h: Amend prototypes of bar_ functions
2016-10-13 16:39:59 +02:00
49ab9740e1 Fix progress bar details
* include/wget/wget.h: wget_get_screen_size() must not be pure
* libwget/bar.c (_bar_get_width): Leave one space right to avoid linebreaks
* libwget/utils.c (wget_human_readable): Align # of bytes
* src/bar.c (wget_get_screen_size): Fix return value
2016-10-05 11:49:45 +02:00
40340af300 Export a new function for determining screen dimensions
* include/wget/wget.h: New function wget_get_screen_size. Replaces old
wget_determine_screen_width
* libwget/bar.c (_bar_get_width): Use wget_get_screen_size instead of
wget_determine_screen_width
* libwget/utils.c: Replace function wget_determine_screen_width with a
more generic wget_get_screen_size.
2016-10-04 21:31:34 +02:00