2 @setfilename ../../info/url.info
3 @settitle URL Programmer's Manual
9 @c @setchapternewpage odd
14 %\global\baselineskip 30pt % for printing in double space
16 @dircategory Emacs lisp libraries
18 * URL: (url). URL loading package.
22 This is the manual for the @code{url} Emacs Lisp library.
24 Copyright @copyright{} 1993--1999, 2002, 2004--2016 Free Software
28 Permission is granted to copy, distribute and/or modify this document
29 under the terms of the GNU Free Documentation License, Version 1.3 or
30 any later version published by the Free Software Foundation; with no
31 Invariant Sections, with the Front-Cover Texts being ``A GNU Manual,''
32 and with the Back-Cover Texts as in (a) below. A copy of the license
33 is included in the section entitled ``GNU Free Documentation License''.
35 (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and
36 modify this GNU manual.''
42 @title URL Programmer's Manual
43 @subtitle First Edition, URL Version 2.0
44 @author William M. Perry @email{wmperry@@gnu.org}
45 @author David Love @email{fx@@gnu.org}
47 @vskip 0pt plus 1filll
61 * Introduction:: About the @code{url} library.
62 * URI Parsing:: Parsing (and unparsing) URIs.
63 * Retrieving URLs:: How to use this package to retrieve a URL.
64 * Supported URL Types:: Descriptions of URL types currently supported.
65 * General Facilities:: URLs can be cached, accessed via a gateway
66 and tracked in a history list.
67 * Customization:: Variables you can alter.
68 * GNU Free Documentation License:: The license for this documentation.
78 @cindex uniform resource identifier
79 @cindex uniform resource locator
81 A @dfn{Uniform Resource Identifier} (URI) is a specially-formatted
82 name, such as an Internet address, that identifies some name or
83 resource. The format of URIs is described in RFC 3986, which updates
84 and replaces the earlier RFCs 2732, 2396, 1808, and 1738. A
85 @dfn{Uniform Resource Locator} (URL) is an older but still-common
86 term, which basically refers to a URI corresponding to a resource that
87 can be accessed (usually over a network) in a specific way.
89 Here are some examples of URIs (taken from RFC 3986):
92 ftp://ftp.is.co.za/rfc/rfc1808.txt
93 http://www.ietf.org/rfc/rfc2396.txt
94 ldap://[2001:db8::7]/c=GB?objectClass?one
95 mailto:John.Doe@@example.com
96 news:comp.infosystems.www.servers.unix
98 telnet://192.0.2.16:80/
99 urn:oasis:names:specification:docbook:dtd:xml:4.1.2
102 This manual describes the @code{url} library, an Emacs Lisp library
103 for parsing URIs and retrieving the resources to which they refer.
104 (The library is so-named for historical reasons; nowadays, the ``URI''
105 terminology is regarded as the more general one, and ``URL'' is
106 technically obsolete despite its widespread vernacular usage.)
111 A URI consists of several @dfn{components}, each having a different
112 meaning. For example, the URI
115 http://www.gnu.org/software/emacs/
119 specifies the scheme component @samp{http}, the hostname component
120 @samp{www.gnu.org}, and the path component @samp{/software/emacs/}.
123 The format of URIs is specified by RFC 3986. The @code{url} library
124 provides the Lisp function @code{url-generic-parse-url}, a (mostly)
125 standard-compliant URI parser, as well as function
126 @code{url-recreate-url}, which converts a parsed URI back into a URI
129 @defun url-generic-parse-url uri-string
130 This function returns a parsed version of the string @var{uri-string}.
133 @defun url-recreate-url uri-obj
134 @cindex unparsing URLs
135 Given a parsed URI, this function returns the corresponding URI string.
139 The return value of @code{url-generic-parse-url}, and the argument
140 expected by @code{url-recreate-url}, is a @dfn{parsed URI}: a CL
141 structure whose slots hold the various components of the URI@.
142 @xref{Top,the CL Manual,,cl,GNU Emacs Common Lisp Emulation}, for
143 details about CL structures. Most of the other functions in the
144 @code{url} library act on parsed URIs.
147 * Parsed URIs:: Format of parsed URI structures.
148 * URI Encoding:: Non-@acronym{ASCII} characters in URIs.
152 @section Parsed URI structures
154 Each parsed URI structure contains the following slots:
158 The URI scheme (a string, e.g., @code{http}). @xref{Supported URL
159 Types}, for a list of schemes that the @code{url} library knows how to
160 process. This slot can also be @code{nil}, if the URI is not fully
164 The user name (a string), or @code{nil}.
167 The user password (a string), or @code{nil}. The use of this URI
168 component is strongly discouraged; nowadays, passwords are transmitted
169 by other means, not as part of a URI.
172 The host name (a string), or @code{nil}. If present, this is
173 typically a domain name or IP address.
176 The port number (an integer), or @code{nil}. Omitting this component
177 usually means to use the ``standard'' port associated with the URI
181 The combination of the ``path'' and ``query'' components of the URI (a
182 string), or @code{nil}. If the query component is present, it is the
183 substring following the first @samp{?} character, and the path
184 component is the substring before the @samp{?}. The meaning of these
185 components is scheme-dependent; they do not necessarily refer to a
189 The fragment component (a string), or @code{nil}. The fragment
190 component specifies a ``secondary resource'', such as a section of a
194 This is @code{t} if the URI is fully specified, i.e., the
195 hierarchical components of the URI (the hostname and/or username
196 and/or password) are preceded by @samp{//}.
206 @findex url-attributes
208 These slots have accessors named @code{url-@var{part}}, where
209 @var{part} is the slot name. For example, the accessor for the
210 @code{host} slot is the function @code{url-host}. The @code{url-port}
211 accessor returns the default port for the URI scheme if the parsed
212 URI's @var{port} slot is @code{nil}.
214 The slots can be set using @code{setf}. For example:
217 (setf (url-port url) 80)
221 @section URI Encoding
223 @cindex percent encoding
224 The @code{url-generic-parse-url} parser does not obey RFC 3986 in
225 one respect: it allows non-@acronym{ASCII} characters in URI strings.
227 Strictly speaking, RFC 3986 compatible URIs may only consist of
228 @acronym{ASCII} characters; non-@acronym{ASCII} characters are
229 represented by converting them to UTF-8 byte sequences, and performing
230 @dfn{percent encoding} on the bytes. For example, the o-umlaut
231 character is converted to the UTF-8 byte sequence @samp{\xD3\xA7},
232 then percent encoded to @samp{%D3%A7}. (Certain ``reserved''
233 @acronym{ASCII} characters must also be percent encoded when they
234 appear in URI components.)
236 The function @code{url-encode-url} can be used to convert a URI
237 string containing arbitrary characters to one that is properly
238 percent-encoded in accordance with RFC 3986.
240 @defun url-encode-url url-string
241 This function return a properly URI-encoded version of
242 @var{url-string}. It also performs @dfn{URI normalization},
243 e.g., converting the scheme component to lowercase if it was
244 previously uppercase.
247 To convert between a string containing arbitrary characters and a
248 percent-encoded all-@acronym{ASCII} string, use the functions
249 @code{url-hexify-string} and @code{url-unhex-string}:
251 @defun url-hexify-string string &optional allowed-chars
252 This function performs percent-encoding on @var{string}, and returns
255 If @var{string} is multibyte, it is first converted to a UTF-8 byte
256 string. Each byte corresponding to an allowed character is left
257 as-is, while all other bytes are converted to a three-character
258 sequence: @samp{%} followed by two upper-case hex digits.
260 @vindex url-unreserved-chars
261 @cindex unreserved characters
262 The allowed characters are specified by @var{allowed-chars}. If this
263 argument is @code{nil}, the allowed characters are those specified as
264 @dfn{unreserved characters} by RFC 3986 (see the variable
265 @code{url-unreserved-chars}). Otherwise, @var{allowed-chars} should
266 be a vector whose @var{n}-th element is non-@code{nil} if character
270 @defun url-unhex-string string &optional allow-newlines
271 This function replaces percent-encoding sequences in @var{string} with
272 their character equivalents, and returns the resulting string.
274 If @var{allow-newlines} is non-@code{nil}, it allows the decoding of
275 carriage returns and line feeds, which are normally forbidden in URIs.
278 @node Retrieving URLs
279 @chapter Retrieving URLs
281 The @code{url} library defines the following three functions for
282 retrieving the data specified by a URL@. The actual retrieval protocol
283 depends on the URL's URI scheme, and is performed by lower-level
284 scheme-specific functions. (Those lower-level functions are not
285 documented here, and generally should not be called directly.)
287 In each of these functions, the @var{url} argument can be either a
288 string or a parsed URL structure. If it is a string, that string is
289 passed through @code{url-encode-url} before using it, to ensure that
290 it is properly URI-encoded (@pxref{URI Encoding}).
292 @defun url-retrieve-synchronously url
293 This function synchronously retrieves the data specified by @var{url},
294 and returns a buffer containing the data. The return value is
295 @code{nil} if there is no data associated with the URL (as is the case
296 for @code{dired}, @code{info}, and @code{mailto} URLs).
299 @defun url-retrieve url callback &optional cbargs silent no-cookies
300 This function retrieves @var{url} asynchronously, calling the function
301 @var{callback} when the object has been completely retrieved. The
302 return value is the buffer into which the data will be inserted, or
303 @code{nil} if the process has already completed.
305 The callback function is called this way:
308 (apply @var{callback} @var{status} @var{cbargs})
312 where @var{status} is a plist representing what happened during the
313 retrieval, with most recent events first, or an empty list if no
314 events have occurred. Each pair in the plist is one of:
317 @item (:redirect @var{redirected-to})
318 This means that the request was redirected to the URL
321 @item (:error (@var{error-symbol} . @var{data}))
322 This means that an error occurred. If so desired, the error can be
323 signaled with @code{(signal @var{error-symbol} @var{data})}.
326 When the callback function is called, the current buffer is the one
327 containing the retrieved data (if any). The buffer also contains any
328 MIME headers associated with the data retrieval.
330 If the optional argument @var{silent} is non-@code{nil}, progress
331 messages are suppressed. If the optional argument @var{no-cookies} is
332 non-@code{nil}, cookies are not stored or sent.
335 @defun url-queue-retrieve url callback &optional cbargs silent no-cookies
336 This function acts like @code{url-retrieve}, but with limits on the
337 number of concurrently-running network processes. The option
338 @code{url-queue-parallel-processes} controls the number of concurrent
339 processes, and the option @code{url-queue-timeout} sets a timeout in
342 To use this function, you must @code{(require 'url-queue)}.
345 @vindex url-queue-parallel-processes
346 @defopt url-queue-parallel-processes
347 The value of this option is an integer specifying the maximum number
348 of concurrent @code{url-queue-retrieve} network processes. If the
349 number of @code{url-queue-retrieve} calls is larger than this number,
350 later ones are queued until earlier ones are finished.
353 @vindex url-queue-timeout
354 @defopt url-queue-timeout
355 The value of this option is a number specifying the maximum lifetime
356 of a @code{url-queue-retrieve} network process, once it is started.
357 If a process is not finished by then, it is killed and removed from
361 @node Supported URL Types
362 @chapter Supported URL Types
364 This chapter describes functions and variables affecting URL retrieval
365 for specific schemes.
368 * http/https:: Hypertext Transfer Protocol.
369 * file/ftp:: Local files and FTP archives.
370 * info:: Emacs "Info" pages.
371 * mailto:: Sending email.
372 * news/nntp/snews:: Usenet news.
373 * rlogin/telnet/tn3270:: Remote host connectivity.
374 * irc:: Internet Relay Chat.
375 * data:: Embedded data URLs.
376 * nfs:: Networked File System.
377 * ldap:: Lightweight Directory Access Protocol.
378 * man:: Unix man pages.
379 * Tramp:: Schemes supported via Tramp.
383 @section @code{http} and @code{https}
385 The @code{http} scheme refers to the Hypertext Transfer Protocol. The
386 @code{url} library supports HTTP version 1.1, specified in RFC 2616.
387 Its default port is 80.
389 The @code{https} scheme is a secure version of @code{http}, with
390 transmission via SSL@. It is defined in RFC 2069, and its default port
391 is 443. When using @code{https}, the @code{url} library performs SSL
392 encryption via the @code{ssl} library, by forcing the @code{ssl}
393 gateway method to be used. @xref{Gateways in general}.
395 @defopt url-honor-refresh-requests
396 If this option is non-@code{nil} (the default), the @code{url} library
397 honors the HTTP @samp{Refresh} header, which is used by servers to
398 direct clients to reload documents from the same URL or a or different
399 one. If the value is @code{nil}, the @samp{Refresh} header is
400 ignored; any other value means to ask the user on each request.
405 * HTTP language/coding::
407 * Dealing with HTTP documents::
413 @findex url-cookie-delete
414 @defun url-cookie-list
415 This command creates a @file{*url cookies*} buffer listing the current
416 cookies, if there are any. You can remove a cookie using the
417 @kbd{C-k} (@code{url-cookie-delete}) command.
420 @defun url-cookie-delete-cookies &optional regexp
421 This function takes a regular expression as its parameters and deletes
422 all cookies from that domain. If @var{regexp} is @code{nil}, delete
426 @defopt url-cookie-file
427 The file in which cookies are stored, defaulting to @file{cookies} in
428 the directory specified by @code{url-configuration-directory}.
431 @defopt url-cookie-confirmation
432 Specifies whether confirmation is required to accept cookies.
435 @defopt url-cookie-multiple-line
436 Specifies whether to put all cookies for the server on one line in the
437 HTTP request to satisfy broken servers like
438 @url{http://www.hotmail.com}.
441 @defopt url-cookie-trusted-urls
442 A list of regular expressions matching URLs from which to accept
446 @defopt url-cookie-untrusted-urls
447 A list of regular expressions matching URLs from which to reject
451 @defopt url-cookie-save-interval
452 The number of seconds between automatic saves of cookies to disk.
457 @node HTTP language/coding
458 @subsection Language and Encoding Preferences
460 HTTP allows clients to express preferences for the language and
461 encoding of documents which servers may honor. For each of these
462 variables, the value is a string; it can specify a single choice, or
463 it can be a comma-separated list.
465 Normally, this list is ordered by descending preference. However, each
466 element can be followed by @samp{;q=@var{priority}} to specify its
467 preference level, a decimal number from 0 to 1; e.g., for
468 @code{url-mime-language-string}, @w{@code{"de, en-gb;q=0.8,
469 en;q=0.7"}}. An element that has no @samp{;q} specification has
472 @defopt url-mime-charset-string
473 @cindex character sets
474 @cindex coding systems
475 This variable specifies a preference for character sets when documents
476 can be served in more than one encoding.
478 HTTP allows specifying a series of MIME charsets which indicate your
479 preferred character set encodings, e.g., Latin-9 or Big5, and these
480 can be weighted. The default series is generated automatically from
481 the associated MIME types of all defined coding systems, sorted by the
482 coding system priority specified in Emacs. @xref{Recognize Coding, ,
483 Recognizing Coding Systems, emacs, The GNU Emacs Manual}.
486 @defopt url-mime-language-string
487 @cindex language preferences
488 A string specifying the preferred language when servers can serve
489 files in several languages. Use RFC 1766 abbreviations, e.g.,
490 @samp{en} for English, @samp{de} for German.
492 The string can be @code{"*"} to get the first available language (as
493 opposed to the default).
496 @node HTTP URL Options
497 @subsection HTTP URL Options
499 HTTP supports an @samp{OPTIONS} method describing things supported by
502 @defun url-http-options url
503 Returns a property list describing options available for URL@. The
504 property list members are:
508 A list of symbols specifying what HTTP methods the resource
513 A list of numbers specifying what DAV protocol/schema versions are
518 A list of supported DASL search types supported (string form).
521 A list of the units available for use in partial document fetches.
525 The @dfn{Platform For Privacy Protection} description for the resource.
526 Currently this is just the raw header contents.
531 @node Dealing with HTTP documents
532 @subsection Dealing with HTTP documents
534 HTTP URLs are retrieved into a buffer containing the HTTP headers
535 followed by the body. Since the headers are quasi-MIME, they may be
536 processed using the MIME library. @xref{Top,, Emacs MIME,
537 emacs-mime, The Emacs MIME Manual}.
540 @section file and ftp
543 @cindex File Transfer Protocol
544 @cindex compressed files
547 The @code{ftp} and @code{file} schemes are defined in RFC 1808. The
548 @code{url} library treats @samp{ftp:} and @samp{file:} as synonymous.
549 Such URLs have the form
552 ftp://@var{user}:@var{password}@@@var{host}:@var{port}/@var{file}
553 file://@var{user}:@var{password}@@@var{host}:@var{port}/@var{file}
557 If the URL specifies a local file, it is retrieved by reading the file
558 contents in the usual way. If it specifies a remote file, it is
559 retrieved using either the Tramp or the Ange-FTP package.
560 @xref{Remote Files,,, emacs, The GNU Emacs Manual}.
562 When retrieving a compressed file, it is automatically uncompressed
563 if it has the file suffix @file{.z}, @file{.gz}, @file{.Z},
564 @file{.bz2}, or @file{.xz}. (The list of supported suffixes is
565 hard-coded, and cannot be altered by customizing
566 @code{jka-compr-compression-info-list}.)
568 @defopt url-directory-index-file
569 This option specifies the filename to look for when a @code{file} or
570 @code{ftp} URL specifies a directory. The default is
571 @file{index.html}. If this file exists and is readable, it is viewed.
572 Otherwise, Emacs visits the directory using Dired.
579 @findex Info-goto-node
581 The @code{info} scheme is non-standard. Such URLs have the form
584 info:@var{file}#@var{node}
588 and are retrieved by invoking @code{Info-goto-node} with argument
589 @samp{(@var{file})@var{node}}. If @samp{#@var{node}} is omitted, the
590 @samp{Top} node is opened.
597 A @code{mailto} URL specifies an email message to be sent to a given
598 email address. For example, @samp{mailto:foo@@bar.com} specifies
599 sending a message to @samp{foo@@bar.com}. The ``retrieval method''
600 for such URLs is to open a mail composition buffer in which the
601 appropriate content (e.g., the recipient address) has been filled in.
603 As defined in RFC 6068, a @code{mailto} URL can have the form
606 @samp{mailto:@var{mailbox}[?@var{header}=@var{contents}[&@var{header}=@var{contents}]]}
610 where an arbitrary number of @var{header}s can be added. If the
611 @var{header} is @samp{body}, then @var{contents} is put in the message
612 body; otherwise, a @var{header} header field is created with
613 @var{contents} as its contents. Note that the @code{url} library does
614 not perform any checking of @var{header} or @var{contents}, so you
615 should check them before sending the message.
617 @defopt url-mail-command
618 @vindex mail-user-agent
619 The value of this variable is the function called whenever url needs
620 to send mail. This should normally be left its default, which is the
621 standard mail-composition command @code{compose-mail}. @xref{Sending
622 Mail,,, emacs, The GNU Emacs Manual}.
625 If the document containing the @code{mailto} URL itself possessed a
626 known URL, Emacs automatically inserts an @samp{X-Url-From} header
627 field into the mail buffer, specifying that URL.
629 @node news/nntp/snews
630 @section @code{news}, @code{nntp} and @code{snews}
637 The @code{news}, @code{nntp}, and @code{snews} schemes, defined in RFC
638 1738, are used for reading Usenet newsgroups. For compatibility with
639 non-standard-compliant news clients, the @code{url} library allows
640 host and port fields to be included in @code{news} URLs, even though
641 they are properly only allowed for @code{nntp} and @code{snews}.
643 @code{news} and @code{nntp} URLs have the following form:
646 @item news:@var{newsgroup}
647 Retrieves a list of messages in @var{newsgroup};
648 @item news:@var{message-id}
649 Retrieves the message with the given @var{message-id};
651 Retrieves a list of all available newsgroups;
652 @item nntp://@var{host}:@var{port}/@var{newsgroup}
653 @itemx nntp://@var{host}:@var{port}/@var{message-id}
654 @itemx nntp://@var{host}:@var{port}/*
655 Similar to the @samp{news} versions.
658 The default port for @code{nntp} (and @code{news}) is 119. The
659 difference between an @code{nntp} URL and a @code{news} URL is that an
660 @code{nttp} URL may specify an article by its number. The
661 @samp{snews} scheme is the same as @samp{nntp}, except that it is
662 tunneled through SSL and has default port 563.
664 These URLs are retrieved via the Gnus package.
666 @cindex environment variable
668 @defopt url-news-server
669 This variable specifies the default news server from which to fetch
670 news, if no server was specified in the URL@. The default value,
671 @code{nil}, means to use the server specified by the standard
672 environment variable @samp{NNTPSERVER}, or @samp{news} if that
673 environment variable is unset.
676 @node rlogin/telnet/tn3270
677 @section rlogin, telnet and tn3270
681 @cindex terminal emulation
682 @findex terminal-emulator
684 These URL schemes are defined in RFC 1738, and are used for logging in
685 via a terminal emulator. They have the form
688 telnet://@var{user}:@var{password}@@@var{host}:@var{port}
692 but the @var{password} component is ignored. By default, the
693 @code{telnet} scheme is handled via Tramp (@pxref{Tramp}).
695 To handle rlogin, telnet and tn3270 URLs, a @code{rlogin},
696 @code{telnet} or @code{tn3270} (the program names and arguments are
697 hardcoded) session is run in a @code{terminal-emulator} buffer.
698 Well-known ports are used if the URL does not specify a port.
703 @cindex Internet Relay Chat
708 The @code{irc} scheme is defined in the Internet Draft at
709 @url{http://www.w3.org/Addressing/draft-mirashi-url-irc-01.txt} (which
710 was never approved as an RFC). Such URLs have the form
713 irc://@var{host}:@var{port}/@var{target},@var{needpass}
717 and are retrieved by opening an @acronym{IRC} session using the
718 function specified by @code{url-irc-function}.
720 @defopt url-irc-function
721 The value of this option is a function, which is called to open an IRC
722 connection for @code{irc} URLs. This function must take five
723 arguments, @var{host}, @var{port}, @var{channel}, @var{user} and
724 @var{password}. The @var{channel} argument specifies the channel to
725 join immediately, and may be @code{nil}.
727 The default is @code{url-irc-rcirc}, which uses the Rcirc package.
728 Other options are @code{url-irc-erc} (which uses ERC) and
729 @code{url-irc-zenirc} (which uses ZenIRC).
736 The @code{data} scheme, defined in RFC 2397, contains MIME data in
737 the URL itself. Such URLs have the form
740 data:@r{[}@var{media-type}@r{]}@r{[};@var{base64}@r{]},@var{data}
744 @var{media-type} is a MIME @samp{Content-Type} string, possibly
745 including parameters. It defaults to
746 @samp{text/plain;charset=US-ASCII}. The @samp{text/plain} can be
747 omitted but the charset parameter supplied. If @samp{;base64} is
748 present, the @var{data} are base64-encoded.
753 @cindex Network File System
756 The @code{nfs} scheme, defined in RFC 2224, is similar to @code{ftp}
757 except that it points to a file on a remote host that is handled by an
758 NFS automounter on the local host. Such URLs have the form
761 nfs://@var{user}:@var{password}@@@var{host}:@var{port}/@var{file}
764 @defvar url-nfs-automounter-directory-spec
766 A string saying how to invoke the NFS automounter. Certain @samp{%}
767 sequences are recognized:
771 The hostname of the NFS server;
773 The port number of the NFS server;
775 The username to use to authenticate;
777 The password to use to authenticate;
779 The filename on the remote server;
784 Each can be used any number of times.
789 @cindex Lightweight Directory Access Protocol
791 The LDAP scheme is defined in RFC 2255.
795 @cindex @command{man}
796 @cindex Unix man pages
799 The @code{man} scheme is a non-standard one. Such URLs have the form
802 @samp{man:@var{page-spec}}
806 and are retrieved by passing @var{page-spec} to the Lisp function
810 @section URL Types Supported via Tramp
812 @vindex url-tramp-protocols
813 Some additional URL types are supported by passing them to Tramp
814 (@pxref{Top, The Tramp Manual,, tramp, The Tramp Manual}). These
815 protocols are listed in the @code{url-tramp-protocols} variable, which
816 you can customize. The default value includes the following
821 The file transfer protocol. @xref{file/ftp}.
825 The secure shell protocol. @xref{Inline Methods,,, tramp, The Tramp
830 The secure file copy protocol. @xref{External Methods,,, tramp, The
835 The remote sync protocol.
841 @node General Facilities
842 @chapter General Facilities
847 * Gateways in general::
852 @section Disk Caching
854 @cindex Persistent Cache
857 The disk cache stores retrieved documents locally, whence they can be
858 retrieved more quickly. When requesting a URL that is in the cache,
859 the library checks to see if the page has changed since it was last
860 retrieved from the remote machine. If not, the local copy is used,
861 saving the transmission over the network.
862 @cindex Cleaning the cache
863 @cindex Clearing the cache
864 @cindex Cache cleaning
865 Currently the cache isn't cleared automatically.
866 @c Running the @code{clean-cache} shell script
867 @c fist is recommended, to allow for future cleaning of the cache. This
868 @c shell script will remove all files that have not been accessed since it
869 @c was last run. To keep the cache pared down, it is recommended that this
870 @c script be run from @i{at} or @i{cron} (see the manual pages for
871 @c crontab(5) or at(1) for more information)
873 @defopt url-automatic-caching
874 Setting this variable non-@code{nil} causes documents to be cached
878 @defopt url-cache-directory
879 This variable specifies the
880 directory to store the cache files. It defaults to sub-directory
881 @file{cache} of @code{url-configuration-directory}.
884 @defopt url-cache-creation-function
885 The cache relies on a scheme for mapping URLs to files in the cache.
886 This variable names a function which sets the type of cache to use.
887 It takes a URL as argument and returns the absolute file name of the
888 corresponding cache file. The two supplied possibilities are
889 @code{url-cache-create-filename-using-md5} and
890 @code{url-cache-create-filename-human-readable}.
893 @defun url-cache-create-filename-using-md5 url
894 Creates a cache file name from @var{url} using MD5 hashing.
895 This is creates entries with very few cache collisions and is fast.
898 (url-cache-create-filename-using-md5 "http://www.example.com/foo/bar")
899 @result{} "/home/fx/.url/cache/fx/http/com/example/www/b8a35774ad20db71c7c3409a5410e74f"
903 @defun url-cache-create-filename-human-readable url
904 Creates a cache file name from @var{url} more obviously connected to
905 @var{url} than for @code{url-cache-create-filename-using-md5}, but
906 more likely to conflict with other files.
908 (url-cache-create-filename-human-readable "http://www.example.com/foo/bar")
909 @result{} "/home/fx/.url/cache/fx/http/com/example/www/foo/bar"
913 @defun url-cache-expired
914 This function returns non-@code{nil} if a cache entry has expired (or is absent).
915 The arguments are a URL and optional expiration delay in seconds
916 (default @var{url-cache-expire-time}).
919 @defopt url-cache-expire-time
920 This variable is the default number of seconds to use for the
921 expire-time argument of the function @code{url-cache-expired}.
924 @defun url-fetch-from-cache
925 This function takes a URL as its argument and returns a buffer
926 containing the data cached for that URL.
929 @c Fixme: never actually used currently?
930 @c @defopt url-standalone-mode
931 @c @cindex Relying on cache
932 @c @cindex Cache only mode
933 @c @cindex Standalone mode
934 @c If this variable is non-@code{nil}, the library relies solely on the
935 @c cache for fetching documents and avoids checking if they have changed
936 @c on remote servers.
939 @c With a large cache of documents on the local disk, it can be very handy
940 @c when traveling, or any other time the network connection is not active
941 @c (a laptop with a dial-on-demand PPP connection, etc.). Emacs/W3 can rely
942 @c solely on its cache, and avoid checking to see if the page has changed
943 @c on the remote server. In the case of a dial-on-demand PPP connection,
944 @c this will keep the phone line free as long as possible, only bringing up
945 @c the PPP connection when asking for a page that is not located in the
946 @c cache. This is very useful for demonstrations as well.
949 @section Proxies and Gatewaying
951 @c fixme: check/document url-ns stuff
952 @cindex proxy servers
954 @cindex environment variables
956 Proxy servers are commonly used to provide gateways through firewalls
957 or as caches serving some more-or-less local network. Each protocol
958 (HTTP, FTP, etc.)@: can have a different gateway server. Proxying is
959 conventionally configured commonly amongst different programs through
960 environment variables of the form @code{@var{protocol}_proxy}, where
961 @var{protocol} is one of the supported network protocols (@code{http},
962 @code{ftp} etc.). The library recognizes such variables in either
963 upper or lower case. Their values are of one of the forms:
965 @item @code{@var{host}:@var{port}}
967 @item Simply a host name.
971 The @code{NO_PROXY} environment variable specifies URLs that should be
972 excluded from proxying (on servers that should be contacted directly).
973 This should be a comma-separated list of hostnames, domain names, or a
974 mixture of both. Asterisks can be used as wildcards, but other
975 clients may not support that. Domain names may be indicated by a
976 leading dot. For example:
978 NO_PROXY="*.aventail.com,home.com,.seanet.com"
980 @noindent says to contact all machines in the @samp{aventail.com} and
981 @samp{seanet.com} domains directly, as well as the machine named
982 @samp{home.com}. If @code{NO_PROXY} isn't defined, @code{no_PROXY}
983 and @code{no_proxy} are also tried, in that order.
985 Proxies may also be specified directly in Lisp.
987 @defopt url-proxy-services
988 This variable is an alist of URL schemes and proxy servers that
989 gateway them. The items are of the form @w{@code{(@var{scheme}
990 . @var{host}:@var{portnumber})}}, says that the URL @var{scheme} is
991 gatewayed through @var{portnumber} on the specified @var{host}. An
992 exception is the pseudo scheme @code{"no_proxy"}, which is paired with
993 a regexp matching host names not to be proxied. This variable is
994 initialized from the environment as above.
997 (setq url-proxy-services
998 '(("http" . "proxy.aventail.com:80")
999 ("no_proxy" . "^.*\\(aventail\\|seanet\\)\\.com")))
1003 @node Gateways in general
1004 @section Gateways in General
1008 The library provides a general gateway layer through which all
1009 networking passes. It can both control access to the network and
1010 provide access through gateways in firewalls. This may make direct
1011 connections in some cases and pass through some sort of gateway in
1012 others.@footnote{Proxies (which only operate over HTTP) are
1013 implemented using this.} The library's basic function responsible for
1014 making connections is @code{url-open-stream}.
1016 @defun url-open-stream name buffer host service
1017 @cindex opening a stream
1018 @cindex stream, opening
1019 Open a stream to @var{host}, possibly via a gateway. The other
1020 arguments are as for @code{open-network-stream}. This will not make a
1021 connection if @code{url-gateway-unplugged} is non-@code{nil}.
1024 @defvar url-gateway-local-host-regexp
1025 This is a regular expression that matches local hosts that do not
1026 require the use of a gateway. If @code{nil}, all connections are made
1027 through the gateway.
1030 @defvar url-gateway-method
1031 This variable controls which gateway method is used. It may be useful
1032 to bind it temporarily in some applications. It has values taken from
1033 a list of symbols. Possible values are:
1037 @cindex @command{telnet}
1038 Use this method if you must first telnet and log into a gateway host,
1039 and then run telnet from that host to connect to outside machines.
1042 @cindex @command{rlogin}
1043 This method is identical to @code{telnet}, but uses @command{rlogin}
1044 to log into the remote machine without having to send the username and
1045 password over the wire every time.
1049 Use if the firewall has a @sc{socks} gateway running on it. The
1050 @sc{socks} v5 protocol is defined in RFC 1928.
1053 @c This probably shouldn't be documented
1054 @c Fixme: why not? -- fx
1057 This method uses Emacs's builtin networking directly. This is the
1058 default. It can be used only if there is no firewall blocking access.
1062 The following variables control the gateway methods.
1064 @defopt url-gateway-telnet-host
1065 The gateway host to telnet to. Once logged in there, you then telnet
1066 out to the hosts you want to connect to.
1068 @defopt url-gateway-telnet-parameters
1069 This should be a list of parameters to pass to the @command{telnet} program.
1071 @defopt url-gateway-telnet-password-prompt
1072 This is a regular expression that matches the password prompt when
1075 @defopt url-gateway-telnet-login-prompt
1076 This is a regular expression that matches the username prompt when
1079 @defopt url-gateway-telnet-user-name
1080 The username to log in with.
1082 @defopt url-gateway-telnet-password
1083 The password to send when logging in.
1085 @defopt url-gateway-prompt-pattern
1086 This is a regular expression that matches the shell prompt.
1089 @defopt url-gateway-rlogin-host
1090 Host to @samp{rlogin} to before telnetting out.
1092 @defopt url-gateway-rlogin-parameters
1093 Parameters to pass to @samp{rsh}.
1095 @defopt url-gateway-rlogin-user-name
1096 User name to use when logging in to the gateway.
1098 @defopt url-gateway-prompt-pattern
1099 This is a regular expression that matches the shell prompt.
1102 @defopt socks-server
1103 This specifies the default server, it takes the form
1104 @w{@code{("Default server" @var{server} @var{port} @var{version})}}
1105 where @var{version} can be either 4 or 5.
1107 @defvar socks-password
1108 If this is @code{nil} then you will be asked for the password,
1109 otherwise it will be used as the password for authenticating you to
1110 the @sc{socks} server.
1112 @defvar socks-username
1113 This is the username to use when authenticating yourself to the
1114 @sc{socks} server. By default this is your login name.
1116 @defvar socks-timeout
1117 This controls how long, in seconds, to wait for responses from the
1118 @sc{socks} server; it is 5 by default.
1120 @c fixme: these have been effectively commented-out in the code
1121 @c @defopt socks-server-aliases
1122 @c This a list of server aliases. It is a list of aliases of the form
1123 @c @var{(alias hostname port version)}.
1125 @c @defopt socks-network-aliases
1126 @c This a list of network aliases. Each entry in the list takes the form
1127 @c @var{(alias (network))} where @var{alias} is a string that names the
1128 @c @var{network}. The networks can contain a pair (not a dotted pair) of
1129 @c @sc{ip} addresses which specify a range of @sc{ip} addresses, an @sc{ip}
1130 @c address and a netmask, a domain name or a unique hostname or @sc{ip}
1133 @c @defopt socks-redirection-rules
1134 @c This a list of redirection rules. Each rule take the form
1135 @c @var{(Destination network Connection type)} where @var{Destination
1136 @c network} is a network alias from @code{socks-network-aliases} and
1137 @c @var{Connection type} can be @code{nil} in which case a direct
1138 @c connection is used, or it can be an alias from
1139 @c @code{socks-server-aliases} in which case that server is used as a
1142 @defopt socks-nslookup-program
1143 @cindex @command{nslookup}
1144 This the @samp{nslookup} program. It is @code{"nslookup"} by default.
1148 * Suppressing network connections::
1150 @c * Broken hostname resolution::
1152 @node Suppressing network connections
1153 @subsection Suppressing Network Connections
1155 @cindex network connections, suppressing
1156 @cindex suppressing network connections
1158 @cindex HTML ``bugs''
1159 In some circumstances it is desirable to suppress making network
1160 connections. A typical case is when rendering HTML in a mail user
1161 agent, when external URLs should not be activated, particularly to
1162 avoid ``bugs'' which ``call home'' by fetch single-pixel images and the
1163 like. To arrange this, bind the following variable for the duration
1166 @defvar url-gateway-unplugged
1167 If this variable is non-@code{nil} new network connections are never
1168 opened by the URL library.
1171 @c @node Broken hostname resolution
1172 @c @subsection Broken Hostname Resolution
1174 @c @cindex hostname resolver
1175 @c @cindex resolver, hostname
1176 @c Some C libraries do not include the hostname resolver routines in
1177 @c their static libraries. If Emacs was linked statically, and was not
1178 @c linked with the resolver libraries, it will not be able to get to any
1179 @c machines off the local network. This is characterized by being able
1180 @c to reach someplace with a raw ip number, but not its hostname
1181 @c (@url{http://129.79.254.191/} works, but
1182 @c @url{http://www.cs.indiana.edu/} doesn't). This used to happen on
1183 @c SunOS4 and Ultrix, but is now probably now rare. If Emacs can't be
1184 @c rebuilt linked against the resolver library, it can use the external
1185 @c @command{nslookup} program instead.
1187 @c @defopt url-gateway-broken-resolution
1188 @c @cindex @code{nslookup} program
1189 @c @cindex program, @code{nslookup}
1190 @c If non-@code{nil}, this variable says to use the program specified by
1191 @c @code{url-gateway-nslookup-program} program to do hostname resolution.
1194 @c @defopt url-gateway-nslookup-program
1195 @c The name of the program to do hostname lookup if Emacs can't do it
1196 @c directly. This program should expect a single argument on the command
1197 @c line---the hostname to resolve---and should produce output similar to
1198 @c the standard Unix @command{nslookup} program:
1200 @c Name: www.cs.indiana.edu
1201 @c Address: 129.79.254.191
1208 @findex url-do-setup
1209 The library can maintain a global history list tracking URLs accessed.
1210 URL completion can be done from it. The history mechanism is set up
1211 automatically via @code{url-do-setup} when it is configured to be on.
1212 Note that the size of the history list is currently not limited.
1214 @vindex url-history-hash-table
1215 The history ``list'' is actually a hash table,
1216 @code{url-history-hash-table}. It contains access times keyed by URL
1217 strings. The times are in the format returned by @code{current-time}.
1219 @defun url-history-update-url url time
1220 This function updates the history table with an entry for @var{url}
1221 accessed at the given @var{time}.
1224 @defopt url-history-track
1225 If non-@code{nil}, the library will keep track of all the URLs
1226 accessed. If it is @code{t}, the list is saved to disk at the end of
1227 each Emacs session. The default is @code{nil}.
1230 @defopt url-history-file
1231 The file storing the history list between sessions. It defaults to
1232 @file{history} in @code{url-configuration-directory}.
1235 @defopt url-history-save-interval
1236 @findex url-history-setup-save-timer
1237 The number of seconds between automatic saves of the history list.
1238 Default is one hour. Note that if you change this variable directly,
1239 rather than using Custom, after @code{url-do-setup} has been run, you
1240 need to run the function @code{url-history-setup-save-timer}.
1243 @defun url-history-parse-history &optional fname
1244 Parses the history file @var{fname} (default @code{url-history-file})
1245 and sets up the history list.
1248 @defun url-history-save-history &optional fname
1249 Saves the current history to file @var{fname} (default
1250 @code{url-history-file}).
1253 @defun url-completion-function string predicate function
1254 You can use this function to do completion of URLs from the history.
1258 @chapter Customization
1260 @cindex environment variables
1261 The following environment variables affect the @code{url} library's
1262 operation at startup.
1267 @vindex url-temporary-directory
1268 If this is defined, @var{url-temporary-directory} is initialized from
1272 The following user options affect the general operation of
1275 @defopt url-configuration-directory
1276 @cindex configuration files
1277 The value of this variable specifies the name of the directory where
1278 the @code{url} library stores its various configuration files, cache
1281 The default value specifies a subdirectory named @file{url/} in the
1282 standard Emacs user data directory specified by the variable
1283 @code{user-emacs-directory} (normally @file{~/.emacs.d}). However,
1284 the old default was @file{~/.url}, and this directory is used instead
1290 Specifies the types of debug messages which are logged to
1291 the @file{*URL-DEBUG*} buffer.
1292 @code{t} means log all messages.
1293 A number means log all messages and show them with @code{message}.
1294 It may also be a list of the types of messages to be logged.
1296 @defopt url-personal-mail-address
1298 @defopt url-privacy-level
1300 @defopt url-uncompressor-alist
1302 @defopt url-passwd-entry-func
1304 @defopt url-standalone-mode
1306 @defopt url-bad-port-list
1308 @defopt url-max-password-attempts
1310 @defopt url-temporary-directory
1312 @defopt url-show-status
1314 @defopt url-confirmation-func
1315 The function to use for asking yes or no functions. This is normally
1316 either @code{y-or-n-p} or @code{yes-or-no-p}, but could be another
1317 function taking a single argument (the prompt) and returning @code{t}
1318 only if an affirmative answer is given.
1320 @defopt url-gateway-method
1321 @c fixme: describe gatewaying
1322 A symbol specifying the type of gateway support to use for connections
1323 from the local machine. The supported methods are:
1327 Run telnet in a subprocess to connect;
1329 Rlogin to another machine to connect;
1331 Connect through a socks server;
1339 @defopt url-user-agent
1340 The User Agent string used for sending HTTP/HTTPS requests. The value
1341 should be a string or a function of no arguments that returns a
1342 string. The default value is @w{@samp{User-Agent: @var{package-name}
1343 URL/Emacs}}, where @var{package-name} is the value of
1344 @code{url-package-name} and its version, if they are non-@code{nil}.
1347 @node GNU Free Documentation License
1348 @appendix GNU Free Documentation License
1349 @include doclicense.texi
1351 @node Function Index
1352 @unnumbered Command and Function Index
1355 @node Variable Index
1356 @unnumbered Variable Index
1360 @unnumbered Concept Index