]> code.delx.au - gnu-emacs/blob - doc/misc/ert.texi
Add 2012 to FSF copyright years for Emacs files
[gnu-emacs] / doc / misc / ert.texi
1 \input texinfo
2 @c %**start of header
3 @setfilename ../../info/ert
4 @settitle Emacs Lisp Regression Testing
5 @c %**end of header
6
7 @dircategory Emacs
8 @direntry
9 * ERT: (ert). Emacs Lisp Regression Testing.
10 @end direntry
11
12 @copying
13 Copyright @copyright{} 2008, 2010-2012 Free Software Foundation, Inc.
14
15 @quotation
16 Permission is granted to copy, distribute and/or modify this document
17 under the terms of the GNU Free Documentation License, Version 1.3 or
18 any later version published by the Free Software Foundation; with no
19 Invariant Sections, with the Front-Cover texts being ``A GNU Manual,''
20 and with the Back-Cover Texts as in (a) below. A copy of the license
21 is included in the section entitled ``GNU Free Documentation License''
22 in the Emacs manual.
23
24 (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and
25 modify this GNU manual. Buying copies from the FSF supports it in
26 developing GNU and promoting software freedom.''
27
28 This document is part of a collection distributed under the GNU Free
29 Documentation License. If you want to distribute this document
30 separately from the collection, you can do so by adding a copy of the
31 license to the document, as described in section 6 of the license.
32 @end quotation
33 @end copying
34
35 @node Top, Introduction, (dir), (dir)
36 @top ERT: Emacs Lisp Regression Testing
37
38 ERT is a tool for automated testing in Emacs Lisp. Its main features
39 are facilities for defining tests, running them and reporting the
40 results, and for debugging test failures interactively.
41
42 ERT is similar to tools for other environments such as JUnit, but has
43 unique features that take advantage of the dynamic and interactive
44 nature of Emacs. Despite its name, it works well both for test-driven
45 development (see
46 @url{http://en.wikipedia.org/wiki/Test-driven_development}) and for
47 traditional software development methods.
48
49 @menu
50 * Introduction:: A simple example of an ERT test.
51 * How to Run Tests:: Run tests in Emacs or from the command line.
52 * How to Write Tests:: How to add tests to your Emacs Lisp code.
53 * How to Debug Tests:: What to do if a test fails.
54 * Extending ERT:: ERT is extensible in several ways.
55 * Other Testing Concepts:: Features not in ERT.
56
57 @detailmenu
58 --- The Detailed Node Listing ---
59
60 How to Run Tests
61
62 * Running Tests Interactively:: Run tests in your current Emacs.
63 * Running Tests in Batch Mode:: Run tests in emacs -Q.
64 * Test Selectors:: Choose which tests to run.
65
66 How to Write Tests
67
68 * The @code{should} Macro:: A powerful way to express assertions.
69 * Expected Failures:: Tests for known bugs.
70 * Tests and Their Environment:: Don't depend on customizations; no side effects.
71 * Useful Techniques:: Some examples.
72
73 How to Debug Tests
74
75 * Understanding Explanations:: How ERT gives details on why an assertion failed.
76 * Interactive Debugging:: Tools available in the ERT results buffer.
77
78 Extending ERT
79
80 * Defining Explanation Functions:: Teach ERT about more predicates.
81 * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes.
82
83 Other Testing Concepts
84
85 * Mocks and Stubs:: Stubbing out code that is irrelevant to the test.
86 * Fixtures and Test Suites:: How ERT differs from tools for other languages.
87
88 @end detailmenu
89 @end menu
90
91 @node Introduction, How to Run Tests, Top, Top
92 @chapter Introduction
93
94 ERT allows you to define @emph{tests} in addition to functions,
95 macros, variables, and the other usual Lisp constructs. Tests are
96 simply Lisp code --- code that invokes other code and checks whether
97 it behaves as expected.
98
99 ERT keeps track of the tests that are defined and provides convenient
100 commands to run them to verify whether the definitions that are
101 currently loaded in Emacs pass the tests.
102
103 Some Lisp files have comments like the following (adapted from the
104 package @code{pp.el}):
105
106 @lisp
107 ;; (pp-to-string '(quote quote)) ; expected: "'quote"
108 ;; (pp-to-string '((quote a) (quote b))) ; expected: "('a 'b)\n"
109 ;; (pp-to-string '('a 'b)) ; same as above
110 @end lisp
111
112 The code contained in these comments can be evaluated from time to
113 time to compare the output with the expected output. ERT formalizes
114 this and introduces a common convention, which simplifies Emacs
115 development, since programmers no longer have to manually find and
116 evaluate such comments.
117
118 An ERT test definition equivalent to the above comments is this:
119
120 @lisp
121 (ert-deftest pp-test-quote ()
122 "Tests the rendering of `quote' symbols in `pp-to-string'."
123 (should (equal (pp-to-string '(quote quote)) "'quote"))
124 (should (equal (pp-to-string '((quote a) (quote b))) "('a 'b)\n"))
125 (should (equal (pp-to-string '('a 'b)) "('a 'b)\n")))
126 @end lisp
127
128 If you know @code{defun}, the syntax of @code{ert-deftest} should look
129 familiar: This example defines a test named @code{pp-test-quote} that
130 will pass if the three calls to @code{equal} all return true
131 (non-nil).
132
133 @code{should} is a macro with the same meaning as @code{assert} but
134 better error reporting. @xref{The @code{should} Macro}.
135
136 Each test should have a name that describes what functionality it tests.
137 Test names can be chosen arbitrarily --- they are in a
138 namespace separate from functions and variables --- but should follow
139 the usual Emacs Lisp convention of having a prefix that indicates
140 which package they belong to. Test names are displayed by ERT when
141 reporting failures and can be used when selecting which tests to run.
142
143 The empty parentheses @code{()} in the first line don't currently have
144 any meaning and are reserved for future extension. They also make
145 the syntax of @code{ert-deftest} more similar to that of @code{defun}.
146
147 The docstring describes what feature this test tests. When running
148 tests interactively, the first line of the docstring is displayed for
149 tests that fail, so it is good if the first line makes sense on its
150 own.
151
152 The body of a test can be arbitrary Lisp code. It should have as few
153 side effects as possible; each test should be written to clean up
154 after itself, leaving Emacs in the same state as it was before the
155 test. Tests should clean up even if they fail. @xref{Tests and Their
156 Environment}.
157
158
159 @node How to Run Tests, How to Write Tests, Introduction, Top
160 @chapter How to Run Tests
161
162 You can run tests either in the Emacs you are working in, or on the
163 command line in a separate Emacs process in batch mode (i.e., with no
164 user interface). The former mode is convenient during interactive
165 development, the latter is useful to make sure that tests pass
166 independently of your customizations; and it allows you to invoke
167 tests from makefiles, and to write scripts that run tests in several
168 different Emacs versions.
169
170 @menu
171 * Running Tests Interactively:: Run tests in your current Emacs.
172 * Running Tests in Batch Mode:: Run tests in emacs -Q.
173 * Test Selectors:: Choose which tests to run.
174 @end menu
175
176
177 @node Running Tests Interactively, Running Tests in Batch Mode, How to Run Tests, How to Run Tests
178 @section Running Tests Interactively
179
180 You can run the tests that are currently defined in your Emacs with
181 the command @kbd{@kbd{M-x} ert @kbd{RET} t @kbd{RET}}. (For an
182 explanation of the @code{t} argument, @pxref{Test Selectors}.) ERT will pop
183 up a new buffer, the ERT results buffer, showing the results of the
184 tests run. It looks like this:
185
186 @example
187 Selector: t
188 Passed: 31
189 Failed: 2 (2 unexpected)
190 Total: 33/33
191
192 Started at: 2008-09-11 08:39:25-0700
193 Finished.
194 Finished at: 2008-09-11 08:39:27-0700
195
196 FF...............................
197
198 F addition-test
199 (ert-test-failed
200 ((should
201 (=
202 (+ 1 2)
203 4))
204 :form
205 (= 3 4)
206 :value nil))
207
208 F list-test
209 (ert-test-failed
210 ((should
211 (equal
212 (list 'a 'b 'c)
213 '(a b d)))
214 :form
215 (equal
216 (a b c)
217 (a b d))
218 :value nil :explanation
219 (list-elt 2
220 (different-atoms c d))))
221 @end example
222
223 At the top, there is a summary of the results: we ran all tests defined
224 in the current Emacs (@code{Selector: t}), 31 of them passed, and 2
225 failed unexpectedly. @xref{Expected Failures}, for an explanation of
226 the term @emph{unexpected} in this context.
227
228 The line of dots and @code{F}s is a progress bar where each character
229 represents one test; it fills while the tests are running. A dot
230 means that the test passed, an @code{F} means that it failed. Below
231 the progress bar, ERT shows details about each test that had an
232 unexpected result. In the example above, there are two failures, both
233 due to failed @code{should} forms. @xref{Understanding Explanations},
234 for more details.
235
236 In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between
237 buttons. Each name of a function or macro in this buffer is a button;
238 moving point to it and typing @kbd{RET} jumps to its definition.
239
240 Pressing @kbd{r} re-runs the test near point on its own. Pressing
241 @kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the
242 definition of the test near point (@kbd{RET} has the same effect if
243 point is on the name of the test). On a failed test, @kbd{b} shows
244 the backtrace of the failure.
245
246 @kbd{l} shows the list of @code{should} forms executed in the test.
247 If any messages were generated (with the Lisp function @code{message})
248 in a test or any of the code that it invoked, @kbd{m} will show them.
249
250 By default, long expressions in the failure details are abbreviated
251 using @code{print-length} and @code{print-level}. Pressing @kbd{L}
252 while point is on a test failure will increase the limits to show more
253 of the expression.
254
255
256 @node Running Tests in Batch Mode, Test Selectors, Running Tests Interactively, How to Run Tests
257 @section Running Tests in Batch Mode
258
259 ERT supports automated invocations from the command line or from
260 scripts or makefiles. There are two functions for this purpose,
261 @code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}.
262 They can be used like this:
263
264 @example
265 emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit
266 @end example
267
268 This command will start up Emacs in batch mode, load ERT, load
269 @code{my-tests.el}, and run all tests defined in it. It will exit
270 with a zero exit status if all tests passed, or nonzero if any tests
271 failed or if anything else went wrong. It will also print progress
272 messages and error diagnostics to standard output.
273
274 If ERT is not part of your Emacs distribution, you may need to use
275 @code{-L /path/to/ert/} so that Emacs can find it. You may need
276 additional @code{-L} flags to ensure that @code{my-tests.el} and all the
277 files that it requires are on your @code{load-path}.
278
279
280 @node Test Selectors, , Running Tests in Batch Mode, How to Run Tests
281 @section Test Selectors
282
283 Functions like @code{ert} accept a @emph{test selector}, a Lisp
284 expression specifying a set of tests. Test selector syntax is similar
285 to Common Lisp's type specifier syntax:
286
287 @itemize
288 @item @code{nil} selects no tests.
289 @item @code{t} selects all tests.
290 @item @code{:new} selects all tests that have not been run yet.
291 @item @code{:failed} and @code{:passed} select tests according to their most recent result.
292 @item @code{:expected}, @code{:unexpected} select tests according to their most recent result.
293 @item A string is a regular expression that selects all tests with matching names.
294 @item A test (i.e., an object of @code{ert-test} data type) selects that test.
295 @item A symbol selects the test that the symbol names.
296 @item @code{(member TESTS...)} selects the elements of TESTS, a list of
297 tests or symbols naming tests.
298 @item @code{(eql TEST)} selects TEST, a test or a symbol naming a test.
299 @item @code{(and SELECTORS...)} selects the tests that match all SELECTORS.
300 @item @code{(or SELECTORS...)} selects the tests that match any SELECTOR.
301 @item @code{(not SELECTOR)} selects all tests that do not match SELECTOR.
302 @item @code{(tag TAG)} selects all tests that have TAG on their tags list.
303 (Tags are optional labels you can apply to tests when you define them.)
304 @item @code{(satisfies PREDICATE)} selects all tests that satisfy PREDICATE,
305 a function that takes a test as argument and returns non-nil if it is selected.
306 @end itemize
307
308 Selectors that are frequently useful when selecting tests to run
309 include @code{t} to run all tests that are currently defined in Emacs,
310 @code{"^foo-"} to run all tests in package @code{foo} (this assumes
311 that package @code{foo} uses the prefix @code{foo-} for its test names),
312 result-based selectors such as @code{(or :new :unexpected)} to
313 run all tests that have either not run yet or that had an unexpected
314 result in the last run, and tag-based selectors such as @code{(not
315 (tag :causes-redisplay))} to run all tests that are not tagged
316 @code{:causes-redisplay}.
317
318
319 @node How to Write Tests, How to Debug Tests, How to Run Tests, Top
320 @chapter How to Write Tests
321
322 ERT lets you define tests in the same way you define functions. You
323 can type @code{ert-deftest} forms in a buffer and evaluate them there
324 with @code{eval-defun} or @code{compile-defun}, or you can save the
325 file and load it, optionally byte-compiling it first.
326
327 Just like @code{find-function} is only able to find where a function
328 was defined if the function was loaded from a file, ERT is only able
329 to find where a test was defined if the test was loaded from a file.
330
331
332 @menu
333 * The @code{should} Macro:: A powerful way to express assertions.
334 * Expected Failures:: Tests for known bugs.
335 * Tests and Their Environment:: Don't depend on customizations; no side effects.
336 * Useful Techniques:: Some examples.
337 @end menu
338
339 @node The @code{should} Macro, Expected Failures, How to Write Tests, How to Write Tests
340 @section The @code{should} Macro
341
342 Test bodies can include arbitrary code; but to be useful, they need to
343 check whether the code being tested (or @emph{code under test})
344 does what it is supposed to do. The macro @code{should} is similar to
345 @code{assert} from the cl package
346 (@pxref{Assertions,,, cl, Common Lisp Extensions}),
347 but analyzes its argument form and records information that ERT can
348 display to help debugging.
349
350 This test definition
351
352 @lisp
353 (ert-deftest addition-test ()
354 (should (= (+ 1 2) 4)))
355 @end lisp
356
357 will produce this output when run via @kbd{M-x ert}:
358
359 @example
360 F addition-test
361 (ert-test-failed
362 ((should
363 (=
364 (+ 1 2)
365 4))
366 :form
367 (= 3 4)
368 :value nil))
369 @end example
370
371 In this example, @code{should} recorded the fact that (= (+ 1 2) 4)
372 reduced to (= 3 4) before it reduced to nil. When debugging why the
373 test failed, it helps to know that the function @code{+} returned 3
374 here. ERT records the return value for any predicate called directly
375 within @code{should}.
376
377 In addition to @code{should}, ERT provides @code{should-not}, which
378 checks that the predicate returns nil, and @code{should-error}, which
379 checks that the form called within it signals an error. An example
380 use of @code{should-error}:
381
382 @lisp
383 (ert-deftest test-divide-by-zero ()
384 (should-error (/ 1 0)
385 :type 'arith-error))
386 @end lisp
387
388 This checks that dividing one by zero signals an error of type
389 @code{arith-error}. The @code{:type} argument to @code{should-error}
390 is optional; if absent, any type of error is accepted.
391 @code{should-error} returns an error description of the error that was
392 signaled, to allow additional checks to be made. The error
393 description has the format @code{(ERROR-SYMBOL . DATA)}.
394
395 There is no @code{should-not-error} macro since tests that signal an
396 error fail anyway, so @code{should-not-error} is effectively the
397 default.
398
399 @xref{Understanding Explanations}, for more details on what
400 @code{should} reports.
401
402
403 @node Expected Failures, Tests and Their Environment, The @code{should} Macro, How to Write Tests
404 @section Expected Failures
405
406 Some bugs are complicated to fix, or not very important, and are left as
407 @emph{known bugs}. If there is a test case that triggers the bug and
408 fails, ERT will alert you of this failure every time you run all
409 tests. For known bugs, this alert is a distraction. The way to
410 suppress it is to add @code{:expected-result :failed} to the test
411 definition:
412
413 @lisp
414 (ert-deftest future-bug ()
415 "Test `time-forward' with negative arguments.
416 Since this functionality isn't implemented, the test is known to fail."
417 :expected-result :failed
418 (time-forward -1))
419 @end lisp
420
421 ERT will still display a small @code{f} in the progress bar as a
422 reminder that there is a known bug, and will count the test as failed,
423 but it will be quiet about it otherwise.
424
425 An alternative to marking the test as a known failure this way is to
426 delete the test. This is a good idea if there is no intent to fix it,
427 i.e., if the behavior that was formerly considered a bug has become an
428 accepted feature.
429
430 In general, however, it can be useful to keep tests that are known to
431 fail. If someone wants to fix the bug, they will have a very good
432 starting point: an automated test case that reproduces the bug. This
433 makes it much easier to fix the bug, demonstrate that it is fixed, and
434 prevent future regressions.
435
436 ERT displays the same kind of alerts for tests that pass unexpectedly
437 as it displays for unexpected failures. This way, if you make code
438 changes that happen to fix a bug that you weren't aware of, you will
439 know to remove the @code{:expected-result} clause of that test and
440 close the corresponding bug report, if any.
441
442 Since @code{:expected-result} evaluates its argument when the test is
443 loaded, tests can be marked as known failures only on certain Emacs
444 versions, specific architectures, etc.:
445
446 @lisp
447 (ert-deftest foo ()
448 "A test that is expected to fail on Emacs 23 but succeed elsewhere."
449 :expected-result (if (string-match "GNU Emacs 23[.]" (emacs-version))
450 :failed
451 :passed)
452 ...)
453 @end lisp
454
455
456 @node Tests and Their Environment, Useful Techniques, Expected Failures, How to Write Tests
457 @section Tests and Their Environment
458
459 The outcome of running a test should not depend on the current state
460 of the environment, and each test should leave its environment in the
461 same state it found it in. In particular, a test should not depend on
462 any Emacs customization variables or hooks, and if it has to make any
463 changes to Emacs's state or state external to Emacs (such as the file
464 system), it should undo these changes before it returns, regardless of
465 whether it passed or failed.
466
467 Tests should not depend on the environment because any such
468 dependencies can make the test brittle or lead to failures that occur
469 only under certain circumstances and are hard to reproduce. Of
470 course, the code under test may have settings that affect its
471 behavior. In that case, it is best to make the test @code{let}-bind
472 all such setting variables to set up a specific configuration for the
473 duration of the test. The test can also set up a number of different
474 configurations and run the code under test with each.
475
476 Tests that have side effects on their environment should restore it to
477 its original state because any side effects that persist after the
478 test can disrupt the workflow of the programmer running the tests. If
479 the code under test has side effects on Emacs's current state, such as
480 on the current buffer or window configuration, the test should create
481 a temporary buffer for the code to manipulate (using
482 @code{with-temp-buffer}), or save and restore the window configuration
483 (using @code{save-window-excursion}), respectively. For aspects of
484 the state that can not be preserved with such macros, cleanup should
485 be performed with @code{unwind-protect}, to ensure that the cleanup
486 occurs even if the test fails.
487
488 An exception to this are messages that the code under test prints with
489 @code{message} and similar logging; tests should not bother restoring
490 the @code{*Message*} buffer to its original state.
491
492 The above guidelines imply that tests should avoid calling highly
493 customizable commands such as @code{find-file}, except, of course, if
494 such commands are what they want to test. The exact behavior of
495 @code{find-file} depends on many settings such as
496 @code{find-file-wildcards}, @code{enable-local-variables}, and
497 @code{auto-mode-alist}. It is difficult to write a meaningful test if
498 its behavior can be affected by so many external factors. Also,
499 @code{find-file} has side effects that are hard to predict and thus
500 hard to undo: It may create a new buffer or reuse an existing
501 buffer if one is already visiting the requested file; and it runs
502 @code{find-file-hook}, which can have arbitrary side effects.
503
504 Instead, it is better to use lower-level mechanisms with simple and
505 predictable semantics like @code{with-temp-buffer}, @code{insert} or
506 @code{insert-file-contents-literally}, and to activate any desired mode
507 by calling the corresponding function directly --- after binding the
508 hook variables to nil. This avoids the above problems.
509
510
511 @node Useful Techniques, , Tests and Their Environment, How to Write Tests
512 @section Useful Techniques when Writing Tests
513
514 Testing simple functions that have no side effects and no dependencies
515 on their environment is easy. Such tests often look like this:
516
517 @lisp
518 (ert-deftest ert-test-mismatch ()
519 (should (eql (ert--mismatch "" "") nil))
520 (should (eql (ert--mismatch "" "a") 0))
521 (should (eql (ert--mismatch "a" "a") nil))
522 (should (eql (ert--mismatch "ab" "a") 1))
523 (should (eql (ert--mismatch "Aa" "aA") 0))
524 (should (eql (ert--mismatch '(a b c) '(a b d)) 2)))
525 @end lisp
526
527 This test calls the function @code{ert--mismatch} several times with
528 various combinations of arguments and compares the return value to the
529 expected return value. (Some programmers prefer @code{(should (eql
530 EXPECTED ACTUAL))} over the @code{(should (eql ACTUAL EXPECTED))}
531 shown here. ERT works either way.)
532
533 Here's a more complicated test:
534
535 @lisp
536 (ert-deftest ert-test-record-backtrace ()
537 (let ((test (make-ert-test :body (lambda () (ert-fail "foo")))))
538 (let ((result (ert-run-test test)))
539 (should (ert-test-failed-p result))
540 (with-temp-buffer
541 (ert--print-backtrace (ert-test-failed-backtrace result))
542 (goto-char (point-min))
543 (end-of-line)
544 (let ((first-line (buffer-substring-no-properties
545 (point-min) (point))))
546 (should (equal first-line
547 " signal(ert-test-failed (\"foo\"))")))))))
548 @end lisp
549
550 This test creates a test object using @code{make-ert-test} whose body
551 will immediately signal failure. It then runs that test and asserts
552 that it fails. Then, it creates a temporary buffer and invokes
553 @code{ert--print-backtrace} to print the backtrace of the failed test
554 to the current buffer. Finally, it extracts the first line from the
555 buffer and asserts that it matches what we expect. It uses
556 @code{buffer-substring-no-properties} and @code{equal} to ignore text
557 properties; for a test that takes properties into account,
558 @code{buffer-substring} and @code{ert-equal-including-properties}
559 could be used instead.
560
561 The reason why this test only checks the first line of the backtrace
562 is that the remainder of the backtrace is dependent on ERT's internals
563 as well as whether the code is running interpreted or compiled. By
564 looking only at the first line, the test checks a useful property
565 --- that the backtrace correctly captures the call to @code{signal} that
566 results from the call to @code{ert-fail} --- without being brittle.
567
568 This example also shows that writing tests is much easier if the code
569 under test was structured with testing in mind.
570
571 For example, if @code{ert-run-test} accepted only symbols that name
572 tests rather than test objects, the test would need a name for the
573 failing test, which would have to be a temporary symbol generated with
574 @code{make-symbol}, to avoid side effects on Emacs's state. Choosing
575 the right interface for @code{ert-run-tests} allows the test to be
576 simpler.
577
578 Similarly, if @code{ert--print-backtrace} printed the backtrace to a
579 buffer with a fixed name rather than the current buffer, it would be
580 much harder for the test to undo the side effect. Of course, some
581 code somewhere needs to pick the buffer name. But that logic is
582 independent of the logic that prints backtraces, and keeping them in
583 separate functions allows us to test them independently.
584
585 A lot of code that you will encounter in Emacs was not written with
586 testing in mind. Sometimes, the easiest way to write tests for such
587 code is to restructure the code slightly to provide better interfaces
588 for testing. Usually, this makes the interfaces easier to use as
589 well.
590
591
592 @node How to Debug Tests, Extending ERT, How to Write Tests, Top
593 @chapter How to Debug Tests
594
595 This section describes how to use ERT's features to understand why
596 a test failed.
597
598
599 @menu
600 * Understanding Explanations:: How ERT gives details on why an assertion failed.
601 * Interactive Debugging:: Tools available in the ERT results buffer.
602 @end menu
603
604
605 @node Understanding Explanations, Interactive Debugging, How to Debug Tests, How to Debug Tests
606 @section Understanding Explanations
607
608 Failed @code{should} forms are reported like this:
609
610 @example
611 F addition-test
612 (ert-test-failed
613 ((should
614 (=
615 (+ 1 2)
616 4))
617 :form
618 (= 3 4)
619 :value nil))
620 @end example
621
622 ERT shows what the @code{should} expression looked like and what
623 values its subexpressions had: The source code of the assertion was
624 @code{(should (= (+ 1 2) 4))}, which applied the function @code{=} to
625 the arguments @code{3} and @code{4}, resulting in the value
626 @code{nil}. In this case, the test is wrong; it should expect 3
627 rather than 4.
628
629 If a predicate like @code{equal} is used with @code{should}, ERT
630 provides a so-called @emph{explanation}:
631
632 @example
633 F list-test
634 (ert-test-failed
635 ((should
636 (equal
637 (list 'a 'b 'c)
638 '(a b d)))
639 :form
640 (equal
641 (a b c)
642 (a b d))
643 :value nil :explanation
644 (list-elt 2
645 (different-atoms c d))))
646 @end example
647
648 In this case, the function @code{equal} was applied to the arguments
649 @code{(a b c)} and @code{(a b d)}. ERT's explanation shows that
650 the item at index 2 differs between the two lists; in one list, it is
651 the atom c, in the other, it is the atom d.
652
653 In simple examples like the above, the explanation is unnecessary.
654 But in cases where the difference is not immediately apparent, it can
655 save time:
656
657 @example
658 F test1
659 (ert-test-failed
660 ((should
661 (equal x y))
662 :form
663 (equal a a)
664 :value nil :explanation
665 (different-symbols-with-the-same-name a a)))
666 @end example
667
668 ERT only provides explanations for predicates that have an explanation
669 function registered. @xref{Defining Explanation Functions}.
670
671
672 @node Interactive Debugging, , Understanding Explanations, How to Debug Tests
673 @section Interactive Debugging
674
675 Debugging failed tests essentially works the same way as debugging any
676 other problems with Lisp code. Here are a few tricks specific to
677 tests:
678
679 @itemize
680 @item Re-run the failed test a few times to see if it fails in the same way
681 each time. It's good to find out whether the behavior is
682 deterministic before spending any time looking for a cause. In the
683 ERT results buffer, @kbd{r} re-runs the selected test.
684
685 @item Use @kbd{.} to jump to the source code of the test to find out exactly
686 what it does. Perhaps the test is broken rather than the code
687 under test.
688
689 @item If the test contains a series of @code{should} forms and you can't
690 tell which one failed, use @kbd{l}, which shows you the list of all
691 @code{should} forms executed during the test before it failed.
692
693 @item Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run
694 the test with debugging enabled, this will enter the debugger and show
695 the backtrace as well; but the top few frames shown there will not be
696 relevant to you since they are ERT's own debugger hook. @kbd{b}
697 strips them out, so it is more convenient.
698
699 @item If the test or the code under testing prints messages using
700 @code{message}, use @kbd{m} to see what messages it printed before it
701 failed. This can be useful to figure out how far it got.
702
703 @item You can instrument tests for debugging the same way you instrument
704 @code{defun}s for debugging --- go to the source code of the test and
705 type @kbd{@kbd{C-u} @kbd{C-M-x}}. Then, go back to the ERT buffer and
706 re-run the test with @kbd{r} or @kbd{d}.
707
708 @item If you have been editing and rearranging tests, it is possible that
709 ERT remembers an old test that you have since renamed or removed ---
710 renamings or removals of definitions in the source code leave around a
711 stray definition under the old name in the running process (this is a
712 common problem in Lisp). In such a situation, hit @kbd{D} to let ERT
713 forget about the obsolete test.
714 @end itemize
715
716
717 @node Extending ERT, Other Testing Concepts, How to Debug Tests, Top
718 @chapter Extending ERT
719
720 There are several ways to add functionality to ERT.
721
722 @menu
723 * Defining Explanation Functions:: Teach ERT about more predicates.
724 * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes.
725 @end menu
726
727
728 @node Defining Explanation Functions, Low-Level Functions for Working with Tests, Extending ERT, Extending ERT
729 @section Defining Explanation Functions
730
731 The explanation function for a predicate is a function that takes the
732 same arguments as the predicate and returns an @emph{explanation}.
733 The explanation should explain why the predicate, when invoked with
734 the arguments given to the explanation function, returns the value
735 that it returns. The explanation can be any object but should have a
736 comprehensible printed representation. If the return value of the
737 predicate needs no explanation for a given list of arguments, the
738 explanation function should return nil.
739
740 To associate an explanation function with a predicate, add the
741 property @code{ert-explainer} to the symbol that names the predicate.
742 The value of the property should be the symbol that names the
743 explanation function.
744
745
746 @node Low-Level Functions for Working with Tests, , Defining Explanation Functions, Extending ERT
747 @section Low-Level Functions for Working with Tests
748
749 Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch}
750 are implemented on top of the lower-level test handling code in the
751 sections of @file{ert.el} labeled ``Facilities for running a single test'',
752 ``Test selectors'', and ``Facilities for running a whole set of tests''.
753
754 If you want to write code that works with ERT tests, you should take a
755 look at this lower-level code. Symbols that start with @code{ert--}
756 are internal to ERT, whereas those that start with @code{ert-} are
757 meant to be usable by other code. But there is no mature API yet.
758
759 Contributions to ERT are welcome.
760
761
762 @node Other Testing Concepts, , Extending ERT, Top
763 @chapter Other Testing Concepts
764
765 For information on mocks, stubs, fixtures, or test suites, see below.
766
767
768 @menu
769 * Mocks and Stubs:: Stubbing out code that is irrelevant to the test.
770 * Fixtures and Test Suites:: How ERT differs from tools for other languages.
771 @end menu
772
773 @node Mocks and Stubs, Fixtures and Test Suites, Other Testing Concepts, Other Testing Concepts
774 @section Other Tools for Emacs Lisp
775
776 Stubbing out functions or using so-called @emph{mocks} can make it
777 easier to write tests. See
778 @url{http://en.wikipedia.org/wiki/Mock_object} for an explanation of
779 the corresponding concepts in object-oriented languages.
780
781 ERT does not have built-in support for mocks or stubs. The package
782 @code{el-mock} (see @url{http://www.emacswiki.org/emacs/el-mock.el})
783 offers mocks for Emacs Lisp and can be used in conjunction with ERT.
784
785
786 @node Fixtures and Test Suites, , Mocks and Stubs, Other Testing Concepts
787 @section Fixtures and Test Suites
788
789 In many ways, ERT is similar to frameworks for other languages like
790 SUnit or JUnit. However, two features commonly found in such
791 frameworks are notably absent from ERT: fixtures and test suites.
792
793 Fixtures are mainly used (e.g., in SUnit or JUnit) to provide an
794 environment for a set of tests, and consist of set-up and tear-down
795 functions.
796
797 While fixtures are a useful syntactic simplification in other
798 languages, this does not apply to Lisp, where higher-order functions
799 and `unwind-protect' are available. One way to implement and use a
800 fixture in ERT is
801
802 @lisp
803 (defun my-fixture (body)
804 (unwind-protect
805 (progn [set up]
806 (funcall body))
807 [tear down]))
808
809 (ert-deftest my-test ()
810 (my-fixture
811 (lambda ()
812 [test code])))
813 @end lisp
814
815 (Another way would be a @code{with-my-fixture} macro.) This solves
816 the set-up and tear-down part, and additionally allows any test
817 to use any combination of fixtures, so it is more flexible than what
818 other tools typically allow.
819
820 If the test needs access to the environment the fixture sets up, the
821 fixture can be modified to pass arguments to the body.
822
823 These are well-known Lisp techniques. Special syntax for them could
824 be added but would provide only a minor simplification.
825
826 (If you are interested in such syntax, note that splitting set-up and
827 tear-down into separate functions, like *Unit tools usually do, makes
828 it impossible to establish dynamic `let' bindings as part of the
829 fixture. So, blindly imitating the way fixtures are implemented in
830 other languages would be counter-productive in Lisp.)
831
832 The purpose of test suites is to group related tests together.
833
834 The most common use of this is to run just the tests for one
835 particular module. Since symbol prefixes are the usual way of
836 separating module namespaces in Emacs Lisp, test selectors already
837 solve this by allowing regexp matching on test names; e.g., the
838 selector "^ert-" selects ERT's self-tests.
839
840 Other uses include grouping tests by their expected execution time,
841 e.g. to run quick tests during interactive development and slow tests less
842 often. This can be achieved with the @code{:tag} argument to
843 @code{ert-deftest} and @code{tag} test selectors.
844
845 @bye
846
847 @c LocalWords: ERT JUnit namespace docstring ERT's
848 @c LocalWords: backtrace makefiles workflow backtraces API SUnit
849 @c LocalWords: subexpressions