Unified · Split
| @@ -1,23 +1,115 @@ | |||
| 1 | FATE Automated Testing Environment | 1 | FATE Automated Testing Environment |
| 2 | ================================== | ||
| 2 | 3 | ||
| 3 | FATE provides a regression testsuite that can be run locally or configured | 4 | FATE is an extended regression suite on the client-side and a means |
| 4 | to send reports to fate.ffmpeg.org. | 5 | for results aggregation and presentation on the server-side. |
| 5 | In order to run, it needs a large amount of data (samples and references) | ||
| 6 | that is provided separately from the actual source distribution. | ||
| 7 | 6 | ||
| 8 | Use the following command to get the fate test samples | 7 | The first part of this document explains how you can use FATE from |
| 8 | your FFmpeg source directory to test your ffmpeg binary. The second | ||
| 9 | part describes how you can run FATE to submit the results to FFmpeg's | ||
| 10 | FATE server. | ||
| 11 | |||
| 12 | In any way you can have a look at the publicly viewable FATE results | ||
| 13 | by visiting this website: | ||
| 14 | |||
| 15 | http://fate.ffmpeg.org/ | ||
| 16 | |||
| 17 | This is especially recommended for all people contributing source | ||
| 18 | code to FFmpeg, as it can be seen if some test on some platform broke | ||
| 19 | with there recent contribution. This usually happens on the platforms | ||
| 20 | the developers could not test on. | ||
| 21 | |||
| 22 | The second part of this document describes how you can run FATE to | ||
| 23 | submit your results to FFmpeg's FATE server. If you want to submit your | ||
| 24 | results be sure to check that your combination of CPU, OS and compiler | ||
| 25 | is not already listed on the above mentioned website. | ||
| 26 | |||
| 27 | In the third part you can find a comprehensive listing of FATE makefile | ||
| 28 | targets and variables. | ||
| 29 | |||
| 30 | |||
| 31 | 1. Using FATE from your FFmpeg source directory | ||
| 32 | ----------------------------------------------- | ||
| 33 | |||
| 34 | If you want to run FATE on your machine you need to have the samples | ||
| 35 | in place. You can get the samples via the build target fate-rsync. | ||
| 36 | Use this command from the top-level source directory: | ||
| 9 | 37 | ||
| 10 | # make fate-rsync SAMPLES=fate-suite/ | 38 | # make fate-rsync SAMPLES=fate-suite/ |
| 39 | # make fate SAMPLES=fate-suite/ | ||
| 40 | |||
| 41 | The above commands set the samples location by passing a makefile | ||
| 42 | variable via command line. It is also possible to set the samples | ||
| 43 | location at source configuration time by invoking configure with | ||
| 44 | `--samples=<path to the samples directory>'. Afterwards you can | ||
| 45 | invoke the makefile targets without setting the SAMPLES makefile | ||
| 46 | variable. This is illustrated by the following commands: | ||
| 47 | |||
| 48 | # ./configure --samples=fate-suite/ | ||
| 49 | # make fate-rsync | ||
| 50 | # make fate | ||
| 11 | 51 | ||
| 12 | To inform the build system about the testsuite location, pass | 52 | Yet another way to tell FATE about the location of the sample |
| 13 | `--samples=<path to the samples>` to configure or set the SAMPLES Make | 53 | directory is by making sure the environment variable FATE_SAMPLES |
| 14 | variable or the FATE_SAMPLES environment variable to a suitable value. | 54 | contains the path to your samples directory. This can be achieved |
| 15 | NOTE: Do not put a '~' character in the samples path to indicate a home | 55 | by e.g. putting that variable in your shell profile or by setting |
| 56 | it in your interactive session. | ||
| 57 | |||
| 58 | # FATE_SAMPLES=fate-suite/ make fate | ||
| 59 | |||
| 60 | NOTE: | ||
| 61 | Do not put a '~' character in the samples path to indicate a home | ||
| 16 | directory. Because of shell nuances, this will cause FATE to fail. | 62 | directory. Because of shell nuances, this will cause FATE to fail. |
| 17 | 63 | ||
| 18 | For information on how to set up FATE to send results to the official FFmpeg | 64 | |
| 19 | testing framework, please refer to the following wiki page: | 65 | 2. Submitting the results to the FFmpeg result aggregation server |
| 20 | http://wiki.multimedia.cx/index.php?title=FATE | 66 | ----------------------------------------------------------------- |
| 67 | |||
| 68 | To submit your results to the server you should run fate through the | ||
| 69 | shell script tests/fate.sh from the FFmpeg sources. This script needs | ||
| 70 | to be invoked with a configuration file as its first argument. | ||
| 71 | |||
| 72 | # tests/fate.sh /path/to/fate_config | ||
| 73 | |||
| 74 | A configuration file template with comments describing the individual | ||
| 75 | configuration variables can be found at tests/fate_config.sh.template . | ||
| 76 | |||
| 77 | Create a configuration that suits your needs, based on the configuration | ||
| 78 | template. The `slot' configuration variable can be any string that is not | ||
| 79 | yet used, but it is suggested that you name it adhering to the following | ||
| 80 | pattern <arch>-<os>-<compiler>-<compiler version>. The configuration file | ||
| 81 | itself will be sourced in a shell script, therefore all shell features may | ||
| 82 | be used. This enables you to setup the environment as you need it for your | ||
| 83 | build. | ||
| 84 | |||
| 85 | For your first test runs the `fate_recv' variable should be empty or | ||
| 86 | commented out. This will run everything as normal except that it will omit | ||
| 87 | the submission of the results to the server. The following files should be | ||
| 88 | present in $workdir as specified in the configuration file: | ||
| 89 | |||
| 90 | - configure.log | ||
| 91 | - compile.log | ||
| 92 | - test.log | ||
| 93 | - report | ||
| 94 | - version | ||
| 95 | |||
| 96 | When you have everything working properly you can create an SSH key and | ||
| 97 | send its public part to the FATE server administrator. | ||
| 98 | |||
| 99 | Configure your SSH client to use public key authentication with that key | ||
| 100 | when connecting to the FATE server. Also do not forget to check the identity | ||
| 101 | of the server and to accept its host key. This can usually be achieved by | ||
| 102 | running your SSH client manually and killing it after you accepted the key. | ||
| 103 | The FATE server's fingerprint is: | ||
| 104 | |||
| 105 | b1:31:c8:79:3f:04:1d:f8:f2:23:26:5a:fd:55:fa:92 | ||
| 106 | |||
| 107 | The only thing left is to automate the execution of the fate.sh script and | ||
| 108 | the synchronisation of the samples directory. | ||
| 109 | |||
| 110 | |||
| 111 | 3. FATE makefile targets and variables | ||
| 112 | -------------------------------------- | ||
| 21 | 113 | ||
| 22 | FATE Makefile targets: | 114 | FATE Makefile targets: |
| 23 | 115 | ||
| @@ -27,7 +119,7 @@ fate-list · 3 unmodified lines | |||
| 27 | fate | 119 | fate |
| 28 | Run the FATE test suite (requires the fate-suite dataset). | 120 | Run the FATE test suite (requires the fate-suite dataset). |
| 29 | 121 | ||
| 30 | Fate Makefile variables: | 122 | FATE Makefile variables: |
| 31 | 123 | ||
| 32 | V | 124 | V |
| 33 | Verbosity level, can be set to 0, 1 or 2. | 125 | Verbosity level, can be set to 0, 1 or 2. |