Coding Hints:Test Cases

From Wine-Wiki.org
Jump to: navigation, search

Contents

Introduction to Test Cases

This is not a guide or official recommendations. It is merely a compilation of comments from the mailing list. When reviewing this information be aware that opinions and standards change over time. Corrections and updates are invited

M. Hearn [May 05] briefly described what is wanted with patches for Wine: Unified diff from the source root, test cases, clear explanation, good comments, no extra changes ...Wine archives

McCormack gave some insight why the developers like test cases when commenting on an implementatioon of MkParseDisplayName: I like that you've taken the time to write a test case! I don't know alot about monikers, but seeing that you've tested it gives me confidence...

When writing a test case, make sure that it runs on Wine. A. Jullard[Jun 05]:[A patch that] fails on Wine can't be applied Wine Archive All the tests must always pass in CVS, otherwise the test suite is useless. If you want the test to get in before the code that's certainly possible, but you have to add todo_wine around all the failing tests. Wine Archive

S. Leichter: you have to include your test into programs/winetest/Makefile.in too, to get your test into the winetest.exe, which delivers the results for http://test.winehq.org/data/ Wine Archive

The official Wine Wiki has a page which covers writing test cases which would be the first place to look.

Tests results and testing history can be checked here http://test.winehq.org/data/

J. Hawkins: make test fails if an ok call fails and is not covered by todo_wine, and it also fails when a test in a todo_wine block succeeds. That's how we keep track of the tests. in Feb 09 it was raised about showing failures: We should leave the failing files percentage up (note the name change) and add a failing tests percentage next to it. The failing tests percentage should be total_test_failures / total_tests_run.But a. Julliard wrote: That's not a useful number, many files run a lot of tests, of which a huge majority always succeeds. Having a single failure among 10,000 tests means that the test failed, and it's something bad that should be taken care of. Showing that as a 99.99% success would be very misleading.


What should be tested?

A. Arvey: More specifically, is it worth the time to write tests for ntdll/loader.c and ntdll/virtual.c? It seems that these files would be used to load any program, so if there were any bugs, I'm guessing they would become apparent immediately and thus having formal tests would be somewhat pointless. Please correct (or reaffirm) me as I am going to write some tests in my spare time and I want to make sure that they'd be useful to the wine project.

J. Hawkins: Regression/conformance tests are almost never pointless even if they just reaffirm what is already obvious. They serve two purposes: to conform to windows behaviour, and to make sure later code changes don't create regressions. It would be well worth all of our time for anyone to write more tests, so please send them in. Wine Archive Link

J. Hawkins [May 29 2009 wine devel]:Please write a completely new test that exposes just the one bug you're trying to fix. They're called 'unit' tests for a reason.

Testing with Different versions of Windows

TestBot

To help this process, you have the Wine Test Bot. https://winetestbot.geldorp.nl/index.pl

If you are known to wine, you should have no problem getting an account, and then you can run tests and check how different versions of Windows work. This knowledge can improve your patches to wine.

Sept 10 G. Geldorp: TestBot uses (mostly) VMware ESX VMs, which do not have virtual hardware devices for sound. The sound drivers I'm adding to the ESX VMs are pure virtual drivers, there's no (virtual) hardware below them.

Writing a test

A. English [Apr 09] The unit tests should be readable for someone not familiar with Wine, but probably with the Windows API. J. Hawkins added: With a unit test, you're testing one piece of functionality. Each of these tests has a comment above it explaining what you're testing.


D. Kegel 08 Bug 15384 [a] test case as a C program that renders a pass/fail judgement without human attention, and fits into our conformance test suite? See http://www.winehq.org/site/docs/winedev-guide/testing The test should probably live in a C file [for this bug dlls/shell32/tests/brsfolder.c]. If your test needs to move or click the mouse, use the win32 functions to simulate user input. Then make sure the test passes on Windows XP and, with your patch [if you have a patch], Wine. [edited]

McCormack:The best way to write a test is to look at some of the test cases that are there already. Write and run the test under Windows, and make sure it passes on Windows first. The test is something like this:

/* filename is dlls/crypt32/tests/protect.c */
#include "wine/test.h"

START_TEST(protect)
{
     /* basic form of the test */
     ok(TRUE, "This should be true\n");

     /* try with incorrect parameters */
     ok(!CryptProtectData(NULL,NULL,NULL,NULL,NULL,0,NULL), "crypt 
protect data should fail\n");
     ok(GetLastError() == ERROR_INVALID_PARAMETER, "error returned 
incorrect\n");
     ...

     /* try with correct parameters */
     ...

     /* for tests that pass on windows, but not on Wine */
     todo_wine {
         ok( failing_test(), "this test should pass\n");
     }


D.Kegel: [For link - see Wine Archive Reference below]. Since the webmaster there says he's taking down that wiki soon, here's a copy for posterity.

Adding to the Makefile (as part of a new Wine test)

Follow the example of the lzexpand test patch: http://www.winehq.com/hypermail/wine-patches/2004/11/0182.html

Basic steps. (you'll have to chmod mod most of the files to 644 before you can edit them)

  1. In your tests directory, look over wine/dlls/lzexpand/Makefile.in and wine/dlls/lzexpand/tests/Makefile.in
  2. Now look at the Makefile.in in your dll directory. Copy it into your tests dir and change the list of .c files to your test .c file
  3. Edit the Makefile.in file in your dll directory to add a subdirs line (see lzexpand/Makefile.in for an example)
  4. In your wine directory, chmod 644 configure.ac
  5. Edit configure.ac, find your dll directory in the AC_CONFIG_FILES and then add the path to your tests directory after it.
  6. In your wine directory, run autoconf. Make sure you have 2.53 installed (autoconf --version).
  7. Run configure. Your Makefile should be listed at the end now.

D. Kegel: make sure the test passed on both wine and windows when built as a standalone program[...] e.g. if you happen to have msvc and the platform sdk installed, and have your environment set up right, you can just do

cd dlls/kernell/tests
cl -DSTANDALONE -D_X86_ -I../../../include sync.c

to compile the test as a standalone app, and then run[...] on Wine and Windows. wine archive

When sending in a patch to wine J. Hawkins [Apr 06]: You should send one change/fix per patch. The best option is to send one patch for the tests, with todo_wine around them, and then send separate patches to fix each problem. wine archive

M. McCormack explained a little about todo wine: [May 06] For tests that don't pass in wine, please enclose them in todo_wine { }, so that the Wine test suite doesn't fail. wine archive

Todo Wine

todo_wine
{
ok(hr == S_OK, "ScriptStringOut Stub should return S_OK not %08lx\n", hr);
}

J. Hawkins [Sept 06]: todo_wine doesn't stop a test from running, it just makes 'make test' not error out. If you run 'make test' and a test fails and there is no todo_wine around the test, make will stop with an error. wine archive

D. Timoshkov: If you run the test directly using something like 'wine dlls/kernel32/tests/kernel32_test.exe.so', then the tests in the todo_wine blocks will fail because the platform (wine) is not specified. The switch '-P wine' specifies a Wine platform, and todo_wine will work properly.


A new developer asked: What is the difference of test wrapped by todo_wine and test not wrapped?

P. Vriens apr 09 wine dev: todo_wine still runs the test on Wine but it's not counted as a failure. You will see a message about a test being marked though (also on test.winehq.org):

security.c:1601: Test marked todo: LookupAccountSidA() Expected account name: <redacted> got: INTERACTIVE

You have to run the tests locally in verbose mode though to see that message: ../../../tools/runtest -v -P wine -M advapi32.dll -T ../../.. -p advapi32_test.exe.so security.c

If the test within a todo_wine() for some reason succeeds it will be marked as a failure.

Skip vs Broken

P. Vriens [sept 08 winedevel]: But we tend to use broken() if we think a certain windows platform is well uh, broken. And we use skip() if it's accepted behavior.

J. Hawkins: You use skip so that you don't have to add (potentially) hundreds of broken statements.

Win_skip

P. Vriens [Jan 09]:Could you turn the skip in this piece of code:

+ if (!pGdiGetCodePage)
+ {
+ skip("GdiGetCodePage not available on this platform\n"); +
return;
+ }

into a win_skip? That way we fail if the function is not available in Wine. [...] The win_skip() means we skip the test on Windows if this function is not present but it will fail on Wine if this function is not there(exported). This will make sure that nobody messes up this function . The skip() will make sure we skip the tests on all platforms if Arial is not available.

C++ and Wine

A Programmer [May 2007]: I want to write conformance tests for gdiplus and that requires the use of c++. Even if I were to focus on the gdi+ flat api, the windows SDK headers for that have c++ code in them, also eventually when the gdi+ c++ wrappers are written on top of the flat api then we will need to be able to compile c++ in the wine environment. What do I have to do to be able to write these tests?

D. Timoshkov: According to the Wine rules C++ is not acceptable for Wine code. Obviously you have 2 choices:

  1. Cope with that fact like we do for OLE implementation/tests
  2. Create a separate project for gdiplus

A. Julliard: I don't see any need to use C++ for that. All the exported APIs are callable from C code, and the corresponding headers are C compatible too. [wine devel]

Adding a new file

A uuse asked: Which files to change (beside Makefile.in and testlist.c) so that the patch builds and becomes fully integrated into the wine source tree if it were located in a new file, instead of e.g. being added to tests/mmio.c?

M. Karcher: Just change Makefile.in, testlist.c is automatically generated. If you add a new tests dir, you need to run tools/make_makefiles in the top level directory of your wine sourcecode. If your wine sourcecode is a git repository, make_makefiles only considers Makefile.in files already added to the git index, so call git-add before make_makefiles. If you don't use git, make_makefiles scans all directories for Makefile.in to find about buildable directories.

WineTest Tips and Tricks

Running a single test from the command line

P. Vriens bug 13029 Dec 08: With the current winetest it's possible to run individual tests (dlls and/or subtests). Just invoke winetest like:

winetest.exe ntdll

or

winetest.exe kernel32:volume

There is a limit to the number of tests you can pass on the command line and the results will be available in the %TEMP% directory (*.tmp) and will not be sent to test.winehq.org.

Feb 10 A. Julliard: As of today, winetest requires an email address to submit results to test.winehq.org. This is because it's often necessary to contact the submitter of a test to get more details, or to ask them to fix their setup, and it's often hard to guess the identity of the submitter from the tag name.

If you are morally offended at the idea of your email showing up in the result log, you can use some obfuscated format, or use your real name or IRC nick if you are well-known around these parts. The important thing is that Wine developers must be able to get in touch with you easily. If you have some automated scripts submitting results, you'll need to fix them to pass a -m option to winetest.

A. English noted: FWIW, I've updated the build script on winezeug (http://winezeug.googlecode.com/svn/trunk/build_script/daily.sh) to handle it (by setting it yourself, or it will retrieve it from git, and as a last resort use $USER@$HOSTNAME). I'm sure many of you have your own scripts, but I noticed at least a couple people are using my script on test.winehq.org...


Extracting the tests

A. English: 12:47:02 Winehq Bugzilla entry 13029 --- Currently, running:

$ wine winetest-latest.exe -x

pops up winetest for a second, then makes a directory 'wct' in the current directory, with the extracted tests. A dir can also be given (-x foo). This could be improved a bit, there's no reason to launch the gui to extract it. I'll send a patch for that. Perhaps there should be an option for winetest gui to extract the files and not run the tests...?


Errs

When running tests you may see a number of errors which may seem alarming, but the answer is a little more subtle.

V. Margolen explained [apr 09 winedevel]: Lots of tests probe border cases as well as invalid conditions. Wine is 100% correct to complain about such situations. However that's precisely what's being tested. [..] tests are _allowed_ and some really _supposed to_ test invalid cases. [..] But the goal of each test is either succeed or fail. If it succeeds then there is no bug.

All tests should be written to examine one thing at a time. And most are. If test doesn't fail, everything else is irrelevant.


B. Klein summed the problem up: There is no way to distinguish an error caused by a real application from an error caused by a Wine test.

Reliabilty

[Sept 08 wine devel A. Julliard]: a test should not be using random data. We have already enough trouble with sporadic test failures, we don't want to add deliberate randomness.

Defensive Programming

[Sept 08 wine devel V Margolen]: Don't use assert for every single thing. In your test things like GetDC(NULL) and CreateCompatibleDC(hDC) just have to succeed. If they don't something is really really wrong. And the whole test will just crash. So for those things using assert is fine. Memory allocations should "just work". No need to test for that. Or if you really want to just gracefully skip the entire test in this case. But only if it doesn't add too much complexity. Tests are made to test one thing at a time. For calls that are related to what you testing and that should succeed like StretchDIBits() and GetDIBits() just use ok() to test their result. link

Test Results for Wine

A discussion arose about generating more test results on wine. R. Dunn[Feb 08 wine devel]: The easiest thing I found for generating test results on Wine is to treat it like Windows. That is, download the winetest-[buildid]-paul-mingw.exe from the build you want to test and run:

wine winetest-[buildid]-paul-mingw.exe

e.g. on the http://test.winehq.org/data/200801301937/ page, the "Main summary for build 200801301937", and downloaded http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt/winetest-200801301937-paul-mingw.exe. Then, I ran

./wine winetest-200801301937-paul-mingw.exe

from a build of wine-0.9.54 and followed the instructions on the program. It did generate an error about not being able to clean the log at the end, but it generated the "XP wine-0.9.54-notheme" and "XP wine-0.9.54" results without any issues.

Another developer asked: So...what's the convention for build.id etc? And why didn't they get set?

R. Dunn: Looking at the results from the Wine Test Shell, it is using a build timestamp. [He then pointed to the official wiki]: I have added my experiences to the wiki (http://wiki.winehq.org/ConformanceTests) with a link on the main page. Feel free to update it with your own experiences

A developer wrote: If some people went through the trouble of using buildid's and not the precompiled winetest, along with documenting their hardware/OS on a wiki page, for instance, then we could more information on where specific failures are occuring. For instance, I noticed Joapa was reporting some MSI bugs, and was the only one posting an MSI failure in the make test usually failing bug.

J. White replied [wine devel Feb 2008]: Actually, I think there is a better solution. If we want to capture additional system information, and there is a command line tool that would allow that, we can do so in the new 'runtests' script. Right now, I have it capturing gcc, glibc, and kernel version info, and that is all readily available as part of the nifty winetest results page. We could also conceivably add info on processor count and ram size, as well as video and audio card. (But those would require slightly more sophisticated scripts to postprocess; we don't want to just dump the results of xdpyinfo for example...).

MakeFile

+dlls/dplayx/tests/Makefile

D. Riekenberg: That is will not work since today, as the format changed. The correct entry is created by "tools/make_makefiles" I updated the (official Wine Developer) wiki, that autogenerated code should not be included in a Patch [wine devel jun 07]

Testing Windows behaviour

Test must run on Windows

A developer reported [Nov 06]: This is a test for the CreateWindow call to see what is done with width or height that exceed the size of a short storage, or are negative. This test is passing under Wine, but I didn't test with real Windows, so I guess I better monitor: http://test.winehq.org/data Is that the best way to go?

J. Hawkins: No, you need to either test it in Windows yourself, or have someone else run the tests on Windows for you to verify what you've written is correct. It won't be accepted otherwise.

Anon: Previous requests to the list asking for a test in Windows usually result in a quick reply.


A programmer in Aug 08 [wine devel] wanted to build and test wine test suite on window: I did

./tools/winapi/msvcmaker --no-wine

that is ok, but when I try to open the fille winetest.dsw, with Developer Studio 6.0, it compaints that all *.dsp files (such as dlls\advapi32\tests\advapi32_test.dsp) have been modified and can not be loaded as Developer Studio Project. With Developer Studio 2008, half of the *.dsp files (such as dlls\user32\tests\user32_test.dsp) was reported corrupted after been converted into the current Visual C++ project format. Does anyone know what is the problem?

R. Dunn: What is git's core.autocrlf setting configured to? I had this issue (some of the detected filenames are '\r' - which is caused by perl being confused, which then confuses Visual Studio). You need to set core.autocrlf to false:

$ git config --bool core.autocrlf false

D. Kegel: The easiest way is to cross-compile them on wine and then run them on windows. To do this, say

make crosstest

See http://www.winehq.org/site/docs/winedev-guide/cross-compiling-tests [Official wine documentation]


Because the tests test a great deal of unusual things in Windows the results displayed can be confusing. After a discussion about this D. Timoshkov April 09 Winehq Bugzilla entry 17997 wrote: A failing test run has the lines like "Test failed:", if there is no one there are no test failures. Other messages on the console could be safely ignored.

Tests that crash windows

K. Blin wrote regarding one test [Apr 09]: as this test is behind an if(0), it's not going to be run anyway. So all it does is to serve as documentation of Windows behaviour. Arguably, this could just be a comment, ideally in the implementation of GetUserNameExW, but in general it's nice to have this sort of information.

Testing a Windows Service

Restarting Windows

A user commented [July 05]: It suddenly struck me that maybe I should be restarting Windows between each test. Should I do this for best results, or is it OK to run them back to back?

Dimi Paum: You should be able to run them back to back. If that is a problem,please report it, we should fix it.Wine Archive

GetLastError

A new developer to wine wrote [Mar 09]: According to MSDN the error codes are not part of the API documentation: The error codes returned by a function are not part of the Windows API

Vitamin: But they are. MS just made their obligation to document all of their craft that much easer and that much more undefined. According to this logic "Error!" is the only message you need for anything that happened wrong. Whenever someone implements a missing part of Wine, conformance tests that verify normal behavior really help. Checking border cases are not required unless you have a real life application that depends on such a corner case.

To apply it all here - no need to test for all possible errors in conformance tests. However you should test it yourself and use correct error codes where appropriate. Archived-At: http://permalink.gmane.org/gmane.comp.emulators.wine.devel/67944

J. Hawkins [Nov 05]: You have to set the last error to a crazy value like 0xdeadbeef, because if you set the last error to something like 0, which is a possible error, there's always the possibility that wine will regress or a new version of the windows api might return 0 and you would never know. That's why you have to set the last error to a value that can't possibly be a result of GetLastError so you always know when the value changes and when it doesn't. wine archive


+ err=GetLastError();
+ ok(len == strlen(useragent)+1,"Got wrong user agent length %ld len instead of %d\n",len,strlen(useragent));
+ ok(retval == 0,"Got wrong return value %d\n",retval);
+ ok(err == ERROR_INSUFFICIENT_BUFFER, "Got wrong error code%ld\n",err);

D. Timoshkov [Nov 05]: You need to set last error to some bogus value before calling an API in order to make sure that the API really changes it. Usually setting last error to 0xdeadbeef works fine for that purpose. wine archive

D. Timoskov [Apr 06] stick to the style used everywhere else, i.e. use 0xdeadbeef directly. wine archive

A. Julliard: [Oct 06] Testing last error on success is not useful. It will only cause spurious test failures, so it should be done only if the API specifies a certain behavior, or if you have a real app that depends on it. wine archive


+    ok(GetLastError() == ERROR_FILE_NOT_FOUND,
+       "GetLastError() returns %ld\n", GetLastError());
     ok(test.cBytes == 0xA5,
       "LZOpenFileA set test.cBytes to %d\n", test.cBytes);
     ok(test.nErrCode == ERROR_FILE_NOT_FOUND,

J. Hawkins [Oct 06]:GetLastError returns a DWORD, so these formats need to be %d, not %ld. There is already a big effort to clean these up, so we don't need to introduce more. wine archive

0xdeadbeef Magic Value

A developer [May 06] suggested using another value instead of 0xdeadbeaf"

A. Julliard: This may be useful in some cases, but not in tests. In tests you know exactly which test failed, you even get the line number, so there's no reason at all to go hunting for 0xdeadbeef. wine archive

Null Pointer

A developer noted: Windows silently ignores NULL pointers supplied to SendMessageCallback as callback function.

F. Nawothnig [wine devel jun 07]:A testcase would be nice.

The developer queried this: The only way I know of to test this behavior would be to pass a NULL pointer as callback. Naturally this results in a segfault without this patch. Is this okay for a testcase?

F. Nawothnig: Use if(0) instead of todo_wine then (and let the second patch, which fixes the bug, remove the if(0)).

Float Values

D. Kegel feb09 :Exact equality comparison on float is never a good idea in unit tests. You need a range test instead. See http://code.google.com/p/googletest/wiki/GoogleTestAdvancedGuide#Floating-Point_Comparison

Inconsistant Results or Differences between versions of Windows

A programmer noted [Jun 2006]: I meant to document the conditions under which Windows 2000 yields inconsistent results. For example, in the first ok() on Windows 2000, _initterm() returned either 0 or 1. Also, one of the "#if 0" blocks documents a condition that crashes both Wine and Windows 2000.

A. Julliard: As a rule, passing garbage to an API is not an interesting thing to test,unless the API does something special with it on Windows. wine archive


A. Julliard [Nov 06]: If the behavior across Windows versions is inconsistent, it means that no app will depend on a specific behavior, so we don't need to test that we conform to it. If we do find an app that depends on it, presumably after checking the Windows version, then we can add the test again with the appropriate version check.

J. Hawkins [Feb 08 wine devel] The only time the tests should check for the windows version is if we know an app does the same thing, which is hardly ever

winedevel feb 10 A developer wrote: In exploring a bug in a game we ran across an interesting difference between how Windows versions handle a multiple-processor request.[one developer apparently speculated that this was a way of making games run slower in older versions of windows]

J. Lang Usually the newer behavior is the saner behavior, so it's what we want to emulate.

Inconsistant Results or Differences between versions of Windows Service Packs

One wrote Feb 08 winedevel: If I have Msxml3 with SP7 installed all the msxml tests pass without issue, but if I have Msxml3 SP3 installed, some fail. Should I improve the tests so that they can handle both cases?

P. Vriens: The general rule is to have the tests pass on all versions of OS/applications/dll's. This has to be done, if possible, by checking behavior instead of retrieving version information.

Graphical Tests

A programmer noted that he [Jul 06]: was [trying] to find a problem [in Protel] where some lines are drawn a single pixel wide, when they should be several pixels wide.

H. Davies: a Wine regression test would be ideal here. In this case it's slightly tricky as you're testing a graphical component however it's still achievable. I'd suggest you add a test to gdi/tests/pen.c that creates a 1bpp dibsection, selects that into a dc, draws lines on it with a selection of pens in different graphics modes. After each line get the bits from the dibsection and compare them to what you've found out Windows gives you and then clear the dibsection before drawing with the next pen/graphics mode. If you need any help with this then please let me know. wine archive

Messages

A user reported a crash with an EN_MAXTEXT notification and a developer replied "From what I understand, the program crashes when it receives EN_MAXTEXT notification. You should figure out why the textbox is too small to contain the text. Or maybe this notification shouldn't be sent at all, although MSDN says it should be, but maybe there are some exceptions. Probably you'll need to write some test's to figure out what exactly goes wrong."

V. Margolen: You can look at dlls/user/tests/win.c for examples on how to write your own WndProc, attach it to window and test for messages received. Wine Archive Link

warn+heap

M. Lankhorst [Oct 07 wine devel]: run the tests with WINEDEBUG=warn+heap, this will do 2 things:

  1. . Allocated memory will be filled with 0x55's.
  2. . Freed memory will be filled with 0xaa's.

If something dereferences a structure it will use those magic values and then crash with values close to 0x55555555 or 0xaaaaaaaa which makes debugging easier. I just found a crash in crypt32 cert test that way, which would be a good bug report as well. Those issues are normally very hard to track down otherwise.

D. Kegel Oct 07 wine devel: As noted in http://bugs.winehq.org/show_bug.cgi?id=9916 using warn+heap poisons free memory, and finds a few more bugs. People who run 'make test' should consider running WINEDEBUG=warn+heap make test instead. (I don't know whether we should make that the default, though. Anyone have an opinion?)

A. Julliard: I don't think we want to make it the default, it's enough to have people try it from time to time, this way we exercise both cases.

Handling Driver bugs

H. Verbeet: At that point the only real option is keeping a list of blacklisted video drivers (which some applications actually sort of do), but I'm not sure we really want to go there. I do think we should investigate native failures, and fix them if possible, but if tests fail because eg. vmware has a broken d3d implementation, I think the test is essentially doing what it should do.

A. Julliard: Yes, if the code is going to break in normal use too, then it's OK for the test to fail. It simply means the platform is useless both for regression testing and for normal apps that use the functionality. If we really have to add workarounds for broken drivers, this has to be done in the code, not in the tests.

Testing Internet Connection

+  hurl =InternetConnectA(hinet,"www.winehq.com",INTERNET_DEFAULT_HTTP_PORT,NULL,NULL,INTERNET_SERVICE_HTTP,0,0);
+  ok((hurl != 0x0),"InternetConnect Failed\n");

J. Lang: [Nov 05] The ok after InternetConnectA shouldn't be there. [...]This could fail for many reasons other than that InternetConnectA has a flaw (the test machine may be offline, winehq.com may be down, etc.) Therefore you must deal with the failure, but using ok() is not correct--a failure doesn't necessarily indicate a failure in the test. So you must something like the following: wine archive

if (hurl)
{
/* do tests that depend on hurl here */
InternetCloseHandle(hurl);


D. Kegel mar 09 Winehq Bugzilla entry 17525 in general, tests should implement their own server with win32 to connect to rather than connecting to a real server on the internet. There are examples of this already in ws2_32/tests/sock.c.

Dos Tests

A discussion about how to include dos tests arose in Jan 09: D. Kegel suggested nasm vs binary files. D. Timoskov said:The exception tests in ntdll is a nice example how to do that without hurting readability and maintainability.

A. Julliard: If there are really so many DOS test cases that it becomes a problem, then we can consider adding more infrastructure for it. I don't see that happening any time soon.

A. English: How about we merge in the wine16 tests while we're at it?

D.Kegel: That would be great, if someone wants to do it, but it's kind of a pain because it currently requires the windows version of the open watcom c compiler.


A Programmer noticed [Jun 06]: My opinion is that NtSetContextThread call is wrong; __wine_enter_vm86 would restore vm86 registers correctly. I think i know what is the problem [...] any hints would be appreciated.

D. Kegel: How about writing a conformance test that demonstrates the problem? That will help focus discussion and energy on it, and keep it from coming back.

However he notied: I would be happy to do it. Unfortunely, I cannot imagine how to implement such test: 1) DOS code needed - it would need binary or some sort of source to be compiled prior to test or some temporary file with a binary 2) method to catch failure inside of ntdll 3) maybe something else...

A. Turkin: I believe it is better to store binaries as arrays and use temporary files than drop many small .com binaries all over the code tree (for now I have 3 test programs about 30 bytes each). It is possible to comment/disassemble test cases instruction by instruction, if needed. Also, not everyone can use git [Which can handle binary files better than CVS], and I believe diff/patch would fail to produce patchable diffs with binary files in them. wine archive

Win16 tests

D. Kegel [Jul 07 wine devel]: As Andreas Mohr pointed out a while back, http://www.winehq.org/pipermail/wine-devel/2005-August/039602.html we really need a win16 test suite to make sure our win16 support doesn't rot. As far as I know, nobody's managed to get win16 tests into the tree, presumably because the build process or maybe stdio for win16 tests hasn't been figured out. So I've asked [...]J. Lai to try to merge all the win16 tests people have posted over the years together, and demonstrate how to build them using openwatcom.

D. Kegel [Jun 07 wine devel]: So, the original question was "How to test OpenFile16"? Here's a more complete answer. First, install OpenWatcom as previously described. Then create a text file blah.c containing something like

#include <win/win16.h>

int PASCAL WinMain( HINSTANCE this_inst, HINSTANCE prev_inst, LPSTR cmdline, int cmdshow )
{
   HFILE h;
   OFSTRUCT of;
   h = OpenFile("foobar.dat", &of, OF_CREATE); if (h == HFILE_ERROR)
           MessageBox(NULL, "Failed", "foobar", 0);
   else
           MessageBox(NULL, "OK", "foobar", 0);
}

Then compile it with OpenWatcom like this:

$ wine wcl -Zw blah.c

This seems to produce a simple 16 bit testcase that works. To verify OpenFile16 is being called, get a +file log:

WINEDEBUG=+file wine blah.exe 2>&1 | grep OpenFile 
trace:filepenFile16 "Z:[redacted]\\blah.exe" OF_READ OF_SHARE_DENY_WRITE
trace:filepenFile16 found "Z:\\home\\dank\\blah.exe"
trace:filepenFile16 (Z:\home\dank\blah.exe): OK, return = 0x58
trace:filepenFile16 "foobar.dat" OF_READ OF_SHARE_COMPAT OF_CREATE
trace:filepenFile foobar.dat OF_READ OF_SHARE_COMPAT OF_CREATE
trace:filepenFile foobar.dat 1000
trace:filepenFile (foobar.dat): OK, return = 0x78 trace:filepenFile16
(foobar.dat): OK, return = 0x78

The next step would probably be to figure out how to get Wine's test harness, or something like it, to work in 16 bits. [...] winetest.h might not compile in 16 bit mode, and our build system doesn't handle 16 bit binaries, and gcc/ld can't build 16 bit NE executables, and you can't mix 16 and 32 bit easily so we'd need a winetest16,[...] I haven't checked yet whether openwatcom's linux version can build 16 bit NE executables. Sure would be cool!

D. Riekenberg:I did some research on win16 a while ago and cross-compiled on linux with wcl (ow1.6) and a patched "wine/tests.h" a 16-Bit Test for GetWindowsDirectory/GetSystemDirectory. [The] Remaining Problem is stdio for Win16-Apps for the results. [ building 16 bit NE executables] Works! (I used wcl)

Win16 test and the Wine Tree

D. Kegel: Given that not many developers will bother installing openwatcom just to be able to compile the win16 tests, I suspect that might be enough of a test harness for now. Objections?

A. Julliard: I don't think there's any point in putting into the tree test cases what won't be built or executed. They will just bitrot. We already have a hard time keeping the normal test suite in a usable state.

D. Kegel: So if I want these in the tree, they should be built and run by default. [...]. Next question: is it acceptable to require OpenWatcom? (That's the only free compiler I know of that targets win16. Turbo C++ is also a free download, but it's not open source.)

A. Julliard didnt like using the windows version of OpenWatcom: I don't think we want to do it that way, we should use a Unix version of OpenWatcom, and detect it in configure.

D. Kegel:The OpenWatcom guys haven't gotten around to making Linux packages available from their main download site. It should happen eventually.

A. Julliard:[as for where they should go..] It depends how many tests there are, but if it's just a few that's probably the best solution, we wouldn't want to have to maintain a lot of makefile infrastructure just for a couple of tests. Somehow I doubt that we'll see a lot of development on win16 tests...[Dan agreed with a separate download for now]

Feb 08 wine devel, the question was raised: What's the status of getting the win16 tests into wine? 16 bit support broke a while back for many apps and is still broken.

D. Kegel: The sticking point is that most people don't have a 16 bit C compiler installed. The only good free one is Open Watcom. Unfortunately, you still can't just do 'sudo apt-get install openwatcom' on most distributions (see http://openwatcom.org/index.php/Installing_Open_Watcom_on_Linux ), and there were some hitches using the Linux version anyway, so win16test provides a shell script that installs the Windows version under Wine. (See http://code.google.com/p/win16test/source/browse/trunk/src/README ) And that's kind of heavyweight, so we can't just jam it into the Wine build system.

Potentially, though, the Wine build system could notice whether openwatcom was available, and if so, it could build the win16 tests. For now, they're kept out of tree in http://code.google.com/p/win16test/ and I encourage others to try them out and fix problems they find. - Dan

Win64

A. Julliard [Jan 09 wine devel] The 64-bit winetest.exe is now built on WineHQ along with the 32-bit one. You can get the latest build at: http://test.winehq.org/builds/winetest64-latest.exe Please give it a try if you have a 64-bit Windows box.


Wine Test Code Reviews

Message Box

* Implement handling for the ErrorDialog and use it to change media.

A. Julliard [Oct 06] This displays a message box when running the install.c test, which will prevent the tests from running unattended. wine archive

Failures

+    if (MachineName != NULL)
+    {
+        FIXME("remote support is not implemented"); 
+        goto cleanup;
+    }

D. Riekenberg [Jun 06]: A failure should always set an error. You should add a test for this in the testsuite. (winspool as example uses an empty servername the same way as NULL). For external Servers, ERROR_ACCESS_DENIED is a very common Result. wine archive

SetLastError

+    devlist = SetupDiCreateDeviceInfoListExW(NULL, NULL, NULL, NULL); 
+    ok(devlist && devlist != INVALID_HANDLE_VALUE, "SetupDiCreateDeviceInfoListExW 
     failed : %p %ld\n", devlist, GetLastError());

D. Riekenberg [Jun 06] - You missed the "SetLastError(0xdeadbeef);" before the tested function. - The first Part of your "ok()" - message ("SetupDiCreateDeviceInfoListExW failed :") has no additional infos. When a test fails, you will already get: "__FILE__:__LINE__: Test failed:" You know the dll, file and line of the failing "ok()" - statement. wine archive Example:

info.c:932: Test failed: returned 0 with 1801 (expected '!=0' or '0' with ERROR_INVALID_PARAMETER)


Keeping Testing on Windows in mind

+START_TEST(d3d9_test)
+{
+    init_d3d9();
+}

A. Julliard[Dec05]: Please don't add that sort of pseudo-tests just to share code with other tests. If you need init code put it directly in the tests that need it. A test C file should be self-contained as far as possible to make it easier to build on Windows.

Windows Locations can move
char Root_Dir3[]="C:\\Windows\\System\\"; ..
+    Root_Dir3[0] = windowsdir[0];

P. Vriens April 09 wine dev: This won't work on all systems as the windows directory can be different (and can be changed during installation). You could use GetSystemDirectory() instead of GetWindowsDirectory() to find out the drive letter and in the same go use the output from GetSystemDirectory() to fill Root_Dir3. As you're actually interested in an existing directory you could even use the output from GetWindowsDirectory() to fill Root_Dir3.

Windows Versions

 -  ok(DeleteFileA(newdir),"Couldn't delete the temporary file we just created\n"); 
 +  if (GetVersion() & 0x80000000) {

A. Julliard: [May 06]: You should never add version checks in tests, simply check for both behaviors.

The developer queried: Now, since some behavior is mutually exclusive, that implies the wine tests will never see 0 total failures?

A. Julliard: No, it means that the test should accept both cases, since both are valid behaviors of the Windows API. You can replace the ok() by a simple if(), and continue with testing the buffer in the success case. wine archive


====Dealing with strings====

+    { "a.", DnsNameDomain, DNS_ERROR_NON_RFC_NAME },

A. Juliard: [May 06]: It's better to avoid including Latin-1 characters in C source, especially since this means the test results will then depend on the locale.wine archive

-                if (!strcmp(expected[i], me.szModule)) found[i]++; 
+  /* on win2k the names can be upper case */ 
+  if (!strcasecmp(expected[i], me.szModule)) found[i]++;

D. Timoshkov: It's better to use win32 API lstrcmpi [comparing upper and lower case dll names] since strcasecmp might be not available on a target windows platform. wine archive


+    retval2=pStringTableAddString(table,String,0); 
+    ok(retval2!=-1,"Failed to add String to String Table\n");

M. McCormack [May 06]: You get a much better regression test by checking for an exact match of the output. Only retval2==-1 will make the above test fail. What about -2? Assuming retval2 should be zero, how about:

ok(retval2==0,"Failed to add String to String Table\n") wine archive


Odds bods

diff --git a/dlls/kernel/tests/Makefile.in b/dlls/kernel/tests/Makefile.in index bfeae14..5f6c12e 100644
--- a/dlls/kernel/tests/Makefile.in
+++ b/dlls/kernel/tests/Makefile.in
@@ -3,7 +3,7 @@ TOPOBJDIR = ../../..
 SRCDIR    = @srcdir@
 VPATH     = @srcdir@
 TESTDLL   = kernel32.dll
-IMPORTS   = kernel32
+IMPORTS   = kernel32 user32

A. Julliard [Aug 06]: It would be cleaner to avoid importing user32 in the kernel tests. What you probably should do is have the test create more instances of itself the way the process test does.wine archive

tchar
@@ -55,6 +55,9 @@
ok ( cond tmp, msg, GetCurrentThreadId(), err); \
    } while (0);
 
+/* Function pointers */
+static HMODULE hws2_32 = 0;
+static PCTSTR  (WINAPI  *pInetNtop)(int,LPVOID,LPTSTR,size_t) = 0;

A. Juliard: apr09 You can't use TCHAR types in tests.

Headers

F. Gouget: [Oct 06] Including each header manually surely is not normal practice on Windows, but the nice thing is that it lets us detect bugs in our header dependencies. If we just include windows.h we will completely miss those and thus reduces the value of the test. M. McCormack: These tests don't compile in MSVC 6 without this fix.

F. Gouget: Precisely. And this is very useful information. The right fix is to:

only way to make sure this problem won't come back later.

I have started to look at why some of these tests were not compiling on Windows and this is how I stumbled on the COM_NO_WINDOWS_H issue which has side-tracked me a bit. I know of issues in, at least, advpub.h and objbase.h. I hope to get back to them soon, but there's certainly room for someone else to fix them in the meantime. Also note that it's important to use the very latest SDK headers when compiling the Wine tests. A number of them won't compile with the stock VC6 headers. wine archive


D. Timoshkov [feb 09] The reason why we avoid windows.h inclusion in Wine is compilation time by avoiding inclusion of all-in-one headers windows.h does. It's always a good programming practice to include only those .h files that are really needed, inclusion of windows.h usually shows the ignorance and lack of knowledge of a programmer

Differences between versions of Windows

A Developer asked; I've just finished another test for NtQuerySystemInformation. In this test I test the structure SYSTEM_PROCESS_INFORMATION. The structure is however different for NT4. If I add an osversion (> 4) check I can check the structure for W2K/WinXP and W2K3 and skip this structure-check for NT4 (for now). Wine however defaults to win2k unless I put in the config:

[AppDefaults\\ntdll_test.exe.so\\Version] "Windows" = "winxp"

Is there another approach or will this test be accepted as is ?

A. Julliard: Ideally you should determine which structure to check by observing the function behavior, not by checking the Windows version. For instance if the structures have different sizes you can use the returned size to determine if you got the NT4 structure or the win2k one.

Unicode

P. Vriens [Jun 2007 wine devel]: There are several tests where we use W-calls where we could use A-calls to accomplish the same test goal. Most of the time this means some of these will fail on Win9x or WinME as they are not by default Unicode enabled. Do we need write our tests with A-calls ? For Wine this doesn't matter if we're forwarding the call to a W-call. I do think it could matter on Windows or does Windows also do the forwarding?

As an example: there are several CryptAcquireContextW calls in the cert tests for crypt32. These fail with an ERROR_CALL_NOT_IMPLEMENTED on win98 and following tests that rely on the return valuse also fail. Should these tests be re-written to use CryptAcquireContextA or should we skip to avoid the hassle. I just want some opinions. I know Alexandre already mentioned not being to worried about failing win98 tests but I like to see an all green (with some blue borders) tests.winehq.org page

A. Juliard: When the call is simply forwarded it's OK to use the A version, since that will test both. Under Windows it will probably be forwarded too, and if it isn't and there are behavior differences then we would need to have the test call both A and W versions anyway.

Differences between Microsoft Documentation and Windows behavour

A programmer started to write a conformance test which led to several questions.

  1. What action should we take if MS documentation and the implementation do not agree?
    1. leave out the test
    2. test to implementation

U. Bonnes: If a program relies on undocumented features or windows behaving another was as documented, try to extract that case and code a case for the test suite. Test that the test tests the right thing on as many windows versions as possible, at least XP or NT2K3. Win95 is not so high on our list. If the test is right, submit it for inclusion. If the undocumented feature is only found by poking around, without any program found to rely on it, only write a comment. wine archive

  1. If a function takes several parameters, do you test all permutations, or just the ones you feel are important?

Dimi O Paun: As much as possible, within reason. I guess this means "the ones you feel are important

U. Bonnes: If your implementation path for the different options is clearly distincted, permuations are not that nescessary. But if the code pathes interweave, test more. At test what you feel important and what applications rely on.wine archive



Rewrite Tests
A. Julliard on IRC: You need a very good reason, when you really want to replace the existing Tests and create different Files for the Testsuite. What is totaly wrong with the current Tests? Filesize is not important enought: See dlls/user/tests/msg.c [Author: This File is ~ 250kb]. Insert non-formatted text here

Using Grep to your advantage

D. Kegel: Here's a grep that's handy when looking at the output of 'make test':

egrep '__test|make.*ok|Backtrace'

This shows which .ok file failures and crashes occur, and also makes it easy to see which dll's they're in. <math>Insert formula here</math>

Documenting Windows API

K.Cook: I realize I should provide full documentation for the actual Windows API calls themselves. I documented everything BUT those.  :) What's the convention for the number after the API name? I've seen some with numbers, and some with just an "@" sign?

J.Huizer:I think it's an "@" for normal calls, and a number for an 'ordinal' call; (though don't really know what the difference is... ;))

M. McCormack:Describing the function is good, but IMO, test cases are a more accurate description :) You're probably safe using [crypt32.@], as crypt32 probably doesn't export functions via ordinal (number). You can check the crypt32.spec file to see. Wine Archive Link

MSDN and Copyright

+/* MSDN 672579: "CCS_NORESIZE: Prevents the control from using the [...]g." This test checks both creation and sizing dimensions. */

D. Timoshkov [Jun 07]: You can't quote MSDN in the Wine code, but you can explain the problem using your own wording.

Troubleshooting Test Cases

Slow

Aug 08 wine devel a developer asked: the dplayx tests make these tests run very slow (or there is just a lot tested?) [..] on W2K upwards.[..] the msi/install tests [..] only seems to be an issue on XP and Vista.

J. Hawkins: The install tests should only take forever if you have some form of logging enabled. All logging should be disabled.

HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\Installer

Debug:DWORD should be 0 or nonexistent Logging:REG_SZ should be empty or nonexistent There may be other policies somewhere else that cause logging to happen. The tests run fine for me in XP, and for others as well.

R. Dunn: My experience has been that the msi:automation tests are slower on Vista - I suspect that this is because of logging, but could not see an entry in the registry key you provided - even when UAC is disabled. IIRC, logging is enabled by default on Vista. When UAC is enabled, the tests will timeout because of the countless number of times you have to accept the installer: yes I *do* want to run this installer! ... but then when UAC is enabled, almost everything fails!

__WINESRC__ ifdef
A programmer commented [Jun 05] that his: next patches will change the struct (and related files) to use UNICODE_STRING for the process name,[... the] patches will also contain tests to prove the above. Can anybody (who's not tired of reading until now) tell me if we really need the __WINESRC__ in that structure, the Windows structure shown is wrong anyway

R. Shearman: We shouldn't need the __WINESRC__ ifdef. No code that is designed to be portable to Microsoft's headers should be using winternl.h. [checking your struct in a test by copying it to the test and renaming it to something non-existent] I think that is the only way to do it. Wine Archives


Namespace

P. Millar discussed if it would be useful to separate the different tests' namespace. P. Millar: This would allow identical filenames to appear in different DLL tests directories without confusion (c.f. "generated.c" currently).

The proposed namespace would be simply: <dllname> "/" <filename> ":" <lineno> Examples are:

    user/sysparams:186
    winspool/info:128

A. Julliard: The problem with this is that the result is not a valid path, which breaks tools that parse the output to let you jump to the error, like emacs compile mode.

P. Millar: In principle, one can reconstruct the path from the test-name by sticking "dlls" and "tests" in the right places. Presumably emacs (or whatever) can be taught these tricks, so one could display the tests as

What would people prefer?

F. Wagner: I'd prefer you don't break tools/winetest/dissect, which matches on the closing line. :) The change is trivial but essential for the winetest web summaries. Wine Archive Link

Troubleshooting Building Wine Tests

A programmer noted [Sept 07 wine devel] I've been trying to cross-compile the Wine tests, but I keep getting this error:

$ ./configure && make crosstest [SNIP]
i586-mingw32msvc-gcc advpack.cross.o files.cross.o install.cross.o testlist.cross.o -o advpack_crosstest.exe -lcabinet -luser32 -ladvapi32 -lkernel32  
/usr/lib/gcc/i586-mingw32msvc/3.4.5/../../../../i586-mingw32msvc/bin/ld: cannot find -lcabinet
collect2: ld returned 1 exit status
make[2]: *** [advpack_crosstest.exe] Error 1 make[2]: Leaving directory
[...]> [dlls/__crosstest__] Error 2

Am I missing anything? Thanks in advance.


J. Klehm: Sounds like you need to patch your mingw w32api (not surprising as the way it comes is way out of sync with wine).

These pages should help you: [1] http://www.winehq.org/site/docs/winedev-guide/cross-compiling-tests [2] http://www.winehq.org/site/docs/winedev-guide/testing-windows

You will actually need more patches than are included in Hans Leidekker's RPM (linked from [1]). You'll get different missing lib errors otherwise. I believe Stefan Liechter has sent the needed patches since Han's has updated his rpm. They are attached to this mail. If you just want a binary winetest to run on windows its much easier to grab paul millars: Linked to from [2].

I haven't tested this lately and the w32api may need more patches still in order to get it to compile the latest wine git tests. Paul millars latest winetest is usually a good indication of when the last time wine's tests were cross compilable.


A freeBSD developer asked if a wine installation was necessary. M. McCormack: No, it should work without Wine installed, and without a ~/.wine. Perhaps try running wine/tools/wineprefixcreate first? [He then gave an example of running the tests without his .wine folder]:

bash-3.00$ rm -rf ~/.wine
bash-3.00$ cd dlls/kernel
bash-3.00$ make test
make[1]: Entering directory `/home/mike/wine/dlls/kernel/tests'
../../../tools/runtest -q -P wine -M kernel32.dll -T ../../.. -p 
kernel32_test.exe.so alloc.c && touch alloc.ok
wine: creating configuration directory '/home/mike/.wine'...
wine: '/home/mike/.wine' created successfully.
../../../tools/runtest -q -P wine -M kernel32.dll -T ../../.. -p 
kernel32_test.exe.so atom.c && touch atom.ok
...


I Sysoev reported the solution: It seems the tests tried to use my old .wine for Wine-20030508. I have renamed it and now I can run [the] tests. wine archive

Test failures - some gotcha's

Nothing to be done for test

This was seen a few times in the list so we added it here. You apparently need to be in the right directory..:

[redacted]:/media/hdc6/home/rasz/source/wine/dlls/ws2_32$ make test
make: Nothing to be done for `test'.
[redacted]:/media/hdc6/home/rasz/source/wine/dlls/ws2_32$ make crosstest
make: Nothing to be done for `crosstest'.

D. Timoshkov [30 May 2007]: Try the commands above after 'cd tests'.


registry problems

$ make -C dlls/shell32/tests/ test
make: Entering directory `/mnt/antras/usr/src/wine-cvs/wine/dlls/shell32/tests'[...]
shlexec.c:275: Test failed: ShellExecute(null, "C:\windows\temp\\test_shortcut_shlexec.lnk", null, null) failed: rc=31 err=0
make: *** [shlexec.ok] Error 1

F. Gourget: I found why it was failing on my box: my registry was a bit out of date. In order for .lnk files to work you need some registry settings to register a COM thingy (Mike McCormack would have the details). So just rerunning your wine.inf should sort this out. Wine Archive

crash created as success during make test

It dies by cathing exeption when ir tries to read 0x00000000, though "make test" doesn't say this is an error.

S. Krasukas: [Sept 05] is this intentional behaviour of an in-tree testing framework? (to treat crashes during "make test" as a success)

A. Julliard: Not really, but it's pretty hard to fix since the debugger cannot set the Unix exit status of the parent process.Wine Archive


Insfficient Disk Space

winetest.Ayu6Za.s:251949: Warning: end of file not at end of a line; newline inserted
winetest.Ayu6Za.s:251987: Fatal error: cannot write to output file
winebuild: as failed with status 256

S. Krasuckas" Hehehe, this is an error I get frequently. I gues you've run out of disk space on this partition. Check this for example by running:

$ df -h .

I need about 50 MBs of free space to make 'make' inside programs/winetest happy. wine archive

Dual CPU

msg.c failure

   user32_test.exe.so msg.c && touch msg.ok..
   f050!  msg.c:5524: Test failed: ScrollWindowEx: the msg 0x000f was expected, but got msg 0x0021 instead msg.c:5524: Test failed:

U. Bonnes [July 05]: I learned on IRC that this test is very sensitive on any keyboard/mouse interaction. On a single processor machine I really could succeed by "holding my breath" while the test ran. Also the verbose mode (-v) for the tests gave better chances for success. However on the dual processor machine, the test _never_ succeeded. Only you running "yes > & /dev/null &" so one processor was active made the test succeed. Wine Archive


Building an exe with an export table

A programmer: Process Explorer tries to load a dll with LoadLibraryA. This fails and NULL is returned. PE is however still trying to do GetProcAddress on this (not loaded) dll. Windows returns ERROR_PROC_NOT_FOUND, [but with Wine] we get into an exception. [the discussion then turned to writing a test]

D. Timoshkov: Probably the test also should confirm whether it's a responsibility of dlls/ntdll/loader.c,LdrGetProcedureAddress() to return STATUS_ENTRYPOINT_NOT_FOUND which will be converted by GetProcAddress to ERROR_PROC_NOT_FOUND.


The programmer: Obviously a seperate test case for LdrGetProcedureAddress can be made but that doesn't prove your question.

D. Timoshkov: That's the only way to [dis]prove that IMO. [...], if GetProcAddress is just a wrapper around of LdrGetProcedureAddress?


A. Juliard: I suspect what is happening here is that GetProcAddress, like most other kernel functions, treats 0 as meaning 'main exe module', since passing other invalid module handles results in ERROR_MOD_NOT_FOUND as expected. Testing this will be a bit tricky since it requires building an exe with an export table, but that should be feasible with Mingw.

P. Vriens: [The trace shows] in this is case it's definitly not a 'main exe module'.

A. Juliard: sure, it's not a feature that normal apps would be using anyway, it just explains why the behavior of GetProcAddress is not identical to that of LdrGetProcedureAddress. BTW I managed to build a binary with mingw to confirm my theory, I'll put in a fix.

Further Reading

Debugging

A programmer said [Jun 05]: I'd like to print unicode-strings in some tests (for debugging purposes). The following however does not show nice output on wine (it does on windows -Our trace doesn't seem to be able to cope with %ls (or %S):

static const WCHAR dataW[] = {'S','o','m','e',' ','d','a','t','a',0};
UNICODE_STRING usdata = { sizeof(dataW)-sizeof(WCHAR), sizeof(dataW),(LPWSTR)dataW };
trace("Length is %d\n", usdata.Length);
trace("Maximum Length is %d\n", usdata.MaximumLength);
trace("The strings is %ls\n", usdata.Buffer);

A. Julliard: You can't use [wine_dbgstr_w to convert the string to ASCII] in tests since they need to build on Windows. You'll need to provide your own debugging routine inside the test.Wine Archives

Building Standalone Tests on Windows

D. Kegal wrote and posted a draft for updating Wine documentation on the Wine developer forum Wine Archives:

Using Microsoft C++ Toolkit

Build tests Standalone for Windows, using the Microsoft C++ Toolkit

Sometimes it's nice to be able to build a new unit test on Windows without Wine, and without buying Microsoft Visual C++. Here's the simplest way to do that on a Windows system:

  1. Download and install the free-as-in-beer Microsoft C++ Toolkit from http://msdn.microsoft.com/visualc/vctoolkit2003 and
  2. the Microsoft Platform SDK from http://www.microsoft.com/msdownload/platformsdk/sdkupdate (using IE5 or later!).
  3. Make a directory 'wine' underneath your work directory, and
  4. copy the file wine/test.h from the Wine source tree. (You can download the latest revision at http://cvs.winehq.org/cvsweb/wine/include/wine/test.h ).
  5. Copy some existing test from the Wine source tree, or create your test program (say, mytest.c) using Notepad.
  6. Following the usual Wine test style, being sure to begin it with #include <wine/test.h>
  7. Finally, in a command prompt window,compile the test with the command
  cl -I. -DSTANDALONE -D_X86_ mytest.c

Once that's working, try running the program under Wine without recompiling it. See? No Wine source required at all, save for that one header, wine/test.h.

(If you want to use the Microsoft C++ Toolkit under Wine, install it under Windows, then copy it to your fake C drive; it'll work fine there. See http://kegel.com/wine/cl-howto.html for some tips on making it easy to use from the Linux commandline.)

This was added to the documents 29/5/2005.

D. Kegel [May 25 2007 Wine Devel Mailing List] : This is more for my own notekeeping, but maybe it'll be of interest to somebody else.)

Sometimes it's nice to build a test with visual C++ on Windows without any makefiles or projects. You just have to get a command prompt that has run vcvars32.bat, e.g. by doing Start / All Programs / Visual C++ 2005 Express Edition / Visual Studio Tools / VS 2005 Command Prompt

and then do e.g.

cd wine-git\dlls\gdi32\tests
cl -DSTANDALONE -D_X86_ -I../../../include pen.c gdi32.lib

But for some reason today that didn't work for me; cl couldn't find gdi32.lib unless I gave an absolute path, which made no sense (since the same path was already in LIB). To try to track this down, I wrote a tiny batch file to reproduce everything:

set PSDK=C:\Program Files\Microsoft Platform SDK
set VS8=C:\Program Files\Microsoft Visual Studio 8
set LIB=%VS8%\VC\LIB;%VS8%\SDK\v2.0\lib;%PSDK%\Lib
set INCLUDE=%VS8%\VC\INCLUDE;%PSDK%\Include
set PATH=%VS8%\VC\BIN;%VS8%\Common7\IDE;%PATH%
cd [your path too..]\wine-git\dlls\gdi32\tests
cl -DSTANDALONE -D_X86_ -I../../../include pen.c gdi32.lib

Using CGWIN

Using cgwin

[Mar 04 wine devel] D. Kegel: So the full procedure, for the record, is:

  1. Download and run the installer from http://cygwin.com Use it to install gcc, git, make, flex, and bison.
  2. Open a cygwin window
  3. Grab a copy of the Wine sources using git as normal
  4. Configure wine, build widl, and use it to expand .idl files to .h files.
$ cd wine
$ ./configure
$ cd libs/port; make; cd -
$ cd libs/wpp; make; cd -
$ cd tools/widl; make; cd -
$ cd include; make; cd -
  1. cd wine/dlls/oleaut32/tests
  2. to avoid conflicts between the win32 headers from wine and from cygwin, just use the wine ones. The easiest way to nuke the cygwin ones is to do
$ mv /usr/include/w32api /usr/include/w32api.x
  1. Try compiling and linking your test. e.g.
$ cd wine/dlls/oleaut32/tests
$ gcc -DSTANDALONE -I../../../include olepicture.c -mno-cygwin -loleaut32 -lole32 -luuid

This fails with all sorts of errors about LOGFONTA. That's because the include lines in olepicture.c are in the wrong order! Edit olepicture.c and change the lines

#include <winuser.h>
#include <wingdi.h>

to read

  1. include <wingdi.h>
  2. include <winuser.h>
  1. Try compiling again. It should work now. To run your standalone test,
do $ ./a.exe

If you need more verbose output, do

$ WINETEST_DEBUG=1 ./a.out

That will enable trace() statements.


Further Reading

Troubleshooting Standalone Tests

Standalone tests have been changed [May 2005]. Make sure you are using a recent release of Wine. D. Kegel posted to the forum a documentation draft incorporating the changes and this draft has been repeated above.


A Programmer asked[Jun 05]: How do I run the wine test suite under Windows again? I want to do this by compiling from source.

J. Hawkins: Check out http://www.winehq.org/site/docs/winedev-guide/testing-windows

[...] make sure you have the Platform SDK installed. Everything should work for you then. If you have problems, write back to the Wine-devel list and we'll help you out. Wine Archive


Building Standalone Tests on Linux

D. Kegal [Oct 05] gave an example of building a winetest as a standalone test (msvcrt) on Linux using cl:

I have the Platform SDK installed, so I have MS's headers, e.g. /home/dank/.wine/drive_c/Program Files/MicrosoftSDK/include/WinNT.h

cl -? shows:
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.10.3077 for 80x86
Copyright (C) Microsoft Corporation 1984-2002. All rights reserved

and my INCLUDE environment variable is set to

C:\Program Files\msvc2003\include;C:\Program Files\MicrosoftSDK\include

To compile a test standalone, you have to avoid using wine's headers, like so:

    mkdir -p fauxinclude/wine
    ln -s ../../include/wine/test.h fauxinclude/wine
    cd dlls/msvcrt/tests
    cl -DSTANDALONE -D_X86_ -I../../../fauxinclude file.c

And that works fine.

D. Timoshkov: I compile Wine tests (on which I'm working on time from time) with cl.exe under Windows using Wine headers.

You need one more define: -Dinline=__inline

Dimi Paun felt this was non-obvious: Can't we detect MSVC (say in windef.h) and define inline and the other defines that we need? He then posted a patch which was tested as working fine. Wine Archive.

D. Kegal [Oct 05] summarised the two methods in a later post: Wine Archive

cd dlls\msvcrt\tests
cl -DSTANDALONE -D_X86_ -I../../../include file.c

or, if dimi's patch isn't in yet, you need two more defines:

cl -DSTANDALONE -D_X86_ -D__i386__ -Dinline=__inline -I../../../include file.c


Further Reading

MSVC

D. Kegel noted [Feb 2008 wine devel]: Building tests with MSVC 6 broken?

E. Pouech: Yes. MSVC 6 really starts to show its age.It has trouble with some math stuff (rather large constants) used in one of the tests. That can be worked around by applying a service pack. See: http://www.winehq.org/site/docs/winedev-guide/testing-windows But the show stopper is that it is incompatible with the latest SDKs.

The solution is to use Visual Studio 2005 which is a free download. There's still some hitches there:

What this means is that msvcmaker needs to be updated to generate the new XML build files directly, all with the right settings.

J. Klehm wrote about obtaining the latest SDK [Feb 08 wine devel]: Trickily enough the latest SDK is actually published on 2007-03-22:"Microsoft(R) Windows(R) Software Development Kit Update for Windows Vista�" http://www.microsoft.com/downloads/details.aspx?FamilyID=4377F86D-C913-4B5C-B87E-EF72E5B4E065&displaylang=en The only sane way I've found to find out what the heck is going on with the SDK is the blog page: http://blogs.msdn.com/windowssdk/ but lately they have this page now too (which hopefully will stay maintained? who can say..) that lists the latest sdk right now (in theory) http://msdn2.microsoft.com/en-us/windowsserver/bb980924.aspx

So maybe grabbing the latest sdk will help maybe it won't. Hopefully this will save everyone some time [...].


Problems were reported with MSVC in Jun 05. A fix was discussed but A. Julliard posted to the forum a fix for include/wine/test.h

D. Timoshkov reported that it worked. Wine archive

The Application Test Suite

A. English worked on a test suite for testing programs. May 09 : [report on] the test suite. I've got a few basic tests set up with notepad, and I'm currently working on setting up the framework, using AutoHotKey to both run all the tests and parse the logs for failures/passing todo's.

Crosstest - Building the .exe Tests on Linux

A user asked whats mingw32

A. English: It's a build environment to build windows binaries on Linux (possibly other OS's, I'm not sure).


A developer reported [wine devel Oct 07]: I've been consistently failing to 'make crosstest'... I've already tried my distro's repositories, Hans Leidekker's patches, all to no avail. He then gave an example: undefined reference to `_CreateWellKnownSid@16' undefined reference to `_GdiAlphaBlend@44'

H Leidekker: You need more patches Wine tests are a fast moving target which means we're nearly always a couple of patches away from being able to cross compile current git. An easy way out for this type of error is to use Wine's import lib instead of MinGW's. E.g. run 'make libadvapi32.a' in dlls/advapi32 and copy the file over MinGW's version. [this is old may be fixed by now]

[Jan 09 user posted]: Hi, if i run "make crosstest" after ./configure and make it generates exe-files i can run them in wine with expected results, but if i run them under windows xp(32-bit) i get "Program too big to fit in memory" as only response. Why?

running Ubuntu 8.10 64-Bit
wine GIT from today
mingw32 4.2.1

i do just ./configure and make without options. should i force 32-bit? how?

A. English:Try:

$ CC="gcc -m32" ./configure

Though it could still be a wine bug


A developer reported: I get a failure with make crosstest:

A. Juliard: [jan 10]You have to run make crosstest from the top level for the import libraries to be built.

Using that executable on Windows or Wine is not possible (Entry point not found for GetDiskInfoA) The wininet_test.exe on winehq has an ordinal Import of 102 in that place. How can we fix that?

A. Julliard: You have to use the Wine import libs, not the Mingw ones. From IRC:

Example for building an import-lib for mingw on demand:

make -C dlls/wininet/ libwininet.a

Winrash

Winerash is apparently the downloadable testing software used by the wine developers. C. Morgan [July 05]: I'm still looking for someone to take over winrash development. It's a sourceforge project written in c under visual studio 6. I can give developer access and help to anyone interested.


Troubleshooting winerash

[After a discussion about making winrash interact with the desktop] P. Walker: [July 05] I haven't investigated what winrash is, but _assuming_ that you need it as a service on NT, then the problem appears to be that it needs to display windows (or interact with the user somehow). The main problem will be that services run on their own desktop (unique to the service, not the logon, screensaver or iteractive desktops).

A message box can be displayed on the interactive desktop by using the MB_SERVICE_NOTIFICATION flag with MessageBox().

To actually display some more complex UI, you need to create the window on the correct desktop.Check MSDN (msdn.microsoft.com) for details, but the following is what is probably required:

  1. Call OpenInputDesktop().
  2. Call SetThreadDesktop() to allow the current thread to use that desktop.
  3. Create your windows etc.

Note the security risk msdn notes for SetThreadDesktop. This is because the service is probably running as a priviledged user (the LocalSystem account unless specified otherwise). It will need the elevated priviledges to attach to the desktop which will have an ACL that only allows the current logged in user). Hope that's enough to get someone working on this

Links

WineTest the automated Wine tests

The Wine test results are found at the url http://test.winehq.org/data/ Volunteers allow it to run on their computers running Windows and it reports back. D. Paun [wine devel Oct 07] There is a headless operation mode already [on Windows], it detects if you run it as a service, and skips appropriate tests.

D. Kegel Dec 08 wine devel:I updated http://wiki.winehq.org/Wine64 with easy instructions (thanks,Maarten!) for how to build wine's 64 bit support and try 64 bit pacman (which works!). And then I ran "make -k test". Surprisingly, I got 260 passing tests according to find . -name '*.ok' | wc and only 128 failed tests according to grep ok.*Error test.log | wc -l

Ladies and gentlemen, I believe it's time for a 64 bit winetest.exe to be added to the daily build, and for 64 bit test data to start showing up at http://test.winehq.org. That would let us tamp down errors in the test suite on 64 bits. (Yeah, I know, gcc with win64 ABI isn't released yet, and there are problems with exception handling still, but it still seems worth it...) - Dan


One developer asked if it could be run headless on wine [Oct 07 wine devel]: wine winetest.exe.so'? and another suggested: Maybe "-c" (console mode, no GUI)

P. Vriens: My idea is to visualize the skipped tests by using the blue border at the single test level to indicate tests are skipped (see attached picture).[wine devel jun 07]

D. Riekenberg notied: http://test.winehq.org/data/200707031000/ why is the Wine column always empty?

P. Vriens: Nobody sent the result from winetest.exe on wine for this version of winetest.exe. You can see actual results for wine here: *http://www.astro.gla.ac.uk/users/paulm/WRT/wrt.php

[The 'NT 4' column] is winetest.exe on Windows NT 4.0. You can also see entries for NT3: I have NT3.51 server installed in qemu here. [The Optimal testing would be] different Installations for every Windows-Version, that run winetest after every commit (turn). We are far away from the optimal testing situation.


A developer asked [Aug 07 wine devel] Is there any particular reason we don't use HeapAlloc/HeapReAlloc in winetest? The rest of our programs do, but winetest uses malloc and realloc.

D. Timoshkov: I recall that replacing malloc/realloc by HeapAlloc/HeapReAlloc has fixed user32/tests/msg.c failures under Win95 (that were caused by buggy msvcrt implementations), so if are still aming to run the tests on that platform we should get rid of malloc/realloc.


One developer wrote: it needs [...an official] Wiki page! [ed Note this wiki is unofficial and as soon as we find the right link we will add it here] He also suggested Winetest be used to run the test cases for wine ie make test.

Skipped Wine Tests

There has been a lot work on fixing the wine tests and now they have become much more beneficial for the Wine project. However after it was noted on the dev list that some wine tests were infrequently run by users who had unusual failures, while these failures will need eventual fixing, they were hiding breakages in wine. It was then suggested to hide these from the main page so breakages on the improved tests were more obvious.

P. Vriens How many runs should have a box have before it's considered for the 'stable' list? We have 2 months worth of data, which roughly means 40 runs.


D. Kegel Apr 09: Easy: if it misses >= N reports during the period being displayed, skip it. N=1 would be strict. We'd want to pick the lowest N that left enough data, I suspect.

[ed] Perhaps it will be implemented.

Winetest gpg Signature

A developer queried [wine devel Nov 2 2007]: the script also grabs the winetest.exe signature and attempts to verify it. But I don't know where to find the corresponding public key (651FD487) so I don't know if this really works.

F. Gouget: there a link on Paul's home page pointing to his public key and the required key is in there. Then it's just a matter of telling gpg to use it which I have now done and the signature checking code seems to work as intended. The script can now use a local cache containing the already downloaded files, and can be told to skip the tests if it finds that a file has not changed since the last time it has been used in a given vm+snapshot combination. In fact I've been able to verify that it works in real life since winetest has not been updated since last monday :-( Compilation problems? I've attached the latest version of the script

Interactive Wine Test

A user queried how to run Wine Test in interactive mode,[nov 2008 wine deve] to run the extra tests which need your attention:

The application (winetest.exe) cannot tell whether someone double-clicked on the exe in a file manager, whether it was started by some sort of a cron job, or whether it was started by autorun when you 'inserted a CD in the drive' (which is what I usually do).

[...] the extra tests (which are going to resize your screen to a gazillion different resolutions, play tons of tunes, and ask for your attention again and again).[...]

One possibility would be to add a checkbox to the winetest GUI that you would check if you wanted to run the extra tests. Actually it would be nice if it also let you specify the debug level to run the tests with. But right now the solution is to just do:

set WINETEST_INTERACTIVE=1
winetest.exe

Forum Comments

update P. Vriens Nov 08: Test.winehq.org is the new home of the cross compiled winetest.exe. Alexandre put in some magic to create the new winetest.exe on winehq. The new link:

The link can also be found at the bottom of the test.winehq.org index page. I already changed the wiki to reflect these changes.

This also means that Paul Millars winetest is no longer available. I'd like to thank Paul for all the hard work he did with keeping up that site and providing us for years with a cross compiled version of winetest. I also like to thank the people that provided the necessary MinGW patches in the past to get that winetest compiled again.


P. Millar [wine devel Nov 22 ] The DLLs and executables are built and packaged whenever winetest is rebuilt. They're available both from the old location:

 http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt/

or from a new one:

 http://quisquiliae.physics.gla.ac.uk/cross/results/

As with winetest.exe, they're only updated if the new binaries are different. The ZIP files are signed and there's are sym-links that always point to the most recent version. For example: http://quisquiliae.physics.gla.ac.uk/cross/results/wine-exes-latest.zip

Mono and Wine Test Suite

M. Jones has been working on getting Mono and Wine working together: I am working on getting mono to run happily under wine, thus opening wine (& linux) up to a much larger range of .NET applications. Unfortunately, mono-under-wine fails in rather cryptic ways at the moment, so I've found that running the mono regression test suite has given me a better idea of how to approach debugging mono/wine.

Double unfortunately, running the mono regression test suite on linux, windows & wine presents a large series of interesting problems. So I've developed a python driver for the tests that spits out some html highlighting interesting test results and unifies the testing method on all three platforms.

The code is available at [1] There is a git repository at [2] The scripts can generate static html pages for tests run, which are available at: [3] (you can download the entire set of gzipped pages from [4])

Many of the tests fail as a result of bug 8757, which causes mono to lock up while running a GC cycle. For now the scripts kill test cases after 5 minutes, but I've also included patches against wine to prevent the lockup (horrible hacks - disable modal message boxes & the debugbreak function). If you want to save time when running the tests, you can apply these patches in a git branch and test against that. The tests are designed to require little-to-no user interaction - after an initial linux & windows test run are performed (sample test data at [5]), an automated update script (nightly-update.sh) can be called to automatically update wine, run the tests and publish the results. Feedback is greatly appreciated - the tests are very temperamental, so it wouldn't surprise me at all if things break on other machines (I've tested pretty extensively and documented any non-automated actions necessary, but I want to make sure I didn't miss anything.)

M Jones

  1. http://mhjones.org/wine/wine-tests/wine-test-reader.tar.gz
  2. http://mhjones.org/wine/wine-tests/wine-test-reader.git
  3. http://mhjones.org/wine/wine-tests/htmltest/index.py.html
  4. http://mhjones.org/wine/wine-tests/htmltest.tar.gz
  5. http://mhjones.org/wine/wine-tests/test-runs-example.tar.bz2
Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox