Windows Driver Testing Basics: Tools, Features, and Examples This article is aimed at helping you test drivers for Windows. Since there are many different types of drivers, we cover the specifics of each type and explain how the Windows device driver testing process differs. We also cover supplementary tools and – another quite important topic – driver signatures. Written by: Denys Rudov Senior Tester Driver Testing Team and Dmitriy Yurko Test Designer Driver Testing Team Contents: Definition and Types of Windows Drivers The Main Aspects of Windows Driver Testing How To Test Windows Drivers Utilities for Driver Testing and Analysis Error Localization Driver Testing Report Sample Definition and Types of Windows Drivers A driver is a software component that provides an interface for a physical or virtual device. A driver interprets high-level requests coming from user software and the operating system into low-level commands recognized by a device. Driver Location in Windows In Windows, drivers are stored as binary files with the .sys extension along with optional supplementary files with .inf and .cat extensions. .inf Files The .inf file is a driver installation file that describes the type of device that a driver is designed for, the location of driver files, and any dependencies. During driver installation, data is entered in the system registry from the .inf file. This registry serves as a database for a driver’s configuration. .cat Files A .cat catalogue file contains a list of cryptographic hash sums of all driver files. In Windows, installed drivers are usually stored in the %SystemRoot%\System32\drivers system catalogue, but they can also be stored in any other place. After installation, a driver is loaded in the system and is ready to work. Some driver types require a reboot after installation. Driver Types Drivers can be divided into user-mode and kernel-mode types depending on their mode of execution. User-Mode Drivers User-mode drivers provide an interface between user applications and other operating system components such as kernel-mode drivers. A printer driver is an example of a user-mode driver. Kernel-Mode Drivers Kernel-mode drivers are executed in the privileged kernel mode. Kernel-mode drivers are usually in the form of a chain formed during driver execution. The location of each driver in this chain is defined by its task. Queries from user applications to a device pass through several drivers. Each of these drivers processes and filters the query, which is called an IRP packet (I/O request packet) according to Windows driver terminology. If a driver is loaded in the wrong place in the chain, then the query will be processed incorrectly and, as a result, the system may crash. Figure 1 below represents a simplified model of such a query for disk-based inputoutput (I/O). A user app creates a query to read a file on the disk. This query passes through a chain of drivers, each of which processes incoming and outgoing packets. Simplified model of a driver query from a user application In this article, we’ll focus on kernel-mode drivers. Kernel-mode drivers can be divided into the following types: Plug and play device drivers. These drivers ensure access to physical plug and play (PnP) devices and manage device power. Non plug- and play drivers. These drivers enhance user application functionality and provide access to kernel-mode features that are unavailable via standard API calls. These drivers don’t work with physical devices. File system drivers. These drivers provide access to the file system. They transform high-level queries for reading/recording files to low-level commands for a disk driver (reading/recording a sector on a physical disk). There is a driver development model called the Windows Driver Model (WDM) as well as a Windows Driver Framework (WDF), which consists of the Kernel-Mode Driver Framework (KMDF) and User-Mode Driver Framework (UMDF). Both the WDM and WDF simplify the process of making driver code compatible across Windows versions. Within the WDM are the following driver types: Bus drivers. These drivers support a specific PCI, SCSI, USB, or other port, controlling the connection of new devices to the bus. Functional drivers. These drivers ensure the functionality of a specific device. They usually support reading/recording operations and device power management. Filter drivers. These drivers modify queries to a device. They can be situated either above or below a functional driver in the chain of drivers. The Relationship between a Driver and a Device Each kernel-mode driver works with a specific device, represented in Windows as a device object. This means that the final destination for an I/O query coming through a driver is always a physical or virtual device. This applies to both drivers of physical PnP devices as well as non-PnP software drivers. While testing drivers, it’s important to understand that more than one driver exists between a user application and a device. In the chain, each driver can influence the final result of the query to the device. The Main Aspects of Windows Driver Testing There are certain tests that are required for Windows driver testing in Linux, and Windows regardless of driver type. So before covering the nuances of testing different types of drivers, we’ll consider their common aspects. Operating Systems First, you always have to keep in mind that a particular driver can behave differently on different operating systems. Furthermore, you need to take different kernel versions into account because they can differ even within the same operating system. For instance, Windows 7 and Windows 7 sp1 have different kernels. Therefore, you must test as many systems as possible. It’s worth mentioning that Microsoft supports Windows versions starting from Windows 7/2008. You also have to take into account that the most popular Windows versions now are Windows 7 and 10. Updates It’s necessary to check critical situations for a driver such as shutdown, reboot, and reset. You should also keep a system’s security systems in mind: firewalls, data execution prevention (DEP), user account control (UAC), and antivirus software. Operating system updates can also influence driver functionality. Therefore, it’s crucial to perform testing with the latest updates. In addition, you also need to test driver updates. Hardware Dependency Besides software dependency, there’s also hardware dependency. That’s why you have to check how a driver works with various processor and kernel configurations with an enabled and disabled page file. While testing a driver, you have to enable a driver verifier, which will create an additional load for the driver. During a testing process, check the correctness of driver installation and uninstallation, system reset, and hibernation. How To Test Windows Drivers Testing a driver using a real machine isn’t always secure since incorrect operation can lead to serious consequences. That’s why you should test a driver in a virtual machine until it’s stable. File System Filter Drivers As the name suggests, system filter drivers work with file systems. Therefore, while testing such drivers, you should use file systems such as NTFS, FAT32, exFAT, and ReFS. To test a Windows driver correctly, you should take into account that various file managers can be used besides Explorer, such as FAR or Windows Total Commander. And don’t forget about complex file system changes in addition to simple operations such as copying, deleting, and renaming. Complex file system changes include: Mounting/unmounting new disks: o ISO images; o Network disks o Virtual hard disks o USB flash drives Making sector configuration changes (changing drive letter or name) Actions performed on a disk: o Formatting sectors o Shrinking sectors o Defragmenting sectors o Checking for errors o Compressing sectors o Deleting sectors o Making a disk dynamic o Converting a disk in GPT/MBR o Creating a new sector. You also have to check: Various hardware configurations (SSDs and HDDs with different capacities) Driver behavior when stopping and starting services or installing/uninstalling an app How a driver works with an encrypted disk using Windows tools Driver compatibility with antivirus software, as such software is also a filter driver. Virtual Storage Driver Before testing a Windows driver created for virtual storage, you should understand that your file system will be stable. For virtual storage drivers, you should check the following things: How the driver works with files and folders when o opening o creating o editing o saving o copying o removing o renaming o deleting How the driver works with a search in files and folders. How the driver works with files that have names containing o lots of characters o digits o special symbols o spaces o hieroglyphs o non-unicode symbols o cyrillic characters How the driver works with files of different formats: o text o images o archives o Microsoft Office files How the driver works with files with various attributes: o read-only o hidden o system o archive How the driver handles changing file permissions and using various NTFS functions: o compression o encryption Whether shortcuts (symlink and hard link) and hidden copies are correct. How the driver handles files of different sizes: o very small o many very small files How the driver works with folders that contain a large number of subfolders (more than five). How the driver handles conflicts, for instance copying a file with the name of a file that already exists in the destination or cancelling copying or deleting. How the driver handles saving a file downloaded from the internet or a shared network disk. Disk mounting/unmounting in both standard situations and edge cases. For instance, try unmounting while copying to storage, then check whether the disk has been successfully mounted after rebooting the system. The disk’s read/write speed. USB Device Driver With USB driver testing, you should try to cover as many USB devices as possible. You can start with the most popular, such as flash drives, printers, scanners, mice, keyboards, portable hard drives, smartphones, and card readers. But you should also test less popular devices such as Bluetooth devices, Ethernet devices, USB hubs, microphones and headsets, webcams, and CD-ROM drives. You should take various USB interfaces into account: USB 1.0, 2.0, 3.0, and 3.1. In addition, don’t forget about unplugging/plugging devices, disabling safe and unsafe devices, and deleting devices in the device manager. Furthermore, check the device driver installation and uninstallation. Utilities for Driver Testing and Analysis There are numerous tools for testing Windows drivers that allow you to monitor the status of the driver in the system, verify its functionality, and perform testing. Built-in Windows utilities are enough to get basic information regarding a driver’s status (e.g. whether it’s loaded in the system). Built-in Windows utilities: Msinfo32 Driverquery Sc Driver Verifier Sc Driver Verifier is a built-in utility that allows you to verify driver functionality. To deeply analyze test drivers, you’ll need additional tools available in the Windows Driver Kit (WDK). Built-in Windows Utilities Windows System Information (msinfo32) Msinfo32 allows you to get a list of all registered drivers in the system, the type of each driver, its current status (loaded/not loaded), and start mode (System/Manual). To call the System Information console, call the Run dialog using Win+R and launch msinfo32. On the left sidebar of the System Information console, choose the following tabs: Software Environments > System Drivers. This utility allows you to review and store information about registered drivers. It also lets you view a list of drivers from a remote computer if you have access to its Windows Management Instrumentation (WMI). This option is located in View > Remote Computer. Driverquery, a Command Line Utility Driverquery provides information similar to that found in msinfo32. It can be launched through cmd using the driverquery command: Additional parameters allow you to modify the output to the console: /V is the command for detailed output. It allows you to get driver status information similar to that shown by msinfo32. /SI provides information about signed drivers. /S system allows you to get the information about a driver on a remote system. The sc Command for Communicating with the Service Control Manager The sc, command allows you to review a driver’s status and launch or stop the driver. To see a list of drivers, run the following command: sc query type= driver Windows Driver Kit Utilities The Windows Driver Kit (WDK) provides a wide set of tools for driver testing. WDK is integrated with MS Visual Studio, but it can also be used as an independent set of utilities. The WDK contains a set of testing modules called Device Fundamentals Tests as well as other particular utilities that allow you to manage devices and drivers, monitor resource usage, and has utilities for verification, and so on. Device Fundamentals Test A Device Fundamentals Test set consists of the following tests: Concurrent Hardware and Operating System (CHAOS) test; Coverage test CPU stress test Driver installation test I/O test Penetration test Plug and play test; Reboot test Sleep test To perform testing, WDTF Simple I/O plugins must support your tested device. Follow this link to learn more about WDTF Simple I/O plugins. Device Fundamental Tests are organized in the form of dll libraries and are situated in the %ProgramFiles%\Windows Kits\10\Testing\Tests\Additional Test directory (in Windows 10). These tests can be launched by the TE.exe utility that’s part of the Text Authoring and Execution Framework (TAEF), and they have to be installed with the WDK. You can find TE.exe in the %ProgramFiles%\Windows Kits\10\Testing\Runtimes\TAEF directory. Here’s an example of how we might launch a test: TE.exe Devfund_Device_IO.dll /P:”DQ=DriverBinaryNames=testdriver.sys” At this stage, we launch a device I/O test with a test driver called testdriver.sys as a parameter. Device Fundamentals Tests and TAEF are both perfectly suitable for automated driver testing. Windows Device Console (devcon.exe) The Windows Device Console is a command line utility that gives information about plug and play devices and their system drivers and manage devices and filter drivers for specific device classes. Using devcon, you can install, remove, connect, disconnect, and configure devices. Devcon allows you to set a template while searching for a specific device. An asterisk (*) can replace one or several symbols in queries. Examples of commands: devcon.exe hwids * displays a list of names and IDs of all devices devcon.exe classes displays a list of all device classes devcon.exe driverfiles * displays a list of driver files for all system devices devcon.exe classfilter USBDevice upper displays filter drivers for the DiskDrive device class devcon.exe /r classfilter DiskDrive upper !OldFilter +NewFilter replaces a filter driver for the DiskDrive device class PoolMon (poolmon.exe) The Memory Pool Monitor displays information about the allocation of downloadable and non-downloadable core memory. This utility is used for detecting memory leaks while testing. The memory allocation statistics for a driver are sorted by tag rather than by driver name. This tag should be set in the driver code using the ExAllocatePoolWithTag and ExAllocatePoolWithQuotaTag procedures. If a tag is not set in the code, then the system sets a None tag. Therefore, localization of memory issues can be complicated. Windows Hardware Lab Kit The Windows Hardware Lab Kit (HLK) is a framework for testing Windows 10-based devices. For Windows device testing on Windows 7, Windows 8, and Windows 8.1, you should use Windows HLK’s predecessor, the Windows Hardware Certification Kit (HCK). While testing with Windows HLK, you should use an environment consisting of two components: an HLK test server (controller) and a test system (client). An HLK controller manages a set of tests, connects them with a test system, and defines an execution schedule. The controller allows you to manage testing on a set of client machines. In the client system, devices and drivers are configured for further testing and test scenarios are performed. The preparation and testing process can be completed with the following steps: 1. Install the HLK controller on a dedicated machine. 2. Install the agent on one or several test machines. 3. Create a set of test machines that connects one or several machines in a logical manner. 4. Create a controller-based project that defines the elements to be tested. 5. Choose a test target, such as external devices of a test machine or software components such as filter drivers. 6. Choose and launch tests. You can use playlists to perform a specific set of scenarios. 7. Review and analyze test results. The Windows HLK allows you to test many types of devices. You can learn more about Windows HLK by following this link. Driver Verifier The Driver Verifier is a built-in Windows utility created to verify kernel-mode drivers. The Driver Verifier allows you to detect driver bugs that can damage the operating system. The Driver Verifier is most effective with manual or automated testing using WDK tools. The Driver Verifier is stored as a binary Verifier.exe file in the %WinDir%\system32 directory. This utility can be launched in two modes: via command line and via the Driver Verifier Manager. To launch the command line version, run the Command Prompt as administrator and enter the verifier command with minimum one parameter, for instance help - verifier /?). To open the Driver Verifier Manager, run verifier without parameters. Let’s look at a driver verification procedure using the Driver Verifier Manager as an example: 1. Run the Driver Verifier Manager: Win + R > verifier 2. Choose a set of standard tests or create custom tests. The manager also can display and delete current settings as well as display information about verified drivers: 3.Choose one or several drivers to verify. 4.Reboot the computer. The driver will be tested according to the chosen settings until it is removed from the list of verified drivers. Driver Verifier Standard Settings Below, we’ll describe the standard settings of the Driver Verifier for Windows 10. The list of standard and supplementary settings may be different in different Windows versions. Here are the standard options for the Driver Verifier in Windows 10: Special Pool Force IRQ Checking Pool Tracking I/O Verification Deadlock Detection DMA Verification Security Checks Miscellaneous Checks DDI Compliance Checking Let’s look into each setting in more detail. Special Pool The Special Pool option allows the Driver Verifier to allocate memory for a driver in a special place that’s monitored for memory damage, in other words access to the released memory. Force IRQ Checking In Windows, a driver cannot access unloaded memory with high IRQL if the spin lock option is enabled. The Force IRQ Checking option detects such issues. Pool Tracking Pool Tracking monitors a driver’s memory allocation. The Driver Verifier checks that the allocated memory for a driver eventually gets released. This helps detect memory leaks. I/O Verification The I/O Verification option detects a driver’s incorrect use of input/output procedures. In Windows 7 and higher, this option also contains an Enhanced I/O Verification function that performs stress testing for the following elements: PnP IRPs, power IRPs, and WMI IRPs. With Deadlock Detection, the Driver Verifier monitors synchronization objects such as mutexes and spin locks used by a driver. This is how potential deadlocks can be detected. DMA Verification Using DMA Verification, the Driver Verifier monitors the use of Direct Memory Access procedures. The DMA allows devices to work directly with memory without the CPU. Security Checks The Driver Verifier detects security issues such as calls of kernel-mode procedures with user-mode memory addresses or incorrect parameters. Miscellaneous Checks With Miscellaneous Checks enabled, a driver is tested for potential errors that can lead to a driver or system crash – for example, if a driver releases memory that still contains working driver structures. DDI Compliance Checking A driver is checked for potential errors in the communication (Device Driver Interface) with a kernel interface of the operating system. To efficiently detect bugs within the Driver Verifier, you should follow the recommendations below: 1. Don’t verify several drivers at the same time except if that’s precisely your goal. 2. Enable memory dump collection for cases where the operating system crashes (BSOD). 3. If needed, enable debug mode and connect to the test system with a debugger using the network or COM/USB port. Digital Driver Signature In Windows XP, Windows Vista, and Windows 7, there is no strict requirement for a package signature in order to install a driver package. Therefore, you can easily install a driver without a signature. However, if a package isn’t signed, you’ll see this warning: For a driver to be recognized as coming from a trusted publisher, the driver package has to be signed with a Windows Hardware Quality Labs (WHQL) signature in Window XP. In Windows Vista and Windows 7, a driver package has to be signed with a Trusted Root CA certificate. In Windows 8, Windows 8.1, and Windows 10 a driver package signature is required, as you cannot install a driver package without it. It used to be required to have the certificate encrypted with an SHA-1 algorithm. Now, SHA-1 is outdated and SHA-2 algorithms are usually applied to certificates. You can learn more about SHA-2 algorithms by following this link. Driver .sys File Signature Before running a driver in kernel mode, Windows checks the digital signature of the driver’s binary .sys file. It’s worth noting that Windows XP and Windows Vista 32-bit don’t require a digital driver signature. Windows Vista 64-bit, Windows 7, Windows 8, and Windows 8.1 require a signature with a certificate containing the Microsoft Code Verification Root in its root or with another certificate trusted by the kernel. Windows 10 version 1607 and newer requires a driver to be signed with the Windows Hardware Developer Center Dashboard portal. To start a Windows driver test, you can temporarily disable the digital driver signature check. In Windows 10, you can do this the following way: 1. Hold the Shift button and choose the Restart option in the main Windows menu. 2. Select Troubleshoot -> Advanced Options -> Startup Settings -> Restart 3. In Startup Settings, push F7 to choose the Disable driver signature enforcement option. To test drivers in Windows with the Secure Boot option enabled, you must ensure that your drivers have valid signatures. Error Localization An existing bug in a driver can lead to a system crash. That’s why, besides defining specific steps, localizing a bug means understanding whether your driver has caused the BSOD or not. In order to determine that, you have to review the system memory dump. This is collected automatically after the BSOD and you can find it in the C:\Windows\Memory.dmp directory. To review the full kernel dump, you should tell the system to collect it. The kernel dump also contains necessary information regarding free memory on the disk. To get this information, you should open Advanced system settings > Startup and Recovery and click Settings. Check whether the Complete memory dump or Kernel memory dump option is enabled. In these settings, you can change the storage directory for the memory dump. Once you have the full dump, you should analyze it. This is where WinDbg will be useful. Before using this tool, you have to download the specific Microsoft symbols. You can read this guide on how to do it. Open the dump and run the !analyze –v command. Pay attention to the stack and you’ll see the reason for the BSOD. In some cases, you won’t be able to get the dump file from the system because it’s continually crashing. In this case, you should use the Windows advanced startup options. You can then run the system in several special modes. The simplest and most reliable one is afe mode. In safe mode, your options within the system will be limited. But the only thing you need to get is the dump file from the C:\Windows folder, and this mode allows you to get it. You can also try the Disable automatic restart on system failure option, which may help prevent continual restarts. In case of a system crash, you should check whether a driver verifier was involved. This information can be very helpful for developers when they try to reproduce the bug. Besides a system crash, you can also face other issues such as those related to functionality or performance. To localize functionality issues, you should take the following actions: Try to recreate the same issue in another environment (in a different operating system, without antivirus software, on another file system, using a real machine, etc.). You also can try other conditions, such as different types of files and different file sizes. If the issue is related to your USB device, check other devices of different types. If the network is a potential cause of an error with a network device driver, check various network settings (latency, bandwidth, enable or disable the firewall). When the issue appears solely for users with specific permissions, use different permissions (administrator or standard) to localize the issue. If the issue is related to performance, your actions will depend on the element whose performance you are trying to improve: If the issue is in the network device driver, emulate network latencies or try recreating this issue in a high-speed network. If the issue is related to file operations, then check it using a different number of files of different sizes. If the issue is related to the USB camera, check its performance with different applications. Driver Testing Report Sample Testing type Regression testing Driver DriverFsfilter.sys (file system filter driver) Version 1.0.0.1 Windows Versions Windows 10 x64, Windows 8.1 x64, Windows 7 x64 Tests performed Installation/Uninstallation Processing of read/write operations: o Sectors with the following file systems: NTFS, FAT32, exFAT, ReFS o Different file sizes (1Kb to 10 Gb). o On physical mounted and network disks A stress test with unsafe disk disconnection Compatibility with Windows 10 updates (from 1607 to 1703) Driver Verifier: standard settings plus the Randomized low resources simulation option Errors Periodic BSOD emerging with Randomized low resources simulation enabled in Driver Verifier Result With the Randomized low resources simulation option enabled in Driver Verifier, the driver periodically causes the BSOD in Windows 10 x64. In other situations, the driver is stable. Application Windows 10 x64 Full memory dump attached: MEMORY.dmp Conclusion In this article, we have described the main types of drivers and approaches and utilities for testing them. Windows kernel driver testing differs significantly from testing desktop apps. If a driver contains a bug, it usually influences the stability of the whole system and eventually leads to the BSOD. Detecting, localizing, and eliminating driver errors significantly decreases the risk of unstable system behavior for end-users.
