Introducing Test Engine: An open platform for automated testing of canvas apps

Testing is an important part of the software development life cycle. Adequate functional testing of applications helps to ensure that business processes stay unblocked, helps to reduce support costs, and helps to build trust in your applications. And as apps grow and become more complex, the ability to ensure that changes do not break or alter the app experience becomes more and more critical. Our mission is to empower all makers of Power Apps to easily create robust tests that can be seamlessly integrated into your organization’s application lifecycle management (ALM) practices.

Introducing Test Engine

We’re excited to announce the initial preview of Test Engine, an evolution of our Power Apps testing tools. Test Engine builds upon the key use cases of Test Studio, but takes it in a new, powerful direction through open source collaboration and use of the Playwright browser testing platform. The goals of Test Engine are to provide customers with a robust testing platform for all types of Power Apps, and to make it super easy to integrate automated testing into your app development processes.

Today’s announcement is the first step towards those goals. The project is provided as an Open Source project on GitHub that, when built, will create a local executable that you can use to execute tests.

This initial release supports the ability to author tests for Power Apps canvas applications. We plan to continue to iterate on this project, adding support for all types of Power Apps, as well as enhanced tooling to facilitate integration into your CI/CD systems like GitHub and Azure DevOps. For now, we welcome you to exercise this tool and give us feedback as we continue to build out the platform.

Power Fx test authoring

Power Fx is the low-code language for Power Platform, and that includes Test Engine. There is no need to write any C#, JavaScript, or any complex code to define your tests. Tests are defined in easy-to-read YAML files using the familiar Power Fx language. Using a few Power Fx functions that you already know from Test Studio like Select, Assert, SetProperty, and Index, you can define the sequence of steps your test should run. Here is an example of a Test Engine test case:

Connector mocking

Test Engine allows you to define mock responses for connectors.  This technique lets makers continue testing of Power Apps while isolating it from the remote APIs that the app connects to.  This can be useful your apps hit endpoints that have side effects, like inserting rows into Dataverse tables.

Screenshot function and video recording

Test Engine introduces the new Power Fx function Screenshot() that allows you to take a screenshot of the app state at any point during the test run.  For example, you may want to take a screenshot at the beginning and the end of your test suite to capture what the end user sees.  In this example, a screenshot is captured at the beginning of each test case, and upon completion of the test suite:

Test Engine can also automatically record a video of the entire test run.  Simply set the recordVideo setting to true.  You can use the captured video to observe exactly what the end user is seeing during the test run.  This can be very useful when diagnosing and investigating failed tests.

Only update tests when you update your app

We have heard from customers that using traditional browser automation tools to test Power Apps apps is difficult to get right and frequently breaks, requiring lots of updates to tests when the maker hasn’t made any changes to the app.  This is because traditional testing methods require the test author to interact with the browser document object model (DOM) of the app as its UI is rendered in the web player.  With Test Engine, all of this is handled by the platform.  Your tests are written using Power Fx and referencing the names of the controls that you define at design-time.  You will only need to update your test plans if you make changes to the app itself.

Coming Soon: Reuse tests recorded in Test Studio with Test Engine

If you are a user of Test Studio and already have tests recorded using that tool, we are providing a way that you can reuse them directly in Test Engine.  The ability to download test suites from Test Studio will start rolling out to customer tenants towards the end of October 2022.  The downloaded test suite can be used in Test Engine without any modification to the test plan file.  Watch the Power Apps Studio release notes for information about when the rollout starts. 

Next steps

Visit Test Engine on GitHub and try it out. You can build the solution with a couple commands.  Once you’ve done that, the Test Engine repo has a library of sample test plans and solutions you can use to get started.  To use the samples, import the provided solutions into your tenant and run the corresponding test plan.  Since it’s open source, we welcome contributions to both code and documentation.  Let us know about any bugs, feedback, and feature requests using the Issues list within the GitHub project.

Happy testing!