Introducing Test Studio to build end-to-end tests and maintain your app quality

I’m excited to announce a new tool, which is currently rolling out, to allow makers create and run tests for their applications. As more and more apps are used in critical business processes, we need a way to ensure that changes to those apps do not make the app stop working, and we can do that with tests. Test Studio provides a way to add and manage test cases, organize them into test suites, and run them to validate that changes in the apps did not break the expected functionality described in the tests. Tests can also be run outside of the test studio in a web browser which allows you to build the automated tests into your app deployment processes.

 

 

But what is a test case? It’s simply a list of steps, which are expressions that can simulate interaction with the app and validate that the app is in an expected state after those interactions. Those expressions use the same functions that can be used in apps, with the addition of new functions that were added for testing purposes. For example, in your test you can fetch a record from a data source to validate whether the interaction with the app had the expected result. When a test finishes you also have the option to persist the test results, such as whether the test passed or failed, how long the test took and any additional information generated by the test, to a data table, SharePoint or via a flow for example.

To help with creating the test steps or expressions that are needed to interact with the app, you can open an existing app in “recording mode”. As you interact with the app the test studio will generate the expressions that are used to perform that action. This gives you a great way to start creating tests and get acquainted with the syntax for the new test functions.

This first version is focused on Canvas app tests and it currently supports the most popular controls used in apps. But this is just an initial step. We’ll be adding support for more controls soon, as well as new features to help you schedule your automated tests into your processes. This feature is still experimental, so we recommend you use it to write tests for non-production apps and provide us any feedback.

 

Testing your app

So, let’s dive into testing in Power Apps. But before we can test something, we need an app. For this post, I’ll work with one created from the “Start from your data” wizard, based on a SharePoint list.

 

Opening Test Studio

At this point, we can only add tests to apps that have been saved to Power Apps, so since this iteration of the app is ready, let’s save it. You will also need to enable the Formula-level-Error management feature for the app as some test errors require this feature to work properly. 

And now it’s time for some testing. Just like the Monitor tool, you can find the entry point to the Test Studio in the new Advanced tools tray:

If you select the Open tests option, it will open a new browser tab with the Test Studio, connected to the existing authoring session:

 

Creating a new test case

Let’s start creating a test right away. The easiest way to do this is to click the ‘Record’ button on the menu and to start interacting with the app. You will see your actions being recorded in the left pane. Select ‘Done’ when you have finished recording.

 

Editing your test case

Once you click ‘Done’, you’ll see that the expressions used to recreate the actions you took in your the app are added as steps for the selected test case. We can now edit the existing steps, including updating the expressions, adding / deleting steps, and reordering them. Let’s update the third step (where we set the title of the new entry) to

SetProperty('DataCardValue4'.Text, "Test entry")

Let’s also add a new step at the end of the test to validate that a new entry was created in the SharePoint list. We can use a new test expression called an Assert to test this. Add the new test step with the following description:

Validate that there is one test entry

and with the expression below. This will assert that we can find the entry with the given title in the data source:

Assert(!IsBlank(LookUp(TodoList, Title = "Test entry")), "The entry should have been inserted")

The test steps should now resemble the image below:

 

Running the test case

Now click ‘Publish’ on the ribbon of the Test Studio. For now, we can only run tests once the app has been published. The tests are part of the app and we need to publish the app for the steps to be available for the “test runner”. Once the app is published, you can now click the ‘Play’ button, and you’ll see the test being executed.

 

Detecting problems in the app

Now that we have a test case, let’s see how it can detect problems in the app. Going back to the original tab where the app is being edited, preview the app then delete the entry that we added (with the title ‘Test entry’). Update the OnSelect property of the IconNewItem1 in the first screen to comment out the navigation to the edit screen:

NewForm(EditForm1); // Navigate(EditScreen1, ScreenTransition.None)

Save and publish the app. You may need to wait about a minute or so before the publishing is done in the back end. Go back to the tab with the Test Studio, select the test case again, and click Play one more time:

As we can see, the test failed when it tried to interact with the text input control that holds the title of the new item. If someone had inadvertently introduced that “bug” in the app, running the test would have caught it right away. We can then fix the app to avoid giving the users a bad experience.

 

Running the tests outside the Test Studio

If you select the ellipsis for a test case or test suite in the tree view, or select the ‘Copy play link’ in the top-right corner of the screen, you can copy a URL that can be pasted in a browser window. This will open the app in the browser and run that selected test or test suite. This allows for several scenarios, such as having different people executing the test (and, as the test runs with their own credentials, catch issues that the app maker may not be able to),  or even provide a “walk-through” of the app by simulating the interaction of the app controls. It can also be used to schedule or automate the running of your tests from a release pipeline or using Power Automate UI flows.

 

Storing test results

When the test is run outside the Test Studio, you don’t get to see the green/red marks indicating whether the test passed or failed. To access the test outcome, you can use one of the behaviors that you can see from the test settings:

 

The formula that you enter for the OnTestCaseComplete action will be executed whenever a test is finished (successfully or not) in the app. In the formula you can, for example, store the result of the test into a data source, or start a flow that you have in your app with it.

Within the OnTestCaseComplete expression you have access to a new variable, scoped to that rule only, called ‘TestCaseResult’, which is an object with the following properties:

• StartTime: the moment the test started
• EndTime: the moment the test finished
• TestCaseId: the identifier for the test case; this is a value that uniquely identifies the case, so even if you have multiple cases with the same name, they can still be reason separately
• TestCaseName: the name that you can set for the test case in the Test Studio
• TestCaseDescription: the description for the test case in the Test Studio
• TestSuiteId: the identifier for the test suite that contains the test case
• TestSuiteName: the name that you can set for the test suite in the Test Studio
• TestSuiteDescription: the description for the test suite in the Test Studio
• Success: a value that tells you whether the test passed (true) or failed (false)
• TestFailureMessage: if the test failed, a message that indicates which step failed and why
• Traces: a table of values with additional information about the test (all calls to Assert and Trace generate entries in this table)

If we have a data source with corresponding properties, we could have this expression in the OnTestCaseComplete to save the results for the test:

Patch(
TestResults,
Defaults(TestResults),
{
StartTime: TestCaseResult.StartTime,
EndTime: TestCaseResult.EndTime,
Result: If(TestCaseResult.Success, "PASS", "FAIL"),
SuiteId: TestCaseResult.TestSuiteId,
SuiteName: TestCaseResult.TestSuiteName,
SuiteDesc: TestCaseResult.TestSuiteDescription,
TestId: TestCaseResult.TestCaseId,
TestName: TestCaseResult.TestCaseName,
TestDesc: TestCaseResult.TestCaseDescription,
FailReason: TestCaseResult.TestFailureMessage
})

Or if you want to send the entire object to be analyzed by a flow, you can use the JSON function to wrap the object and send to the flow:

MyTestResultsFlow.Run(JSON(TestCaseResult))

 

Organizing test cases in suites

One test case represents one certain scenario where the user is interacting with the app. An app may have many tests, and to better manage them it may make sense to group the test cases into related areas. Those are called test suites, a grouping of test cases that “belong together”. Just like you can play a single test case (either directly in the Test Studio or in a separate browser tab/window after copying the play link), you can play an entire test suite which is equivalent to playing all test cases of that suite in order. Once the first case of the suite is done (and the logic defined on the OnTestCaseComplete action is executed), the second test case is executed (and again the OnTestCaseComplete logic once it’s finished), then the third, and so on.

It’s important to reinforce the behavior explained in the previous paragraph. The first test in a suite will always start as if the app had been just opened. Once that test finishes, the next test will start from that point in the app. One advantage to this mode is that you can use it to break down a large test case into multiple ones within a single suite. However, if a test case expects the app to be in its initial state when it starts, it may fail if the previous test in its suite modified it. For example, leaving the app in a screen other than the first one. In many cases you can use the OnTestCaseComplete action to reset the app state if that’s what your suite needs.

Like when a test case finishes, when you’re running an entire suite, Power Apps will execute the formula defined in the OnTestCaseComplete behavior of the test object. That behavior has access to a new variable, called ‘TestSuiteResult’, which is an object with the following properties:

• StartTime: the moment the test suite started
• EndTime: the moment the test suite finished
• TestSuiteId: the identifier for the test suite
• TestSuiteName: the name that you can set for the test suite in the Test Studio
• TestSuiteDescription: the description for the test suite in the Test Studio
• TestsPassed: the number of tests that run successfully in the suite
• TestsFailed: the number of tests that had failures in the suite

 

Going deeper

Now that we’ve taken a quick tour through the Test Studio, let’s dive a little deeper into some of the new test functions that can be used within test cases. Here’s a summary of them (for more details, go to the function reference)

• SetProperty:  used to simulate interaction with most of the input controls.
• Assert: used to verify that a certain condition is true in the test and cause the test to fail if not. The first parameter to the function can be any logical (true/false) expression that you can write with the Power Apps language, and it can also take an optional second parameter that describes what the assertion is about.
• Trace: used to add additional information to the TestCaseResult record that can be accessed in an expression within the OnTestCaseComplete behavior.  Trace is a table with a row for each of the Assert and Trace functions that were called within the test. The records of the rows have two properties: Timestamp (date/time value) and Message (text value).
• Select: this function has been in the product for a while, but it could only be used to simulate the select action of a control within the same screen. In test cases this restriction has been lifted, so a step in a test case (which doesn’t belong to any screen) can use this function to simulate the selection of a control on any screen – although during the test execution if the control is not in the current screen the action will fail, as a user could not do that in the app. The function has also been enhanced so that we can also select controls inside galleries, with additional (optional) arguments to that function to determine the row of the gallery to be selected, and the control inside the gallery. Currently only 1 level of nesting is supported, galleries inside of other galleries are not for this new functionality.

 

Test runner details

If you took a look at the URL generated by the ‘Copy play link’ option in the Test Studio that we mentioned before, you’ll see that it’s very similar to the URL that you get when you play the app from the Power Apps Maker Portal (make.powerapps.com). It has an extra query string parameter that triggers the test runner logic. It means that the test cases are great at testing the app exactly because they’re running with the same app that is used by users. You can, for example, add additional query string parameters to that URL and access them via the Param function.

This also has another very important point to be aware of . Since the tests run as if a user was interacting with the app, it can also create / update / delete records from your data sources, so be careful in your tests not to take actions that could, for example, remove records that it shouldn’t from your data source. One option is to have a specific column in your data source that flags items as test-only, a prefix in the name / title of the records, or something that can be used to identify test objects which can be ignored when the app is being used by “regular” users.

 

We want to hear from you

I already mentioned it before but it’s worth saying it again, but this is the first version of this feature and we want to let you start exploring and using it now. We’d love to hear your feedback, so we can prioritize the next set of improvements, any features that are missing, or any of the current behaviors that may not be ideal.

What kinds of tests are you writing? Is there anything that is too hard to understand, and that we could make easier?  There is a ‘Provide feedback’ link in the Test Studio which leads to our community forum for sharing your ideas.  We really want to hear your feedback!
 

There is some work that we already know that we need to complete which we have already started such as:

• Full control coverage for recording and playback capabilities. This includes both native and those based on the Power Apps Component Framework (PCF), and controls within components
• Better support for automation, such as running tests in Azure DevOps on a daily / scheduled cadence. Stay tuned as we’ll have further posts on options to schedule and automatically run your tests soon.

Let us know what you think as you start building tests for your apps!