Thanks for coming!  Today it's our pleasure to tell you more about the Common Data Service SDK.  We've been talking about this in some select circles for the past few months and today we're here to share this info with all this with our broader community!  

The Common Data Service provides a lot of benefits to our customers (to learn more about the Common Data Service in general, please see this blog post from April 2017).  One of the most tangible benefits of the CDS is that it allows data from many disparate sources to be integrated into a single data model. This data, however, is only useful when you can build apps on it for your company. To provide customers with even more meaningful ways to build apps, we are releasing the Common Data Service SDK. This SDK allows you to create, read, update, delete (CRUD) and even query your business data residing in the Common Data Service.  

Alright.  With that out of the way, I'd like to introduce Peter Villadsen who will take you on a quick tour of the SDK in the rest of this blog post.  And for reference, you can learn all the details about the SDK here.

 

The Common Data Service SDK 

Hello!  I'm Peter Villadsen and I am the Program Manager owner of the CDS SDK!  I'm extremely excited to talk with you today about our SDK.  But before we get into the details, let's set some context.

Underlying Architecture

In order to understand the role of the SDK with CDS, it is useful to first understand a little of the underlying architecture. All work accessing the data is done on the server tier – the client tier where the SDK code is running merely instructs the server what to do, and transports data to and from the server. The client and the server tiers communicate through JSON documents that describe the operations required. The server tier does not really care where these documents come from, so you can get to your business data in a variety of ways.

In principle you could build perfectly viable applications by building the JSON documents with string operations and using standard HTTP protocols to transmit them over the wire to the Common Data System endpoints. While this may be viable on severely constrained client devices, you will probably prefer using an API at a higher abstraction level, offering a strong type system to help you at design time rather than running the risk of failing at runtime. The role of the SDK is simply to generate the JSON documents representing the operations that the user specifies from the code the user writes. Therefore, the SDK is a relatively thin layer, and in principle an SDK can be written in any language running on any platform that supports an HTTP stack. Microsoft ships SDKs for managed languages, like C# and Visual Basic, and for JavaScript.  It is useful to think of these SDKs as domain-specific languages (DSLs) implemented in their host languages. That is, sub languages useful for developing business applications. In the rest of this blog, we will be using the managed version of the SDK, and we will be presenting examples in C#.

The DSLs that we are providing are not tied to any particular data storage paradigm. While the relational data model has served well for around half a century (ever since Codd released his seminal work in 1969), there are now ways to organize data that are not based on rows and columns. There are some scenarios, like representing social networks or advanced bill of materials, that may benefit from representing data as graphs containing nodes and edges and others where the more natural representation is as documents. The SDK will leverage the benefits provided by those data paradigms naturally and effectively as well. In this blog we will be dealing only with the relational model.

Examples

The functionality we're shipping with the SDK is the result of many years of experience building extensive development solutions for complex business domains such as Enterprise Resource Planning (ERP). We have designed the SDK to make it as easy as possible to implement effective, fast systems, while at the same time discouraging ineffective ways of doing things. Also, security is vital to any data usage in the enterprise, and security is baked into the Common Data Service by default. The security is not managed by the SDKs that we are describing here. Instead, it is set up centrally the administrative portal, and then applies to usage of the data, through any apps including apps that use the SDK.

In the terminology of the SDK, tables are called entitysets, as opposed to entities. Entities are in turn the records in the entitysets. The Common Data System ships with a large number of predefined entitysets out of the box.  You can learn more about the common data model and entities here. Should you still need to define your own, rather than use or extend one of the existing ones, you can use powerapps.com to define entities and the relationships between them. 

Now that we've talked through a bit of the underlying architecture, let's look at how you can use the SDK to interact with data in the Common Data Service.

Inserting data

Let us look at some code that inserts product categories into the database.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace SdkConsoleApp
{
    // Include the namespaces for readability
    using Microsoft.CommonDataService;
    using Microsoft.CommonDataService.CommonEntitySets;
    using Microsoft.CommonDataService.Configuration;
    using Microsoft.CommonDataService.ServiceClient.Security;

    using System.Threading.Tasks; // Needed for async

    class Program
    {
        private static async Task InsertAsync(Client client)
        {
            // Insert Surface and Phone product lines
            var surfaceCategory = new ProductCategory()
            {
                Name = "Surface",
                Description = "Surface product line"
            };
            var phoneCategory = new ProductCategory()
            {
                Name = "Phone",
                Description = "Phone product line"
            };

            var executor = client.CreateRelationalBatchExecuter(
                    RelationalBatchExecutionMode.Transactional);

            executor
                .Insert(surfaceCategory)
                .Insert(phoneCategory);

            await executor.ExecuteAsync();
        }

        static void Main(string[] args)
        {
            using (var client = ConnectionSettings.Instance.CreateClient().Result)
            {
                Task.Run(async () => await InsertAsync(client));
            }
            System.Console.ReadLine();
        }
    }
}

Typically you access the SDK through assemblies that are managed via NuGet. This makes it easy to keep up to date with the newest versions, since Visual Studio will download the newest versions and their dependencies from the NuGet repository automatically.

Because this example is a console application, execution will start in the static Main mathod. The first thing we need to do to talk to the data is to authenticate the user against Azure Active Directory. In this case we do that through the call to CreateClient, which will access information held in a config file called App.config. This file contains information about the Application Id, which is the Id that Azure assigned for your application, and the environmentId that defines which database you wish to access. Once that is all done, you will have a valid client instance to work with.

The client instance is the gateway for all data operations. The Client type implements IDisposable, so it is good practice to use it in a using statement, so that it will be disposed of as quickly as possible. In this case we pass the client instance to the InsertAsync method that will actually do the work. There are a few things to note here before we go into the details of the InsertAsync method. First, the method naming: The method has an Async suffix and that is a convention that you should always adhere to when a method is asynchronous. As you will see, almost all the methods provided by the SDK are available as asynchronous methods, which is crucial to using resources properly. If you need a brush up on your async skills you can refer to Asynchronous Programming with async and await (C#). In our case the InsertAsync method does not return a meaningful value, so its return type is simply Task. When you want to return values, you return a Task with the required resulting type as a generic type parameter.

The first thing we do in our example above is to create instances of the product categories that we wish to insert into our ProductCategory entityset. You will notice that the types representing entitysets are merely C# classes without much fanfare, what is called Plain Old C# Objects or simply POCOs in nerd argot. All the out-of-box entities are available to use in this way, and you can define the POCOs yourself if you have to, as long as they match up with the definition in the maker portal.

Once the entities (i.e. the records, or POCOs) are created they are ready to be inserted into the database. As you know, this sort of operation is usually done inside the context of a transaction; however, in the SDK the concept of transactions does not exist! What happens instead is that you add all the entities to an executor (these is what happens in the insert calls). However, these calls only register the entities for insertion later, namely when the ExecuteAsync call is made. At this point the entities are serialized on the client side, then sent over the wire to the server. The server layer will deserialize the entities, start a transaction, insert the records, and commit the transaction. As you see, the burden of managing the transaction falls on the system, not on you. This is one of the lessons we took away from the decades of experience from writing code for the Dynamics 365 for Operations platform, where managing transaction was always a source of problems.

Querying data

Above we introduced a lot of concepts that will help us understand how to query data residing in the relational database.

static async Task SimpleSelectAsync(Client client)
{
    var queryBuilder = client.GetRelationalEntitySet<ProductCategory>()
        .CreateQueryBuilder();

    var query = queryBuilder
        .Where(pc => pc.Name == "Electronics")
        .OrderByAscending(pc => new object[] { pc.CategoryId })
        .Project(pc => pc.SelectField(f => f.CategoryId)
            .SelectField(f => f.Name)
            .SelectField(f => f.Description));

    // Execute the query:
    OperationResult<IReadOnlyList<ProductCategory>> queryResult = null;
    var executor = client.CreateRelationalBatchExecuter(
        RelationalBatchExecutionMode.Transactional)
        .Query(query, out queryResult);

    await executor.ExecuteAsync();

    foreach (var pc in queryResult.Result)
    {
        Console.WriteLine(pc.Name);
    }
}

The code above queries the ProductCategory entityset for entities satisfying certain requirements. First, as before, the client is used to provide the functionality to access the data. In this example the client is used to create a Query builder over the entityset provided as the generic type parameter. This query is then enhanced by adding a where clause,talking about which entities we are looking for. In this case, we are looking for product categories that have the name "Electronics". It is important to understand that the where expression is expressed as a lambda function from the entityset type onto boolean. At runtime the compiler will create a tree for this function, and that tree is materialized in the documemnt that describes the query. In fact, the function provided is never actually run, it is only used to provide the definition of the where clause. Using lambda functions has other advantages: The compiler can diagnose problems cause by type violations, and capable editors can provide IntelliSense to aid the development experience.

This technique is reused in the clause that specifies the order in which the records are required. Here the lambda maps a ProductCategory onto an array of objects. This is because there can be any number of fields to sort by, obviously, but again, no array is ever created! We then want to specify what fields we want to project, i.e. the fields we want populated from the database. Note, there is no option for specifying "all fields", since misuse caused performance issues in other systems. You have to specify each field individually. Here the calls to SelectField are chained together to specify the list of three fields.

Now the query has been specified, but it has not been run. For this we need an executer, just like before. As you can see, we have specified the resulting type of the query explicitly, as a readonly list of ProductCategory entities. Sometimes it becomes cumbersome to describe the resulting type. In those cases, it it useful to use the CreateqQueryResultType function, that just returns null, but typed correctly as defined by the query.

            var queryResult = query.CreateQueryResultType();

OK. What's next?

We hope that the information in this blog has been useful in helping you to understand some of the initial capabilities of the Common Data Service SDK.  If you are interested in getting early access to the SDK by participating in the preview program, please contact us at cdspreviewprogs_at_microsoft.com.  You can learn a lot more about the SDK and keep up to date with the latest developments by visiting our documentation site here.

Thanks!

Peter