Azure.ResourceManager.HDInsight.Containers 1.0.0-beta.2

Microsoft Azure HDInsight Containers management client library for .NET

Microsoft Azure HDInsgiht Containers(HDInsight On AKS) simplifies deploying hdinsight cluster based the AKS.

This library supports managing Microsoft Azure HDInsgiht Containers resources.

This library follows the new Azure SDK guidelines, and provides many core capabilities:

- Support MSAL.NET, Azure.Identity is out of box for supporting MSAL.NET.
- Support [OpenTelemetry](https://opentelemetry.io/) for distributed tracing.
- HTTP pipeline with custom policies.
- Better error-handling.
- Support uniform telemetry across all languages.

Getting started

Install the package (Since now we are in private preview status, the bellow method doesn't work please send email to Askhilo@microsoft.com to install the nuget from our private nuget feed)

Install the Azure HDInsight On AKS management library for .NET with NuGet:

dotnet add package Azure.ResourceManager.HDInsight.Containers --prerelease

Prerequisites

• You must have an Microsoft Azure subscription.

Authenticate the Client

To create an authenticated client and start interacting with Microsoft Azure resources, see the quickstart guide here.

Key concepts

Key concepts of the Microsoft Azure SDK for .NET can be found here.

Documentation

Documentation is available to help you learn how to use this package:

• Quickstart.
• API References.
• Authentication.

Examples

Namespaces for this example:

using System;
using System.Linq;
using Azure.ResourceManager;
using Azure.Identity;
using Azure.ResourceManager.Resources;
using Azure.Core;
using Azure.ResourceManager.HDInsight.Containers;
using Azure.ResourceManager.HDInsight.Containers.Models;

When you first create your ARM client, choose the subscription you're going to work in. You can use the GetDefaultSubscription/GetDefaultSubscriptionAsync methods to return the default subscription configured for your user:

ArmClient armClient = new ArmClient(new DefaultAzureCredential());
SubscriptionResource subscription = armClient.GetDefaultSubscription();

Create Cluster Pool

// define the prerequisites information: subscription, resource group and location where you want to create the resource
string subscriptionResourceId = "/subscriptions/{subscription id}"; // your subscription resource id like /subscriptions/{subscription id}
string resourceGroupName = "{your resource group name}"; // your resource group name
AzureLocation location = AzureLocation.EastUS; // your location

SubscriptionResource subscription = armClient.GetSubscriptionResource(new ResourceIdentifier(resourceId: subscriptionResourceId));
ResourceGroupResource resourceGroupResource = subscription.GetResourceGroup(resourceGroupName);
HDInsightClusterPoolCollection clusterPoolCollection = resourceGroupResource.GetHDInsightClusterPools();

// create the cluster pool
string clusterPoolName = "{your cluster pool name}";
string clusterPoolVmSize = "Standard_E4s_v3"; // the vmsize

// get the available cluster pool version
var availableClusterPoolVersion = subscription.GetAvailableClusterPoolVersionsByLocation(location).FirstOrDefault();

// initialize the ClusterPoolData instance
HDInsightClusterPoolData clusterPoolData = new HDInsightClusterPoolData(location)
{
    ComputeProfile = new ClusterPoolComputeProfile(clusterPoolVmSize),
    ClusterPoolVersion = availableClusterPoolVersion?.ClusterPoolVersionValue,
};

var clusterPoolResult = clusterPoolCollection.CreateOrUpdate(Azure.WaitUntil.Completed, clusterPoolName, clusterPoolData);

Create Simple Trino Cluster

// define the prerequisites information: subscription, resource group and location where you want to create the resource
string subscriptionResourceId = "/subscriptions/{subscription id}"; // your subscription resource id like /subscriptions/{subscription id}
string resourceGroupName = "{your resource group}"; // your resource group name
AzureLocation location = AzureLocation.EastUS; // your location

SubscriptionResource subscription = armClient.GetSubscriptionResource(new ResourceIdentifier(resourceId: subscriptionResourceId));
ResourceGroupResource resourceGroupResource = subscription.GetResourceGroup(resourceGroupName);
HDInsightClusterPoolCollection clusterPoolCollection = resourceGroupResource.GetHDInsightClusterPools();

// create the cluster
string clusterPoolName = "{your cluster pool name}";
string clusterName = "{your cluster name}";
string clusterType = "Trino"; // your cluster type

// get the available cluster version
var availableClusterVersion = subscription.GetAvailableClusterVersionsByLocation(location).Where(version => version.ClusterType.Equals(clusterType, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();

// set the identity profile
string msiResourceId = "{your user msi resource id}";
string msiClientId = "{your user msi client id}";
string msiObjectId = "{your user msi object id}";
var identityProfile = new HDInsightIdentityProfile(msiResourceId: new ResourceIdentifier(msiResourceId), msiClientId: msiClientId, msiObjectId: msiObjectId);

// set the authorization profile
var userId = "{your aad user id}";
var authorizationProfile = new AuthorizationProfile();
authorizationProfile.UserIds.Add(userId);

// set the cluster node profile
string vmSize = "Standard_D8s_v3"; // your vms ize
int workerCount = 5;
ClusterComputeNodeProfile nodeProfile = new ClusterComputeNodeProfile(nodeProfileType: "worker", vmSize: vmSize, count: workerCount);

// initialize the ClusterData instance
var clusterData = new HDInsightClusterData(location)
{
    ClusterType = clusterType,
    ClusterProfile = new ClusterProfile(clusterVersion: availableClusterVersion?.ClusterVersion, ossVersion: availableClusterVersion?.OssVersion, identityProfile: identityProfile, authorizationProfile: authorizationProfile)
    {
        TrinoProfile = new TrinoProfile(),  // here is related with cluster type
    },
};
clusterData.ComputeNodes.Add(nodeProfile);

var clusterCollection = clusterPoolCollection.Get(clusterPoolName).Value.GetHDInsightClusters();

var clusterResult = clusterCollection.CreateOrUpdate(Azure.WaitUntil.Completed, clusterName, clusterData);

Create Simple Spark Cluster

// define the prerequisites information: subscription, resource group and location where you want to create the resource
string subscriptionResourceId = "/subscriptions/{subscription id}"; // your subscription resource id like /subscriptions/{subscription id}
string resourceGroupName = "{your resource group}"; // your resource group name
AzureLocation location = AzureLocation.EastUS; // your location

SubscriptionResource subscription = armClient.GetSubscriptionResource(new ResourceIdentifier(resourceId: subscriptionResourceId));
ResourceGroupResource resourceGroupResource = subscription.GetResourceGroup(resourceGroupName);
HDInsightClusterPoolCollection clusterPoolCollection = resourceGroupResource.GetHDInsightClusterPools();

// create the cluster
string clusterPoolName = "{your cluster pool name}";
string clusterName = "{your cluster name}";
string clusterType = "Spark"; // your cluster type here is Spark

// get the available cluster version
var availableClusterVersion = subscription.GetAvailableClusterVersionsByLocation(location).Where(version => version.ClusterType.Equals(clusterType, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();

// set the identity profile
string msiResourceId = "{your user msi resource id}";
string msiClientId = "{your user msi client id}";
string msiObjectId = "{your user msi object id}";
var identityProfile = new HDInsightIdentityProfile(msiResourceId: new ResourceIdentifier(msiResourceId), msiClientId: msiClientId, msiObjectId: msiObjectId);

// set the authorization profile
var userId = "{your aad user id}";
var authorizationProfile = new AuthorizationProfile();
authorizationProfile.UserIds.Add(userId);

// set the cluster node profile
string vmSize = "Standard_D8s_v3"; // your vms ize
int workerCount = 5;
ClusterComputeNodeProfile nodeProfile = new ClusterComputeNodeProfile(nodeProfileType: "worker", vmSize: vmSize, count: workerCount);

// initialize the ClusterData instance
var clusterData = new HDInsightClusterData(location)
{
    ClusterType = clusterType,
    ClusterProfile = new ClusterProfile(clusterVersion: availableClusterVersion?.ClusterVersion, ossVersion: availableClusterVersion?.OssVersion, identityProfile: identityProfile, authorizationProfile: authorizationProfile)
    {
        SparkProfile = new SparkProfile(),  // here is related with cluster type
    },
};
clusterData.ComputeNodes.Add(nodeProfile);

var clusterCollection = clusterPoolCollection.Get(clusterPoolName).Value.GetHDInsightClusters();

var clusterResult = clusterCollection.CreateOrUpdate(Azure.WaitUntil.Completed, clusterName, clusterData);

Create Simple Flink Cluster

// define the prerequisites information: subscription, resource group and location where you want to create the resource
string subscriptionResourceId = "/subscriptions/{subscription id}"; // your subscription resource id like /subscriptions/{subscription id}
string resourceGroupName = "{your resource group}"; // your resource group name
AzureLocation location = AzureLocation.EastUS; // your location

SubscriptionResource subscription = armClient.GetSubscriptionResource(new ResourceIdentifier(resourceId: subscriptionResourceId));
ResourceGroupResource resourceGroupResource = subscription.GetResourceGroup(resourceGroupName);
HDInsightClusterPoolCollection clusterPoolCollection = resourceGroupResource.GetHDInsightClusterPools();

// create the cluster
string clusterPoolName = "{your cluster pool name}";
string clusterName = "{your cluster name}";
string clusterType = "Flink"; // cluster type

// get the available cluster version
var availableClusterVersion = subscription.GetAvailableClusterVersionsByLocation(location).Where(version => version.ClusterType.Equals(clusterType, StringComparison.OrdinalIgnoreCase)).LastOrDefault();

// set the identity profile
string msiResourceId = "{your user msi resource id}";
string msiClientId = "{your user msi client id}";
string msiObjectId = "{your user msi object id}";
var identityProfile = new HDInsightIdentityProfile(msiResourceId: new ResourceIdentifier(msiResourceId), msiClientId: msiClientId, msiObjectId: msiObjectId);

// set the authorization profile
var userId = "{your aad user id}";
var authorizationProfile = new AuthorizationProfile();
authorizationProfile.UserIds.Add(userId);

// set the cluster node profile
string vmSize = "Standard_D8s_v3"; // your vm size
int workerCount = 5;
ClusterComputeNodeProfile nodeProfile = new ClusterComputeNodeProfile(nodeProfileType: "worker", vmSize: vmSize, count: workerCount);
// initialize the ClusterData instance
var clusterData = new HDInsightClusterData(location)
{
    ClusterType = clusterType,
    ClusterProfile = new ClusterProfile(clusterVersion: availableClusterVersion?.ClusterVersion, ossVersion: availableClusterVersion?.OssVersion, identityProfile: identityProfile, authorizationProfile: authorizationProfile),
};
clusterData.ComputeNodes.Add(nodeProfile);

// set flink profile
string storageUri = "abfs://{your storage account container name}@{yoru storage account}.dfs.core.windows.net"; // your adlsgen2 storage uri
FlinkStorageProfile flinkStorageProfile = new FlinkStorageProfile(storageUri);

ComputeResourceRequirement jobManager = new ComputeResourceRequirement((float)1.0, 2048);
ComputeResourceRequirement taskManager = new ComputeResourceRequirement((float)1.0, 2048);

clusterData.ClusterProfile.FlinkProfile = new FlinkProfile(storage: flinkStorageProfile, jobManager: jobManager, taskManager: taskManager);

var clusterCollection = clusterPoolCollection.Get(clusterPoolName).Value.GetHDInsightClusters();

var clusterResult = clusterCollection.CreateOrUpdate(Azure.WaitUntil.Completed, clusterName, clusterData);

Create Trino Cluster With Hms

// define the prerequisites information: subscription, resource group and location where you want to create the resource
string subscriptionResourceId = "/subscriptions/{subscription id}"; // your subscription resource id like /subscriptions/{subscription id}
string resourceGroupName = "{your resource group}"; // your resource group name
AzureLocation location = AzureLocation.EastUS; // your location

SubscriptionResource subscription = armClient.GetSubscriptionResource(new ResourceIdentifier(resourceId: subscriptionResourceId));
ResourceGroupResource resourceGroupResource = subscription.GetResourceGroup(resourceGroupName);
HDInsightClusterPoolCollection clusterPoolCollection = resourceGroupResource.GetHDInsightClusterPools();

// create the cluster
string clusterPoolName = "{your cluster pool name}";
string clusterName = "{your cluster name}";
string clusterType = "Trino"; // your cluster type

// get the available cluster version
var availableClusterVersion = subscription.GetAvailableClusterVersionsByLocation(location).Where(version => version.ClusterType.Equals(clusterType, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();

// set the identity profile
string msiResourceId = "{your user msi resource id}";
string msiClientId = "{your user msi client id}";
string msiObjectId = "{your user msi object id}";
var identityProfile = new HDInsightIdentityProfile(msiResourceId: new ResourceIdentifier(msiResourceId), msiClientId: msiClientId, msiObjectId: msiObjectId);

// set the authorization profile
var userId = "{your aad user id}";
var authorizationProfile = new AuthorizationProfile();
authorizationProfile.UserIds.Add(userId);

// set the cluster node profile
string vmSize = "Standard_D8s_v3"; // your vms ize
int workerCount = 5;
ClusterComputeNodeProfile nodeProfile = new ClusterComputeNodeProfile(nodeProfileType: "worker", vmSize: vmSize, count: workerCount);

// initialize the ClusterData instance
var clusterData = new HDInsightClusterData(location)
{
    ClusterType = clusterType,
    ClusterProfile = new ClusterProfile(clusterVersion: availableClusterVersion?.ClusterVersion, ossVersion: availableClusterVersion?.OssVersion, identityProfile: identityProfile, authorizationProfile: authorizationProfile),
};
clusterData.ComputeNodes.Add(nodeProfile);

// set secret profile
string kvResourceId = "{your key vault resource id}";
string secretName = "{your secret reference name}";
string keyVaultObjectName = "{your key vault secret name}";

var secretReference = new ClusterSecretReference(referenceName: secretName, KeyVaultObjectType.Secret, keyVaultObjectName: keyVaultObjectName);
clusterData.ClusterProfile.SecretsProfile = new ClusterSecretsProfile(new ResourceIdentifier(kvResourceId));
clusterData.ClusterProfile.SecretsProfile.Secrets.Add(secretReference);

// set trino profile
string catalogName = "{your catalog name}";
string metastoreDbConnectionUriString = "jdbc:sqlserver://{your sql server name}.database.windows.net;database={your database name};encrypt=true;trustServerCertificate=true;loginTimeout=30;";
string metastoreDbUserName = "{your db user name}";
string metastoreDbPasswordSecret = secretName;
string metastoreWarehouseDir = "abfs://{your adlsgen2 storage account container}@{your adlsgen2 storage account}.dfs.core.windows.net/{sub folder path}";

TrinoProfile trinoProfile = new TrinoProfile();
trinoProfile.CatalogOptionsHive.Add(
    new HiveCatalogOption(
        catalogName: catalogName,
        metastoreDBConnectionPasswordSecret: metastoreDbPasswordSecret,
        metastoreDBConnectionUriString: metastoreDbConnectionUriString,
        metastoreDBConnectionUserName: metastoreDbUserName,
        metastoreWarehouseDir: metastoreWarehouseDir)
);

clusterData.ClusterProfile.TrinoProfile = trinoProfile;

var clusterCollection = clusterPoolCollection.Get(clusterPoolName).Value.GetHDInsightClusters();

var clusterResult = clusterCollection.CreateOrUpdate(Azure.WaitUntil.Completed, clusterName, clusterData);

Create Spark Cluster With Hms

// define the prerequisites information: subscription, resource group and location where you want to create the resource
string subscriptionResourceId = "/subscriptions/{subscription id}"; // your subscription resource id like /subscriptions/{subscription id}
string resourceGroupName = "{your resource group}"; // your resource group name
AzureLocation location = AzureLocation.EastUS; // your location

SubscriptionResource subscription = armClient.GetSubscriptionResource(new ResourceIdentifier(resourceId: subscriptionResourceId));
ResourceGroupResource resourceGroupResource = subscription.GetResourceGroup(resourceGroupName);
HDInsightClusterPoolCollection clusterPoolCollection = resourceGroupResource.GetHDInsightClusterPools();

// create the cluster
string clusterPoolName = "{your cluster pool name}";
string clusterName = "{your cluster name}";
string clusterType = "Spark"; // your cluster type here is Spark

// get the available cluster version
var availableClusterVersion = subscription.GetAvailableClusterVersionsByLocation(location).Where(version => version.ClusterType.Equals(clusterType, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();

// set the identity profile
string msiResourceId = "{your user msi resource id}";
string msiClientId = "{your user msi client id}";
string msiObjectId = "{your user msi object id}";
var identityProfile = new HDInsightIdentityProfile(msiResourceId: new ResourceIdentifier(msiResourceId), msiClientId: msiClientId, msiObjectId: msiObjectId);

// set the authorization profile
var userId = "{your aad user id}";
var authorizationProfile = new AuthorizationProfile();
authorizationProfile.UserIds.Add(userId);

// set the cluster node profile
string vmSize = "Standard_D8s_v3"; // your vms ize
int workerCount = 5;
ClusterComputeNodeProfile nodeProfile = new ClusterComputeNodeProfile(nodeProfileType: "worker", vmSize: vmSize, count: workerCount);

// initialize the ClusterData instance
var clusterData = new HDInsightClusterData(location)
{
    ClusterType = clusterType,
    ClusterProfile = new ClusterProfile(clusterVersion: availableClusterVersion?.ClusterVersion, ossVersion: availableClusterVersion?.OssVersion, identityProfile: identityProfile, authorizationProfile: authorizationProfile),
};
clusterData.ComputeNodes.Add(nodeProfile);

// set secret profile
string kvResourceId = "{your key vault resource id}";
string secretName = "{your secret reference name}";
string keyVaultObjectName = "{your key vault secret name}";

var secretReference = new ClusterSecretReference(referenceName: secretName, KeyVaultObjectType.Secret, keyVaultObjectName: keyVaultObjectName);
clusterData.ClusterProfile.SecretsProfile = new ClusterSecretsProfile(new ResourceIdentifier(kvResourceId));
clusterData.ClusterProfile.SecretsProfile.Secrets.Add(secretReference);

// set spark profile
string defaultStorageUriString = "abfs://{your adlsgen2 storage account container}@{your adlsgen2 storage account}.dfs.core.windows.net/";
string dbServerHost = "{your sql server name}.database.windows.net";
string dbUserName = "{your db user name}";
string dbName = "{yoru db name}";
string dbPasswordSecretName = secretName;

SparkProfile sparkProfile = new SparkProfile();
sparkProfile.DefaultStorageUriString = defaultStorageUriString;
sparkProfile.MetastoreSpec = new SparkMetastoreSpec(dbServerHost: dbServerHost, dbName: dbName, dbUserName: dbUserName, dbPasswordSecretName: dbPasswordSecretName, keyVaultId: kvResourceId);

clusterData.ClusterProfile.SparkProfile = sparkProfile;

var clusterCollection = clusterPoolCollection.Get(clusterPoolName).Value.GetHDInsightClusters();

var clusterResult = clusterCollection.CreateOrUpdate(Azure.WaitUntil.Completed, clusterName, clusterData);

Create Flink Cluster With Hms

// define the prerequisites information: subscription, resource group and location where you want to create the resource
string subscriptionResourceId = "/subscriptions/{subscription id}"; // your subscription resource id like /subscriptions/{subscription id}
string resourceGroupName = "{your resource group}"; // your resource group name
AzureLocation location = AzureLocation.EastUS; // your location

SubscriptionResource subscription = armClient.GetSubscriptionResource(new ResourceIdentifier(resourceId: subscriptionResourceId));
ResourceGroupResource resourceGroupResource = subscription.GetResourceGroup(resourceGroupName);
HDInsightClusterPoolCollection clusterPoolCollection = resourceGroupResource.GetHDInsightClusterPools();

// create the cluster
string clusterPoolName = "{your cluster pool name}";
string clusterName = "{your cluster name}";
string clusterType = "Flink"; // cluster type

// get the available cluster version
var availableClusterVersion = subscription.GetAvailableClusterVersionsByLocation(location).Where(version => version.ClusterType.Equals(clusterType, StringComparison.OrdinalIgnoreCase)).LastOrDefault();

// set the identity profile
string msiResourceId = "{your user msi resource id}";
string msiClientId = "{your user msi client id}";
string msiObjectId = "{your user msi object id}";
var identityProfile = new HDInsightIdentityProfile(msiResourceId: new ResourceIdentifier(msiResourceId), msiClientId: msiClientId, msiObjectId: msiObjectId);

// set the authorization profile
var userId = "{your aad user id}";
var authorizationProfile = new AuthorizationProfile();
authorizationProfile.UserIds.Add(userId);

// set the cluster node profile
string vmSize = "Standard_D8s_v3"; // your vm size
int workerCount = 5;
ClusterComputeNodeProfile nodeProfile = new ClusterComputeNodeProfile(nodeProfileType: "worker", vmSize: vmSize, count: workerCount);

// initialize the ClusterData instance
var clusterData = new HDInsightClusterData(location)
{
    ClusterType = clusterType,
    ClusterProfile = new ClusterProfile(clusterVersion: availableClusterVersion?.ClusterVersion, ossVersion: availableClusterVersion?.OssVersion, identityProfile: identityProfile, authorizationProfile: authorizationProfile),
};
clusterData.ComputeNodes.Add(nodeProfile);

// set secret profile
string kvResourceId = "{your key vault resource id}";
string secretName = "{your secret reference name}";
string keyVaultObjectName = "{your key vault secret name}";

var secretReference = new ClusterSecretReference(referenceName: secretName, KeyVaultObjectType.Secret, keyVaultObjectName: keyVaultObjectName);
clusterData.ClusterProfile.SecretsProfile = new ClusterSecretsProfile(new ResourceIdentifier(kvResourceId));
clusterData.ClusterProfile.SecretsProfile.Secrets.Add(secretReference);

// set flink profile

string storageUri = "abfs://{your adlsgen2 storage account container}@{your adlsgen2 storage account}.dfs.core.windows.net";
FlinkStorageProfile flinkStorageProfile = new FlinkStorageProfile(storageUri);

ComputeResourceRequirement jobManager = new ComputeResourceRequirement((float)1.0, 2048);
ComputeResourceRequirement taskManager = new ComputeResourceRequirement((float)1.0, 2048);

// set flink catalog
string metastoreDbConnectionUriString = "jdbc:sqlserver://{your sql server name}.database.windows.net;database={your database name};encrypt=true;trustServerCertificate=true;loginTimeout=30;";
string metastoreDbUserName = "{your db user name}";
string metastoreDbPasswordSecret = secretName;

FlinkHiveCatalogOption flinkHiveCatalogOption = new FlinkHiveCatalogOption(metastoreDBConnectionPasswordSecret: metastoreDbPasswordSecret, metastoreDBConnectionUriString: metastoreDbConnectionUriString, metastoreDBConnectionUserName: metastoreDbUserName);
clusterData.ClusterProfile.FlinkProfile = new FlinkProfile(storage: flinkStorageProfile, jobManager: jobManager, taskManager: taskManager);
clusterData.ClusterProfile.FlinkProfile.CatalogOptionsHive = flinkHiveCatalogOption;

var clusterCollection = clusterPoolCollection.Get(clusterPoolName).Value.GetHDInsightClusters();
var clusterResult = clusterCollection.CreateOrUpdate(Azure.WaitUntil.Completed, clusterName, clusterData);

More examples

More Code samples for using the management library for .NET can be found in the following locations

• .NET Management Library Code Samples

Troubleshooting

• File an issue via GitHub Issues.
• Check previous questions or ask new ones on Stack Overflow using Azure and .NET tags.

Next steps

For more information about Microsoft Azure SDK, see this website.

Contributing

For details on contributing to this repository, see the contributing guide.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (for example, label, comment). Follow the instructions provided by the bot. You'll only need to do this action once across all repositories using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opencode@microsoft.com with any other questions or comments.