How to Create HDInsight Cluster on Microsoft Azure
Login to portal.azure.com . You should have your account created and an active Subscription – either Free Trial or a Paid Subscription to create a Resource . If you don’t have an Azure Account and would like to create one , follow the step by step tutorial here – How to Create an Azure Account For Free .
Now in order to create a HDInsight Cluster 🡪 Go to +Create A Resource and Under Analytics Tab 🡪 You can see the HDInsight
This will open up the Below Page in which you need to enter some Basic Information to get your HDInsight (Hortonworks Hadoop Platform (HDP) on Cloud Azure with storage on Azure ADLS rather than HDFS) up and running.
As you can see in the screen shot it will take 20 mins time to set up this cluster on the cloud.
Fill up the Details as below. Depending on the Type of HDInsight Cluster we want to create we can chose the Cluster type.
The different Cluster type that is available while creating HDInsight Cluster on Microsoft Azure are as below:
- ML Services (R Server)
- Interactive Query
Currently we are creating Hadoop Cluster, so we will select this Option i.e. Hadoop. For Hadoop On Azure we only have the Linux Operating System and below are the Features of Hadoop on Azure and Click on “Select”
You will need a create a new Resource Group .
sshuser is the username for login from putty or any other SSH Tool.
Click on Next and you will get the below screen.
Now you need to chose the Storage Account Settings.
Then there is some optional setting that you can leave to default.
Once you click “Next” you will see the below details after validation and you are ready to create an HDInsight Big Data Cluster on Azure
You can also see the cluster details and the price that you will have to pay once your free trial or free amount of the months end .
Remember For HDInsight Cluster – Once created you will be billed until your cluster is deleted . So Make sure you Delete your Cluster once after using for testing purpose or you will exhaust all your Free Money very soon.
So this basically says you will have 2 Primary Node (1 Main & 1 for Disaster Recovery) Cluster and 4 Data Node i.e. Worker Node. So for 6 Node Cluster Azure Charges around 205 Rs per Hour i.e. Around 5000 Rs per Day and 1,50,000 Rs per Month.
We can edit the Cluster Size to lower down the price. So if I chose only 1 Data Node with 4 Cores , the Price will go down.
So Now you can see the Price of the HDInsight Cluster for 3 Node Cluster has come down to 74 Rs per Hour. Even this is nearly 1,800 Rs Per Day i.e around 50,000 Rs per Month – if you create the cluster for testing and leave it unattended.
Click on Create Button and you will get the Notification that the Cluster is getting created. It takes some time around 15 mins for this to get created and the cluster to be up & running.
Once the Deployment completes you can see in the all resources and you can see the HDInsight Cluster and Storage Account That you created