LOADING

Type to search

Log Analysis of Web Application using Spark ( java ) Using Intellij & Maven

To Know more about the Different Corporate Training & Consulting Visit our website www.Instrovate.com Or Email : info@instrovate.com or WhatsApp / Call at +91 74289 52788

Intellij Maven Spark

Log Analysis of Web Application using Spark ( java ) Using Intellij & Maven

Share

 USE CASE – Log Analysis of Web Application using Spark (Java)        

Link to download Intellij Idea –

https://www.jetbrains.com/idea/download/#section=windows

1. Create a Maven Project in Intellij Idea.

     File 🡪 New 🡪Project              

    Select Maven and then next.                

Log Analysis of Web Application using Spark      ( java )  Using Intellij & Maven 29

Fill the GroupId and ArtifactID as shown in the image. Click on next and finish.

Log Analysis of Web Application using Spark      ( java )  Using Intellij & Maven 30

2. Add  spark-core_2.10 &  spark-sql_2.10 as dependency in Pom.xml

<!– https://mvnrepository.com/artifact/org.apache.spark/spark-core –>
<dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-core_2.10</artifactId>
   <version>1.1.0</version>
</dependency>

<!– https://mvnrepository.com/artifact/org.apache.spark/spark-sql –>
<dependency<!– Spark SQL –>
   
<groupId>org.apache.spark</groupId>
   <artifactId>spark-sql_2.10</artifactId>
   <version>1.1.0</version>
</dependency>

Log Analysis of Web Application using Spark      ( java )  Using Intellij & Maven 31

3. Create a Java Class under src 🡪 main 🡪 java and save it as LogAnalysis.

Log Analysis of Web Application using Spark      ( java )  Using Intellij & Maven 32

3.1 This application analyses the Apache Web Application Access log and provides information after analysis the logs. The Below points are covered in this analysis –

                                                                 Access Log Analysisusing Spark SQL
1.Load data from access.log and Print schema using Spark SQL
2. Print the Content Average Size, Count of requests, Min & Max of Content Size
3. Find out what types of Response Code received by users.
4. Any IP Address that has accessed the server more than 10 times.
5. Peak traffic load timings – top 10 timeframes for high traffic. This will be used to analyze the peak      hours of traffic in Application
6. Top Endpoints accessed by the users.

4. Build the Project using Maven. Under Maven tab, first click on clean and then on install.

Log Analysis of Web Application using Spark      ( java )  Using Intellij & Maven 33

This will provide a jar of the project at the specified location.

Log Analysis of Web Application using Spark      ( java )  Using Intellij & Maven 34

5. Upload the Jar to the server using WinScp.

Log Analysis of Web Application using Spark      ( java )  Using Intellij & Maven 35

6. Transfer the log file (access.log) from local to server and copy the log file to HDFS location.

  • hdfs dfs -copyFromLocal access.log /user/jai/log_analysis/
  • hdfs dfs -ls /user/jai/log_analysis/
Log Analysis of Web Application using Spark      ( java )  Using Intellij & Maven 36

7. Finally, type the below Command to Submit the Spark Job to the Cluster

    spark-submit \

    –class LogAnalysis  \

   –master local[*] \

   Log_Analysis-1.0-SNAPSHOT.jar \

  /user/jai/log_analysis/access.log