DAD - Distributed Apps Dev and Cloud - Dezvoltarea Aplicatiilor Distribuite si Cloud

Mission

Technological transfer from university to the students of practical and theoretical issues related with Distributed Applications Development and Cloud.

Please follow the info @:

* For the period September 2023 - February 2024, we have at room 2001 the following activities: Lecture - on Tue from 18:00 | Seminar/Laboratories - on Tue, from 19:30

We have used this platform - Zoom including in the university rooms (old link: https://us02web.zoom.us/j/82880201368?pwd=SitqdW9NUDNMVTd3VnRBSHJucENPUT09), because we had to compensate the poor image quality from the video projector and as preventive "training" for any unforeseen situations. The Cyber Security Master Program: http://www.ism.ase.ro has been upgraded on Tue, March 9-th, 2020, the Zoom platform, so that the working session will be extended from 40 minutes to 12 hours.

The presence is mandatory both in the course, as well as in the seminar / lab.

Lectures & Labs

No.  Lecture 
PDF
Lecture Description Sources for the 
Lecture
Lab Description Sources for the
Lab 
-

INTRO - Distributed Applications Development administrative issues

How does a PC work? 
https://www.youtube.com/watch?v=d86ws7mQYIg

Computational Thinking:
https://acs.ase.ro/code-club 

Linux OS Commands Recap:

https://www.youtube.com/watch?v=ghTJS52ghGU&list=PL1bEcqKguUdOHl1nqggWCqTWx9vyoyWJ9
https://www.youtube.com/playlist?list=PL1bEcqKguUdOHl1nqggWCqTWx9vyoyWJ9

OS Recap: 

https://www.youtube.com/watch?v=b6MxozZv6-M
https://www.youtube.com/playlist?list=PLRJP-g0nSX0e0kgIW6bom0kQtx4dyHyZx

Distributed Systems:
https://www.youtube.com/watch?v=NYBKXzl5bWU&list=PLRJP-g0nSX0e0kgIW6bom0kQtx4dyHyZx

https://www.youtube.com/watch?v=sGzQT_ZrsFI

1.

RECAP: Java Multi-threading mechanisms, java.util.concurrent package in samples for HPC - High Performance Computing samples | IPC - Inter-Process Communication in Linux using C/C++, Linux C POSIX Multi-Threading (PThread), Java Multi-threading and Parallel Computing Intro. Multi-tenancy via multi-process or multi-threading.

For the Java intro full details (RECAP sections), please see the lectures and labs of Multi-paradigm Programming - Java @ http://acs.ase.ro/java and first three activities

For Parallel Computing (incl. C OpenMP) intro please see lectures and labs details @ http://acs.ase.ro/parallel-processing

Distributed vs Parallel Computing Systems
* HTC (High Throughput Computing) vs. HPC (High Performance Computing)

RECAPJava Multi-threading mechanisms, java.util.concurrent package in samples for HPC - High Performance Computing samples 
2.

RECAP: TCP/IP Network Concepts - OSI Model, IP v4 vs. v6, UDP headers, NAT, PAT, Network Programming in Java - UDP, SNMP client over UDP sample - network monitoring (run in Docker containers (Native Cloud: CaaS - Container as a Service).

ANT, Maven, Gradle, OS VM vs. Docker - virtualization, DevOps Jenkins Intro

Intro in Java Reflection & Annotations, JNI - Java Native Interface, Java Exception mechanism, Source Code Design Patterns - Singleton.

RECAP: Network UDP - Java UDP Network programming using Socket, SNMP Client sample - simple and with library snmp4j run in Docker containers (Native Cloud: CaaS - Container as a Service)
3.

RECAP: Java 8+ Lamda with Multi-threading, TCP/IP Network Concepts - Network Programming in Java - UDP & TCP with multi-threading, IP, TCP headers, TCP state machine, NAT, PAT, Network Programming in Java - TCP

RECAP: Java 8+ Lambda - Network HTTP-Multi-threading and TCP with lambda Example - web server 
4.

Java Servlet Intro - Apache Tomcat web server with servlet container distribution, Session Tracking, HTTP Requests and Responses, Cookies, HTTP 1.1 

Cloud PaaS Intro: Deployment of Servlet into Google App Engine - PaaS Cloud.

HTTP/1.1 understanding with Wireshark - Java Servlet Sample with annotation instead web.xml descriptor, deployment in Apache Tomcat 9 + processing a HTML form with Java Servlet.

5.

Java Server Pages - JSP Intro - Apache Tomcat web server with Servlet/JSP container distribution, JSP Architecture, Directives and Actions, Objects, Tag-Libs, Beans, MVC - Model View Controller Type 1 and 2 Intro

Servlet and JSP are part of Web curriculum per se, and they are requred in this disscipline as data gathering/Cloud Gateway for massive distributed and parallel computing in back-end servers: Cloud, GRID, Fog infrastructures 

Java Server Pages - JSP Sample deployment in Apache Tomcat 9 + processing a HTML form with JSP and SQLite/MongoDB access.

Local deployment into VMBox.

6.

RPC - Remote Procedure Call and Java RMI - Remote Method Invocation

Java RMI sample for adding two integer vectors.
7.

SOA (Service Oriented Architectures) WebServices SOAP - Simple Objects Access Protocol

Process Migrations vs. Data Migrations within a Distributed System | TCP Socket example for passing business logic and data migration

SOA WebServices SOAP - Java JAX-WS in Eclipse (XML Recap)

Stand-alone client and server application

JAX-WS Web Server app

8.

RESTlful Services  vs. SOA | JAX-RS with HTTP/2

Difference between HTTP/1.1 and HTTP/2.0 with Wireshark network packets analyzer

HTTP/2 Client API with JDK11+

Open MPI

Cloud IaaS Intro: Deployment of OpenMPI programs in Amazon EC2

RESTful service in Java JAX-RS in Eclipse (JSON Recap)
T9.

TEST on PC

TEST on PC
9.

node.js INTRO/RECAP

Cloud FaaS - Functions as a Service / AWS Lamda Sever-less Programming Example vs. Fog Computing 

Angular.js vs. REACT vs. Apache Cordova + WebAssembly - Web/Mobile Front-End Development is NOT part (beyond) of this curriculum of this lecture - they should be part of Web or Mobile topic

Node.js JavaScript/ECMAScript - incl. WebServer & RESTful service in Node.js + WebSockets (Java/Node.js)
10.

Middleware & MoM: JMS vs. Apache Kafka

Middleware: Apache JMS and Kafka implementation for the Producer and Consumer Clients example
11.

Distributed Components: EJB - Enterprise Java Beans / Java | Jakarta EE platform vs. Actors in Akka 

Apache EJB 3.x and MDB - Message Driven Bean vs. Akka actors examples
12.

Intro n-Tier Architecture, Back-end Servers Clustering & Servers Infrastructures & (Native) Clouds - IaaS, PaaS, CaaS, SaaS, FaaS and Fog Computing

Infrastructures for Microservices / PODs and Containers as a Service Cloud - CaaS - Docker | Kubernetes

Spring Boot: JMS and components beans plus RESTful Services 

13.

14.

Spring Microservices in Action with Java/Kotlin and WASM - Web Assebmly as alternative to Java Applets technology

Spring End2End Microservice Example

E-Learning Solution & E-Framework Solutions

If necessary, the E-Learning solution platform is based on Moodle and SAKAI CLEand it is provided by ISM - IT | Cyber Security Master program: http://ism.ase.ro and ACS - Applied Computer Science And Cyber Security Team: http://acs.ase.ro. Moodle and SAKAI e-Learning Systems are available (OLD address: http://37.120.250.22:8020/portal )

E-Framework solutions for Distributed Applications Development and cloud (DAD) lectures and labs are stored in Git. The students may get them into the Oracle VM VirtualBox client virtual machine or Docker Container Image, and they are provided by Cyber Security Master program - http://ism.ase.ro
The details are the following:

1. The source code for the lectures and labs is in Git @: https://github.com/critoma/dad 

2. Docker Container Image with Linux Ubuntu 20.04 LTS for download (docker pull critoma/linux-ubuntu-dev-security-ism)

https://hub.docker.com/r/critoma/linux-ubuntu-dev-security-ism

Evaluation & Gradebook Mark Solution 

The final exam mark is 70%, and the assesments/homeworks/reports/test&quizzes done by the student, during the semester are 30% from the evaluation mark.
All the content related with the tests, assesments and the final exam will be uploaded accordinglly with the established deadlines in the E-Learning System Solution based on Moodle or SAKAI and provided by IT&C Security Master program - http://ism.ase.ro.

Semester Assignment

Deadline - YYYY/MM/dd HH:mm: 2024/01/15 (Current Year/01/15 23:50 GMT+2)
Problem description: "Create 6 Docker containers from Docker Image: 
https://hub.docker.com/r/critoma/linux-u20-dev-security-ism
for zooming in/out a large BMP picture. Please draw the architecture in draw.io/diagrams.net/Visio/ or equivalent Tools and implement the following requirements.

The picture is received from the Front-end (Plain JS/CSS/HTML or Angular or Vue.js or REACT) into the 1st container Back-End via REST API (Javalin or Jakarta EE Servlet REST API) and then is published as binary message into a JMS Topic from the 2nd container which is a MoM JMS Broker. In the 3rd container a Jakarta EE EJB MDB is subscribed to the 2nd container JMS Topic and it is also Java RMI Client for 2 Java RMI Servers which are running in the 4th and 5th containers. When the BMP picture is ready, then it is stored in 6th container MySQL DB as BLOB - Binary Large Object. The 6th container has 2 DBs - one MongoDB for collecting SNMP values from all the nodes regarding OS name, CPU and RAM usage and a MySQL DB for storing modified pictures. All these are exposed through node.js JS Express REST Endpoints - one for SNMP values display and one for BMP picture rendering. When the zoomed BMP pic is inserted in the MySQL, then the 1st container via REST API/WebSocket is notifying the front-end with HTTP redirection to the node.js URL for downloading the picture. As a list there are the following containers:

Front-end (JS, Angular, Vue.js) -> load pic&params(Zoom In/Out, +/- %) via REST API 
-> C01: Java Javalin/Apache Tomcat 10 Microserv + JMS Client Publisher
-> C02: Apache TomEE 9 w JMS Broker - Topic & Queues
-> C03: Apache TomEE 9 w EJB Client MDB & RMI Client <- zoom pics united into DB via REST API & JMS Client Publisher in Topic/Queue that the job is done
-> C04, C05: Apache TomEE 9 w RMI Server objects
<- C01: Notification that the pic is ready => in the front-end with link for download from node.js C06
-> C06: REST API in node.js w MongoDB with 2 REST API endpoints interfaces: 1 for DB access and 1 for Web Access

#### ---------

Creati 6 containere Docker din Docker Image: https://hub.docker.com/r/critoma/linux-u20-dev-security-ism

pentru a mari/micsora o imagine BMP mare. Desenati arhitectura in draw.io/diagrams.net/Visio/ sau instrumente echivalente si implementati urmatoarele cerinte.

Imaginea este primita de la front-end (Plain JS/CSS/HTML sau Angular sau Vue.js sau REACT) în primul container Back-End prin REST API (Javalin sau Jakarta EE Servlet REST API) si apoi este publicata ca binar mesaj într-un JMS Topic din al 2-lea container, care este un broker JMS MoM. In al 3-lea container, un MDB Jakarta EE EJB este abonat la al 2-lea container JMS Topic si este, de asemenea, Java RMI Client pentru 2 Servere Java RMI care ruleaza în al 4-lea si al 5-lea container. Cand imaginea BMP este gata, atunci este stocata în al 6-lea container MySQL DB ca BLOB - Binary Large Object. Al 6-lea container are 2 DB-uri - un MongoDB pentru colectarea valorilor SNMP de la toate nodurile privind numele sistemului de operare, utilizarea CPU si RAM si o DB MySQL pentru stocarea imaginilor modificate. Toate acestea sunt expuse prin node.js JS Express REST Endpoints - unul pentru afisarea valorilor SNMP si unul pentru redarea imaginii BMP. Când imaginea BMP marita este inserata în MySQL, atunci primul container prin REST API/WebSocket notifica front-end-ul cu redirectionare HTTP catre URL-ul node.js pentru descarcarea imaginii. Ca o lista exista urmatoarele containere:

Front-end (JS, Angular, Vue.js) -> încarcati pic&params (Marire/Marire, +/- %) prin API-ul REST

-> C01: Java Javalin/Apache Tomcat 10 Microserv + JMS Client Publisher

-> C02: Apache TomEE 9 w JMS Broker - topic si cozi

-> C03: Apache TomEE 9 w EJB Client MDB & RMI Client <- zoom imaginile unite în DB prin REST API si JMS Client Publisher în Topic/Queue ca poza este gata

-> C04, C05: Apache TomEE 9 w obiecte RMI Server

<- C01: Notificare ca fotografia este gata => în front-end cu link pentru descarcare de pe node.js C06

-> C06: API REST în node.js w MongoDB cu 2 interfe?e de puncte finale API REST: 1 pentru acces DB si 1 pentru acces web

Upload and packiging: "The ZIP file has the source code files + *.sh/*.bat file for compiling and running the samples. The ZIP filename rule will have DAD2023_A01_SURNAME_FirstName1_FirstName2.zip". The upload is available through online.ase.ro

Prerequisites

Fundamentals of Java SE plus C/C++  with JavaScript/ECMAScript/node.js + Networking + Linux/Windows OS | Optional – Python.

Target Group Profile

References

 

1.  Thomas Erl, Eric Barcelo Monroy, Cloud Computing: Concepts, Technology, and Architecture, Second Edition, Pearson, 2023

2. Justin Domingus, John Arundel, Cloud Native DevOps with Kubernetes, 2nd Edition, O'Reilly Media, Inc., 2022

3. Brendan Burns, "Designing Distributed Systems", Publisher: O'Reilly Media, Inc., Release Date: February 2018, ISBN: 9781491983645; with references to: Kelsey Hightower, Joe Beda, Brendan Burns, "Kubernetes: Up and Running, 2nd Edition", Publisher: O'Reilly Media, Inc., Release Date: October 2019, ISBN: 9781492046530

4. Kalpit Patel, John Carnell, "Spring Microservices in Action", Publisher: Manning Publications, Release Date: July 2017, ISBN: 9781617293986

5. Christina J. Hogan, Strata R. Chalup, Thomas A. Limoncelli, "Practice of Cloud System Administration, The: DevOps and SRE Practices for Web Services, Volume 2", Publisher: Addison-Wesley Professional, Release Date: September 2014, ISBN: 9780133478549 

6. George Coulouris, Jean Dollimore, Tim Kindberg, Gordon Blair, "Distributed Systems: Concepts and Design", Publisher Addison-Wesley; 5 edition (April 27, 2011), ISBN-10: 0132143011, ISBN-13: 978-0132143011 

7. Andrew S. Tanenbaum, Maarten Van Steen, "Distributed Systems: Principles and Paradigms (2nd Edition)", Publisher Prentice Hall; 2 edition (October 12, 2006), ISBN-10: 0132392275, ISBN-13: 978-0132392273

My Experience and lectures/labs slides presentations, examples, virtual machines + your visual, kinetic and auditory memory + Moodle/SAKAI e-Learning System PROVIDED by ISM – IT&C Security Master Program - http://ism.ase.ro | http://acs.ase.ro

###############################################################################################################################################

*** DEPRECATED / OBSOLETE - IGNORE:

Assignment 01.

Deadline - YYYY/MM/dd HH:mm: Current Year/10/27 23:50 GMT+2
Problem description: "Create multiple processes for communicating one each other, through UPD multicast/broadcast packages, in order to find out the best mark in the room. Each process has its own mark from 1 to 10 stored in command line for launching or it is stored in a file on the disk. The problem and the solutions have been explained in the lecture 01. You may develop the solution for this problem in Java/Kotlin or C++ / C# / node.js / Swift / Python." 
Upload and packiging: "The ZIP file has the source code files + *.sh/*.bat file for compiling and running the sample. The ZIP filename rule will have DAD2022_A01_SURNAME_FirstName1_FirstName2.zip". The upload is available through online.ase.ro..

Assignment 02.

Deadline - YYYY/MM/dd HH:mm: Current Year/10/27 23:50 GMT+2 
Problem description: "Build SNMP client program which is able to send SNMP traps in Java over HTTP or Websocket."

Upload and packiging: "ZIP file uploaded in SAKAI. The ZIP file has the PDF file that contains the steps + *.sh file for compiling and running the sample + *.java files with the source code + readme.txt which contains the configuration steps. The ZIP filename rule will have DAD2022_A02_SURNAME_FirstName1_FirstName2.zip". The upload is available through online.ase.ro

Assignment 03.

Deadline - YYYY/MM/dd HH:mm: Current Year/11/03 23:50 GMT+2
Problem description: "

3.1 Create a C program under Linux for adding to two arrays of float using IPC multi-process techniques - e.g. fork - https://www.geeksforgeeks.org/fork-system-call/ | https://www.includehelp.com/c-programs/c-fork-function-linux-example.aspx | https://stackoverflow.com/questions/15102328/how-does-fork-work

3.2 Create a C program under Linux for adding to two arrays of float using POSIX threads - https://www.cs.cmu.edu/afs/cs/academic/class/15492-f07/www/pthreads.html

3.3 Create a C++ '11/'17 program under Linux for adding to two arrays of float using objects from class thread - https://stackoverflow.com/questions/266168/simple-example-of-threading-in-c | https://www.geeksforgeeks.org/multithreading-in-cpp/ 

3.4 Create a C/C++ program under Linux for adding to two arrays of float using socket programming - https://gist.github.com/saxbophone/f770e86ceff9d488396c0c32d47b757e | https://www.geeksforgeeks.org/udp-server-client-implementation-c/  

3.5 Build client-server programs in Java. The client read the arrays from files and send them using Sockets and it transmits TCP packets. The servers receives the array and process them using multi-threading. The results are sent back asynchronous by the server to the clients by using Java NIO API. - Please see the discussion thread @ https://stackoverflow.com/questions/56747881/how-to-use-multiple-cores-in-the-java-non-blocking-i-o-nio-api"

Upload and packiging: "ZIP file uploaded in SAKAI. The ZIP file has the PDF file that contains the steps + *.sh file for compiling and running the sample + *.java files with the source code. The ZIP filename rule will have DAD_A03_SURNAME_FirstName1_FirstName2.zip". The upload is available through Moodle or SAKAI, please see E-Learning Solution section from this page.

Assignment 04.

Deadline - YYYY/MM/dd HH:mm: Current Year/11/10 23:50 GMT+2
Problem description: "Download laboratory/seminary with TCP HTTP Server server - proof of concept, in Linux Ubuntu (download the virtual machine from the current page - http://acs.ase.ro/dad), in order to solve HTTP requests for resources of type *.class. The URL in Internet browser - Mozilla Firefox/Google Chrome for http://ip_server:port_server/myfile.html will display the HTML file if exists, BUT when the browser asks http://ip_server:port_server/MyClassFile.class will display the string chars produced by method 'public String sampleDoGet();'. The one should modify method 'public String processInput(String theInput);' from class 'eu.ase.httpserver.HTTPSeminarProtocol' in order to handle through Java reflection the requiered task."
Upload and packiging: "ZIP file uploaded in SAKAI. The ZIP file has the Java source code *.java files + *.sh file for compiling and running the sample. The ZIP filename rule will have DAD_A04_SURNAME_FirstName1_FirstName2.zip". The upload is available through SAKAI, please see E-Learning Solution section from this page.

Assignment 05.

Deadline - YYYY/MM/dd HH:mm: Current Year/11/30 23:50 GMT+2
Problem description: "Download lecture and laboratory/seminary in Linux Ubuntu (download the virtual machine from this page - http://acs.ase.ro), in order to solve the following problem:
Develop the JSP and Java bean necessary for Apache Tomcat deploy that will receive via HTTP to vectors and will responde with the sum of the vectors. The Java bean from web will be the RMI client for 3 RMI servers objects, each RMI server object is registred in the same rmiregistry or in different rmiregistry program. Both versions, a) and b) are required.
Version a) - The RMI Server is developed in standard mode as in lecture sample (RMI01 and RMI03 directories).
Version b) - The RMI Server is developed in order to receive the bytecode of business logic (adding two vectors) in the same way as in ComputePi solution - ./downloads_readme/rmidocs/rmiDocsExec"
Upload and packiging: "ZIP file uploaded in SAKAI. The ZIP file has the Java source code *.java files + *.sh file for compiling and running the sample. The ZIP filename rule will have DAD_A05_SURNAME_FirstName1_FirstName2.zip". The upload is available through SAKAI, please see E-Learning Solution section from this page.

Assignment 06.

Deadline - YYYY/MM/dd HH:mm: Current Year/12/15 23:50 GMT+2 
Problem description: "Download activity with OpenMPI examples - proof of concept, in Linux Ubuntu (download the Docker container / virtual machine from the current page - http://acs.ase.ro/dad), in order to highlight 'broadcast', 'gather', 'scatter' on addition operation of two large arrays of integers - e.g. each having 100 000 000 elements. => each array elements occuping in RAM aprox. 300 - 400 MB" 
Upload and packiging: "ZIP file uploaded in SAKAI. The ZIP file has the C source code *.c files + *.sh file for compiling and running the sample. The ZIP filename rule will have DAD_A06_SURNAME_FirstName1_FirstName2.zip". The upload is available through SAKAI, please see E-Learning Solution section from this page.

* The Linux Ubuntu 16.04 LTS OS VM as Oracle VirtualBox OVA Appliance for download - with GUI and Eclipse 2018-09 & IntelliJ IDEA 2018 as Java-IDE - 14GB @: 

https://drive.google.com/open?id=1kKZChf2qbYME9PtO3Gu9ZkecRoNykWsD

* Partial alternative to Oracle VirtualBox OVA, Docker Container Image with Linux Ubuntu 16.04 LTS for download - without GUI - 4GB @: 

https://hub.docker.com/r/critoma/ubuntu-java-node-py-dev/ 

Beyond the GUI and Eclipse & IntelliJ Java-IDE differences, both, the Oracle VirtualBox OVA appliance and the Docker Container Image contain:

Linux OS Ubuntu 16.04 x64 Xenial with: Java - Oracle JDK 8 and 9 and 11Node.js 8.9 - JavaScript / ECMAScript, Python 2.7 and 3, GitApache ANT 1.10.1, Apache Maven 3.5.2, Gradle 4.5, Apache Tomcat 9.0.4, Jenkins Generic WAR 2.89.3 LTS.

E-Framework solutions for DAD lectures and labs were based till year 2017 on VMWare client virtual machines, and they have been provided by IT&C Security Master program - http://ism.ase.ro 

The OLD web address for download will be available for: 

- Linux Ubuntu 12 LTS (Can be updated to Ubuntu 14 with Java 8): with JDK 6 + Eclipse Indigo JEE + Netbeans + Apache Tomcat + JBOSS + GCC - Download 
- Alternatively, the latest versions of software may be installed by students in Ubuntu 14 VM (Oracle VM Box): 
JDK 8 + Eclipse LUNA + GCC - Download U14

Old Sakai instances: http://37.120.250.22:8088/portal https://86.55.177.110:7443