Dynamic Real-time Infrastructure Planner (DRIP)

Zhiming Zhao (Principal investigator)

z.zhao@uva.nl

System and Network Engineering Research Group

University of Amsterdam

 

 

 

About DRIP

 

About

Architecture

Publications

Demo

Use cases

Support

Download

Home

The Dynamic Real-time infrastructure planner (DRIP) allows application developers to seamlessly plan a customized virtual infrastructure based on application level constraints on QoS and resource budgets, provisioning the virtual infrastructure using standardized interfaces (e.g., TOSCA and OCCI), deploy application components onto the virtual infrastructure, and start execution on demand.

 

The DRIP is developed by the research team of Quality Critical Applications on Programmable Infrastructure (QCAPI) in the System and Network Engineering Research Group (SNE) at University of Amsterdam. The development of DRIP is coordinated by Dr. Zhiming Zhao. Core developers include Junchao Wang, Huan Zhou, Yang Hu, Dr. Paul Martin, Dr. Arie Taal and Dr. Spiros Koulouzis.

 

The DRIP components are made available as open source under the Apache 2.0 license; the software has been containerized and can be provisioned and deployed on federated virtual infrastructures. The development of DRIP is supported by the European Union Horizon 2020 research and innovation program under grant agreements No. 643963 (SWITCH), No. 654182 (ENVRIPLUS) and No. 676247 (VRE4EIC).

 

 

 

Architecture and key components

 

 

 

The DRIP system contains a number of micro services (agents):

 

 

Key components:

1.     The DRIP manager will be implemented as a Web service that allows DRIP functions to be invoked individually or in tandem by outside clients via a RESTful interface.

2.     The infrastructure planner (planning agent) will use an adapted partial critical path algorithm to produce efficient infrastructure topologies based on application workflows specified in YAML in compliance with the TOSCA specification.

3.     The infrastructure provisioner (provisioning agent) takes the infrastructure plans produced by the planner using TOSCA as input, and directs the provisioning of virtual infrastructure onto underlying infrastructure services. It uses OCCI as default provisioning interface, and currently has been tested on ExoGENI, EC2 and EGI FedCloud.

4.     The deployment agent installs application components onto provisioned infrastructure.

5.     Discovery agents periodically update cloud provider information used for the planning and provisioning of infrastructure (including resource types), and helps acquire information regarding the performance of particular application components on specific resource types to put into the knowledge base.

6.     A knowledge base uses the semantic models developed in the SWITCH and ENVRIPLUS projects, and maintains the descriptions of the cloud providers, resource types, performance characteristics, and other relevant information. The knowledge base also provides an interface for these agents to look up providers, resources and runtime status data during the execution of an application.

7.     A number of control agents provide access to the underlying programmability provided by the virtual infrastructures, e.g., horizontal and vertical scaling of virtual machines, by providing interfaces by which the infrastructure hosting an application can be dynamically manipulated at runtime.

 

 

 

Publications          

 

 

Part of the results have been published in:

  1. Wang, J., Taal, A., Martin, P., Hu Y., Zhou, H., Pang, J., de Laat, C., Zhao, Z. (*), (2017) Planning Virtual Infrastructures for Time Critical Applications with Multiple Deadline Constraints, International journal of Future Generation Computer System [doi:10.1016/j.future.2017.02.001]
  2. Wang, J., de Laat, C., and Zhao, Z. (2017) QoS-Aware Virtual SDN Network Planning, IFIP/IEEE International Symposium on Integrated Network Management, Lisbon, Portugal, May 8-12, 2017. (to appear)
  3. Zhao Z., Martin, P., de Laat C., Jeffery, K., Jones A., Taylor T., Hardistry A., Arkinson M., Zuiderwijk A., Yin Y., Chen Y. (2016) Time critical requirements and technical considerations for advanced support environments for data-intensive research, 2nd International workshop on Interoperable infrastructures for interdisciplinary big data sciences (IT4RIs 16), in the context of IEEE Real-time System Symposium (RTSS), Porto, Portugal, November 29-December 2, 2016. [doi:10.5281/zenodo.204756]
  4. Zhao, Z., Altmeyer, S., Keith J., Atkinson, M. and Ulisses A.(2016), Nearly real time data processing and time critical cloud applications, Proceedings of the 2nd International workshop on Interoperable infrastructures for interdisciplinary big data sciences (IT4RIs 16), in the context of IEEE Real-time System Symposium (RTSS), Porto, Portugal, November 29-December 2, 2016. [doi:10.5281/zenodo.204685]
  5. Zhou, H., Martin, P., Su, J., de Laat, C. and Zhao, Z. (2016) A Flexible Inter-locale Virtual Cloud For Nearly Real-time Big Data Applications, Proceedings of the 2nd International workshop on Interoperable infrastructures for interdisciplinary big data sciences (IT4RIs 16), in the context of IEEE Real-time System Symposium (RTSS), Porto, Portugal, November 29-December 2, 2016. [doi:10.5281/zenodo.204774]
  6. Zhou, H., Hu Y., Wang, J., Martin, P., Su, J., de Laat, C. and Zhao, Z., (2016) Fast Resource Co-provisioning for Time Critical Application Based on Networked Infrastructure, IEEE International Conference on CLOUD (CLOUD) 2016, San Francisco US.
  7. Zhou, H., Hu Y., Wang, J., Martin, P., de Laat, C. and Zhao, Z., (2016) Fast and Dynamic Resource Provisioning for Quality Critical Cloud Applications, IEEE International Symposium On Real-time Computing (ISORC) 2016, York UK [doi:10.1109/ISORC.2016.22]
  8. Martin P., Taal A., Quevedo F., Rogers D., Evans K., Jones A., Stankovski V., Taherizadeh S., Trnkoczy J., Suciu G., Zhao Z, (2016) Information modelling and semantic linking for a software workbench for interactive, time critical and self-adaptive cloud applications, the International Workshop on Cloud Computing Project and Initiatives (CCPI-2016), in the proceedings of The 30th IEEE International Conference on Advanced Information Networking and Applications (AINA-2016), Crans-Montana, Switzerland, March 23-25, 2016.[doi:10.1109/WAINA.2016.38]
  9. Evans, K., Jones, A., Preece, A., Quevedo, F., Rogers, D., Spasić, I., Taylor, I., Stankovski, V., Taherizadeh, S., Trnkoczy, J., Suciu, G., Suciu, V., Martin, P., Wang, J., Zhao, Z. (2015) Dynamically Reconfigurable Workflows for Time-Critical Applications, International workshop on Workflows in support of large-scale science (WORKS 15), in the context of IEEE Supercomputing 2015. [doi:10.1145/2822332.2822339]
  10. Mork, R., Martin, P., Zhao, Z. (2015) Contemporary Challenges for Data-intensive Scientific Workflow Management Systems, International workshop on Workflows in support of large-scale science (WORKS 15), in the context of IEEE Supercomputing 2015.[ doi:10.1145/2822332.2822336]
  11. Zhao, Z., Martin, P., Wang, J., Taal, A., Jones, A., Taylor, I., Stankovski, V., Garcia Vega, I., Suciu, G., Ulisses, A., Cees de Laat, C. (2015), Developing and operating time critical applications in clouds: the state of the art and the SWITCH approach, In the proceedings of HOLACONF - Cloud Forward: From Distributed to Complete Computing, Pisa, Italy.  Elsevier, Procedia Computer Science, Vol (68) page (17-28)[doi:10.1016/j.procs.2015.09.220]
  12. Zhao, Z., Martin, P, Grosso, P., Los, W., de Laat, C., Jeffrey, K., Hardisty, A., Vermeulen, A., Castelli, D., Legr, Y., Kutsch, W. (2015) Reference Model Guided System Design and Implementation for Interoperable Environmental Research Infrastructures, 1st International workshop on Interoperable infrastructures for interdisciplinary big data research (IT4RIs) In the proceedings of IEEE eScience Munich Germany, p583-588 [doi:10.1109/eScience.2015.41]
  13. Martin, P., Grosso, P., Magagna, B., Schentz, H., Chen,Y., Hardisty, A., Los, W., Jeffrey, K., de Laat, C.,Zhao, Z. (2015) Open Information Linking for Environmental Research Infrastructures, 1st International workshop on Interoperable infrastructures for interdisciplinary big data research (IT4RIs) In the proceedings of IEEE eScience Munich Germany, p546-553 [doi:10.1109/eScience.2015.66]
  14. Zhao, Z., Taal, A., Jones, A., Taylor, I., Stankovskic, V., Garcia, I., Jesus, F., Suciue, G., Ulisses, A., Ferreira, P., de Laat, C. (2015), A software workbench for interactive, time critical and highly self-adaptive cloud applications (SWITCH), poster paper, In the proceedings of IEEE CCGrid (15th IEEE/ACM Int'l Symposium on Cluster, Cloud and Grid Computing), May 4-7, Shenzhen, China. [doi:10.1109/CCGrid.2015.73]

 

 

 

Demo                       

 

 

 

1.     Version 1 release, 2017, March [Demo of planner, provisioning agent, deployment agent, and drip as a whole]

2.     Virtual infrastructure planning for time critical applications [Demo in 2016];

3.     Network transparent provisioning for large scale virtual infrastructure on multiple data centers [Publications] [Video];

 

 

 

Use cases                            

 

 

 

DRIP has been applied in a number of use-cases in the context of a number of different projects:

1.     The elastic disaster early warning system (provider: BEIA Consult International). DRIP creates a concrete virtual runtime environment based on input from the SWITCH Interactive Development Environment SIDE (developed in the SWITCH project), creates a Service Level Agreement (SLA) to negotiate with the resource provider, and deploy the services to the virtual runtime infrastructure after it is provisioned.

2.     Collaborative real-time business communication platform (provider Wellness Telecom). The DRIP system creates a concrete virtual runtime environment based on the input from SIDE. In the virtual runtime environment, a number of Software Defined Networking enabled switches are defined in the infrastructure to connect the SoftMCU and the TURN/STUN servers. Afterwards, DRIP creates a Service Level Agreement (SLA) to negotiate with the Cloud provider.

3.     Cloud studio for directing and broadcasting live events (provider MOG Technologies). DRIP creates a concrete virtual runtime environment based on the input received from SIDE, and creates a Service Level Agreement (SLA) to negotiate with the resource provider and deploy the virtual video switch services to the virtual runtime infrastructure after it is provisioned.

4.     Euro-Argo data subscription services (provider Euro-Argo, EMSO and ICOS). Euro-Argo, EMSO and ICOS Research Infrastructures will push a series of data and metadata products onto a future data cloud. A data subscription service will be developed for scientific users. Data provided for the cloud service can come in a variety of forms, e.g., Argo observations updated daily, in situ observations provided by Copernicus, EMSO observatory data, or ICOS-SOCAT carbon data observed from voluntary observation ships. DRIP plans virtual infrastructure for the data subscription and distribution services, and provisions it for the application.

5.     ICOS data processing (provider ICOS). The eddy covariance data are 10Hz measurements of wind (3 directions) and gas concentrations collected at sites based in different ecosystems. To calculate the net exchange of gases, energy and temperature between ecosystems and atmosphere at 30 minutes time resolution needs a number of calculations and processing steps to be performed, forming the basis of the use-case.

 

 

 

Support                   

 

 

 

The DRIP system is developed using the Apache license version 2.0. It is open source, and there is no warrant on the software quality. We welcome any suggestions on software features, use-cases and joint research activities. Please contact us via email: z.zhao@uva.nl.

 

 

 

Copyright reserved 2016