The Science Data Cloud combines a technology framework and a public cloud platform to provide Data-as-a-Service and Info-as-a-Service. This enables a multi-layered ecosystem of services to evolve that leverages the value from EO data provided by the European Space Agency.
The service is built on three stack layers, the front-end layer, orchestration layer and infrastructure layer. The infrastructure layer consists of the cloud orchestration, provisioning platform and core infrastructure, with data sets hosted in CloudSigma’s custom object storage solution. Upon this the Science Data PaaS layer is integrated to provide ontological data access and enable end-users to discover the service either via the cloud service marketplace or via an external third- party store. End-users also have the ability to discover applications, connect directly to the relevant data sets, and provision appropriately sized virtual servers accordingly.
The framework is tested against a Flood Delineation for Damage Assessment use-case, demonstrating how end-users - in this case regional civil protections agencies across Europe - can access datasets provided by ESA (including Sentinel-1) and process such data via the chosen application partner. Our vision is one where everyone can access data sets in an environment that allows immediate development and manipulation of the data and a ready onward market for the results of that effort.
The project OPUS-GMES creates an operational platform for provision and processing of Sentinel-data in support of Copernicus geoinformation services (BStMWi) (2013-2017) and is lead by DLR.
CloudEO platform acts as an example of a third-party cloud platform where processes developed by research or industry partners can be offered to the industry with secured intellectual property. Innovative and flexible license and business models are supported for project work, development, and production.
About cloudeo
cloudeo operates a unique, vendor-independent, data-agnostic market platform through which customers can obtain professional geoinformation services from leading national and international providers at low cost. Services include dedicated solutions for many industries, such as agriculture, urban and landscape planning, logistics, telecommunications, and water management. Customers are provided with high-resolution imagery, 3D terrain models, thematic maps, and sensor data for many applications. The TimeCaster™ service in our cloudeo web application is available on our store.
Big Earth Datacube Analytics Made Easy
The proliferation of earth observation satellites has resulted in a wealth of image data from proprietary sources like Intermap as well as imagery that is available from public agencies like the European Space Agency. But availability does not equal accessibility: the challenges range from knowing what data are actually available to understand how to process the various data on various platforms and consolidate them into one usable product. To solve this problem, Jacobs University Bremen initiated a consortium to explore the possibilities of Array Database Systems (also called datacube) brought together the Germany-based global leader in datacube technology, rasdaman GmbH (Bremen), with German Aerospace Center DLR (Deutsches Zentrum für Luft- und Raumfahrt, Munich), and cloudeo AG (Munich) as the leading provider of cloud-based earth observation data and services. The project, which ran for 18 months, from the beginning of 2018 until mid-2019, was financed by the German Federal Ministry for Economic Affairs and Energy, BMWi.
From the perspective of Jacobs University, the project was conceived to demonstrate the integration of free, publicly available data with commercially available data services, using the innovative “any query, any time, on any size” datacube paradigm – in technical lingo, “analysis-ready Spatio-temporal raster data.” For the user experience, the goal was to create a simple interface that offers seamless access to multiple data sources, which were distributed globally (Germany, US, Australia) and available in different resolutions – but combined and analyzed into one single environment.
For cloudeo, this project offered an opportunity to showcase its platform-as-a-service (PaaS) capabilities. To this end, rasdaman servers were initially installed on the German Copernicus hub, CODE-DE, as well as in the commercially hosted, “mix and match” processing environment of cloudeo: Complementing the services that were already available on CODE-DE, rasdaman offers essential additional functionality, in particular an interactive paradigm that abstracts away the specific domain knowledge for different EO databases. The cloudeo Service API handled everything from order orchestration to data accessing and processing, including the monetization through the cloudeo store. To accompany the pilot project, BigDataCube Terrain profiles were also made available as Data-as-a-Service offerings in the cloudeo store.
Using the cloudeo platform as a service minimized the administrative effort for potential customers: these users just needed to specify whether they needed terrain profiles – defined by two points and their latitudinal/longitudinal position – or viewsheds, described by one point (latitude/longitude) and an angle between 0° and 360° degree.
The resulting multiresolution elevation services are strictly based on open geo standards. For this particular pilot project, cloudeo combined commercial data, provided by Intermap and PlanetObserver, with open source data (in this case, the OpenData_Bavaria data set); the cloudeo store acted as the customer interface to products and geoservices, and the cloudeo Service API handled everything from authenticating users to generating reports and billing the customers.
Ultimately, this project’s goal was to establish best practices for using data in operational environments and to initiate a federation of rasdaman installations on other EO data centers. This will encourage novel, specialized services by third parties quickly, flexibly, and scalably.
The BigDataCube project team: (from left to right): Stefan Wiehle (DLR), Dr. Ursula Benz (CEO CloudEO AG), Dimitris Bellos (CloudEO AG), Simon Tzvetanov (CloudEO AG), Vlad Merticariu (Jacobs University Bremen, rasdaman GmbH), Sven Jacobsen (DLR), Brennan Bell (Jacobs University Bremen), Bang Pham Huu (Jacobs University Bremen), Dimitar Misev (Jacobs University Bremen, rasdaman GmbH), Prof. Dr. Peter Baumann (CEO rasdaman GmbH, Jacobs University Bremen). Photo: Heike Hoenig, rasdaman GmbH
Contact:
- Prof. Dr. Peter Baumann, Jacobs University Bremen
- Website: www.bigdatacube.org
The project TEISS (Thales ESA INFOaaS Stimulus) was initiated and supported by the European Space Agency ESA to use Earth Observation data and stimulate the development propagation of Information as a Service (INFOaaS). The purpose of this particular project was to analyze how providers of data, service, and cloud resources can use highly automated end-to-end workflows that involve joint resources like Helix Nebula Marketplace to make the information fast and easily accessible to customers while offering reach competitive pricing for this INFOaaS.
The project leader was Thales Services, part of the French technology conglomerate THALES Group and a leading expert in satellite image processing; project partners were cloudeo and the Swiss software and cloud technology provider SixSq. The latter was chosen for their technical leadership in the Helix Nebula initiative, which is supported by ESA, CERN, and the European Molecular Biology Laboratory (EMBL); cloudeo was selected for its substantial experience in providing the actual marketplace for such cloud-based services, i.e., the interface where customers can order services or request quotes.
For the TEISS project, cloudeo was tasked to develop a technical interface between the front-end (practically the cloudeo store) and the back end (in this case, the Helix Nebula MarketPlace), using an API call from the cloudeo store to SixSq SlipStream to access EO data from public sources like the European Sentinel or Landsat from the US, as well as commercial data from providers like Deimos, Airbus, and DigitalGlobe. This API call then launches the appropriate service over the selected area of interest subscribed by an end-user. The various Service Providers would provide these final products/services to cloudeo for delivery to the end-users through a set of appropriate tools like visualization GUI, web services, APIs, or FTP; one example would be the Web Mapping Services provided by EOMAP Service (accessible via a web portal http://eoapp-au.eomap.com/).
This project's primary purpose was to prove that INFOaaS can provide sustainable revenues from existing EU investments, like the EU earth observation program, which in turn has the potential to create many new jobs addressing modern information sector needs. The Earth Observation community, like all the other communities, is deeply impacted by the digital economy. The inevitable changes, driven by this digital economy, represent a paradigm shift that could boost the development of a whole range of innovative services, combining Earth observation data with big data applications.
But to get there, the highly specialized remote sensing industry must find ways to get connected to the mainstream mass market. New low-cost satellite constellations (e.g., Skybox, PlanetLabs, UrtheCast), together with existing public satellite services like the Sentinel, present enormous opportunities to bring satellite-based remote sensing into the economic mainstream – a mass market that is already is looking for value-added EO data and information services that are easy to find and consume, providing images and data for a specified area of interest and time. Most importantly, to be useful for a broad range of clients, data need to available regardless of the type of sensors, software, and processing power used to generate them, which is precisely the cloudeo mission.
CIINDY - CloudEO GeoMarketplace and IPT Poland Integration Study
The European Space Agency (ESA) has funded the EO Innovation Platform Testbed Poland to give with EO Cloud easy access to Sentinel and Landsat data to researchers and industry. CloudFerro is building up the infrastructure and offers since 2016 access for first users.
Within the project, CIINDY studies are performed on how a connection between the GeoMarketplace CloudEO and the EO Cloud in Poland can benefit developers and providers of GeoServices in terms of better and faster access to the market and more cost-efficient production.
Cloudeo teamed up for the studies with Exelis VIS, a Harris Geospatial Solutions company, and with Science [&] Technology from Norway.
CIINDY - I
Within CIINDY-I, the first prototypical connection of CloudEO Store to the EO Cloud was set up to let customers order on CloudEO Store a change detection analysis for a certain area using Landsat or Sentinel data.
We are proud of the success of this project.
- Now, anyone who wants to order the GeoService change detection from Harris Geospatial Solutions can do so on the CloudEO GeoMarketplace and will benefit from the fast execution on the EO Cloud.
- Much more, every remote sensing or GIS scientist developing ENVI or IDL analytics and wanting to take benefit from the connection CloudEO GeoMarketplace and EO Cloud can do so easily and without any extra costs for this powerful software package!
To achieve this goal, Exelis VIS and CloudEO teamed up. With the support of CloudFerro, the workflow was set up, and the first integration between the EO Cloud and CloudEO Store is operational.
The customer can select between a multitude of change detection parameters and select his desired scenes from the full data catalog.
Once the customer orders the change detection, it is executed automatically on the EO cloud. This includes access to the data.
If Sentinel-2 data are used or Landsat data on Europe, the data are on the platform, and thus the EO cloud gives a big-time advantage for the analysis. The process can be executed within a few seconds or minutes. However, the process also executes without human intervention if the data are not available, e.g., Landsat data outside Europe. In this case, an additional process automatically requests the scenes from Amazon Web Services and starts the change detection process as soon as the data are available.
CIINDY - II Onboarding of GeoService Silvisense
Within CIINDY II, the Norwegian company Science and Technology and CloudEO teamed up to onboard the GeoService Silvisense onto CloudEO GeoMarketplace and to explore together the benefits the EO Cloud on IPT Poland brings in terms of easy data access and processing capabilities.
CloudEO uses its experience from CIINDY I to connect the Store through CloudEO Order Orchestration with the EO Cloud of CloudFerro.
Already, the basic version of Silvisense, a detailed land cover map, provides a helpful input for companies and institutions in the forestry, agriculture, and land planning business. This map serves as the entry to advanced forest health analytics from Silvisense or as input for other value adders to accelerate and improve their processes.
Main project tasks
CloudEO presents the GeoService on CloudEO Store and implements the e-Commerce system.
Customers will be able to order this GeoService as a one-time analysis or subscribe to it as a monitoring service. The pricing is determined by the size of the selected area, the geometric resolution of the analysis, the length of the monitoring period, and the analysis's detail.
Different payment options are possible. In any case, a direct bank transfer is enabled. For order volumes below 500 Euro, payment with a credit card or PayPal can be selected. For order volumes larger than 3000 Euro, direct bank transfer and payment on the invoice are offered. New customers need to pay in advance.
With ordering, the customer accepts the Terms and Conditions of CloudEO. These conditions cover not only the needs of CloudEO but also the needs of the GeoService provider and avoid the need for the customer to sign multiple Terms and Conditions to consume one GeoService.
Once the payment is received, the CloudEO Order Orchestration system triggers and controls the processing chain. The customer will find this information in the tab "Current Services" in his dashboard. When the results are produced, the customer gets notified and receives a link to download the results.
Project progress:
- With milestone 1, May 19th, anyone interested can already find the GeoService on CloudEO Store and order it.
- With milestone 2, July 31st, the full order, payment, processing, and the delivery chain will be implemented, providing in a fully automated workflow the results within few hours after data availability.
Is it technically and economically feasible to use commercial remote sensing imagery for crowdsourcing to validate geo-data?
A case study was conducted within the ESA Project "Thematic Exploitation Platform URBAN (TEP Urban)," in cooperation with Pallas Ludens and cloudeo.
Earth Observation imagery is the basis for nearly every geoinformation service. Such large datasets, often in the range of terabytes, require quality assurance, which is generally done based on sample data; similarly, training data are needed to train automated classification algorithms. For geoinformation services like the European Copernicus program, quality assurance through validation is a prerequisite. The collection of sample data is based on reference image data or ground truth data; validation is typically performed by geo-experts in a very elaborate, time-consuming, and costly process, especially for large-area and semantically complex products.
One possible approach to this problem is to use non-experts and crowdsourcing. To explore this possibility, the European Space Agency launched a case study within their project "Thematic Exploitation Platform URBAN" (TEP Urban); the German Aerospace Center DLR (Deutsches Zentrum für Luft- und Raumfahrt) subcontracted the companies Pallas Ludens and cloudeo to answer the question: Is it technically and economically feasible to use commercial remote sensing imagery for crowdsourcing to validate geo-data?
For this study, the DLR Global Urban Footprint (GUF) product was used as an example for a classification layer. Pallas Ludens provided their expertise in crowdsourcing techniques; cloudeo was chosen for their valuable experience with making EO data accessible to many users. In addition to brokering access to data from their content partners, cloudeo also evaluated possible business models for the particular use case of crowdsourcing.
Pallas Ludens was tasked with setting up the technical framework for a state-of-the-art crowd-based collection of sample data, consisting of multi-resolution geo-imagery; cloudeo provided their Service API to set up the interface for the application in one homogenous web service access. About the validation of buildings or group of buildings within GUF, real color imagery was pre-selected, with a pixel size equal to or better than 15 m:
• Very high-resolution imagery ( resolution better than 1m)
• High-resolution imagery (resolution better than 2.5m)
• Visually improved imagery with 15m resolution.
The cloudeo Service API creates homogeneous access to the data and simplifies access for the crowdsourcing application. Besides, the embedded metering supports pay-per-use and application-specific business models.
To control this case study's cost, cloudeo selected services from SI Imaging and Airbus Defence and Space, and PlanetObserver, who offered data for free for the study. California and New Delhi were chosen as areas of interest; a detailed analysis of crowdsourcing performance was executed on the California test site. The New Delhi demo area was used as the second test site.
cloudeo holds agreements with some providers, like PlanetObserver, which allows archive data with temporal licenses for one day, one, or several months for reduced costs. This model is very well suited for crowdsourcing. Due to a smart selection of areas and tasking by the application, a fast access time to the image will be, in most cases, sufficient to get the results. Thus, the application may even benefit from a one-day license. This daily license was offered for PlanetObersver's PlanetSAT 15 imagery. cloudeo hosted the global data set of PlanetSAT 15 and provided this data set as an international web service. The pricing depends on the number of users and the area. It is independent of the number of map views. The pricing seems suitable for the low-cost crowdsourcing application: For a minimum fee of 500 Euro, nearly 30,000 sq. km. can be accessed for one month.
Initiated at the 4th Ground Segment Coordination Body Workshop on 24th September 2015 at ESA-ESRIN (Frascati, Italy), the Copernicus World Alliance (CWA) is gathering 30+ European industry stakeholders addressing Earth Observation (EO) Services.
The Copernicus World Alliance is
- helping the EO community to tackle the challenges coming from the digital economy
- supporting EARSC initiative for creating a European MarketPlace for EO Services (dowload pdf here)
- promoting INFOrmation as a Service (INFOaaS) based on Earth Observation data and services which bring value to public and private end-users able and willing to pay such services
For example, you may be interested in tiny but smart EO services such as the ones described below.
EOhopS: EO data hosted processing for science
When the European Space Agency decided to introduce the European science community to cloud-based services for earth observation data, it approached cloudeo to build the necessary platform. The idea was to include so-called Third-Party Mission (TPM) – i.e., not provided and hosted by ESA¬ – data in this cloud-based approach: currently, users have to physically download such TPM data from ESA archives or the data providers' servers. This process uses extensive bandwidth resources and requires large storage and data management capacities on the users' side.
Cloudeo with many years of experience and expertise in cloud-based Data-as-a-Service (DaaS), Software-as-a-Service (SaaS), and Platform-as-a-Service (PaaS) projects and products. They developed their Service API to create a single interface, allowing platform-independent access to large files (typical for Earth Observation data). In essence, the cloudeo Service API makes homogeneous access to the data and simplifies access for the crowdsourcing application. The idea is that, rather than acquiring expensive software licenses and purchasing large, bandwidth- and storage-heavy EO imagery data packages, customers can pay for the usage and processing of such files using virtual workstations that include time-based software licenses. This model was very well suited for the EOhopS project; mainly, because the embedded metering supports pay-per-use and application-specific business models, allowing Third-Party providers to monetize their data without concerns about their IP rights.
The project's scope was to provide scientific users with access to these third-party EO data in a hosted processing infrastructure (similar to the cloudeo Workbench) using Virtual Machines. Simultaneously, the users were also invited to upload their content, like in-situ measurement data or scientific algorithms, to this virtual Workbench. This concept allows users to process these vast volumes of data in the cloud and to download only the desired results to their local computing environment, thus saving costly bandwidth and storage resources while at the same time protecting the copyright of the TPM providers.
This model came with many logistical challenges that needed to be resolved, such as license conditions of EO Data-as-a-Service, allocation of remote processing capabilities to users, and secure connections of this cloud processing infrastructure TPM data providers for ordering and archive access, etc.
Since this was an exploratory program, projects were selected not only on their scientific merit but also based on how well they were able to use the new technological affordances, namely the capacity to process large amounts of data remotely, to upload their content to a remote environment for analysis, and to make only the processing results available for download.
Only a limited number of projects could be served within ESA resources allocated to the service; projects were accepted, based on their matching with the above objectives and a first-come-first-served basis. The demand exceeded the project capacity: As of now, all EOhopS resources have been allocated, and no further project applications are currently accepted.
