WP2 - System Architecture, Data Management and Processing

---

Objectives

Defining and designing the SEN3APP system for supporting terrestrial cryospheric and boreal Copernicus/GMES services, including data management, processing and system control.
Organizing the high-volume data ingestion and handling.
Developing software for providing a homogenous Near-Real-Time (NRT) Level 1 dataset of Sentinel-1, -2, and -3 data in the form of:
(a) ortho-rectified and terrain-corrected backscattering coefficient (#o or #o, to be defined) in the case of SAR data, and
(b) ortho-rectified and atmospherically corrected ground reflectance or brightness temperature in the case of optical data.
Providing services with mechanisms for automated, continuous processing of data.
Enabling integration of relevant external data sources, for offering combined products that are of interest to specific user segments.

Description of work and role of partners

WP2 - System Architecture, Data Management and Processing [Months: 1-30]

Lead beneficiary: VTT
The WP2 focuses on: (a) the development and adaptation of Sentinel Level 1 (L1) processing lines for improved intermediate products,
(b) the development of techniques for the automatic Sentinel data acquisition,
(c) the establishment of tools and techniques to monitor the status of complete processing lines,
(d) setting up the system hardware solutions and
(e) the specification of all system component interfaces.

T2.1 - System requirement specification and internal interfaces [Months: 1-15]

VTT, GAMMA, ENVEO, SYKE
The final requirements for the system infrastructure, functionality and interfaces are verified and the planning of the work adjusted accordingly. Involving the owners of Copernicus/GMES applications/services of the project, this task will define the interfaces for 1) the incoming Sentinel-data, 2) software modules of the optical and SAR preprocessing systems, and 3) software modules of the Copernicus/GMES applications/services of the project. The required software modules are also defined on a general level. Additionally, relevant external data sources and methods for accessing them are investigated and their integration enabled.

T2.2 - System deployment and control [Months: 3-15]

FMI, SYKE, VTT
This task shall define the system architecture and organize the overall system-level developments, including the computer infrastructure and hardware/software environments to implement the SEN3APP system on. Deployment options include distributed and centralized approaches, either sufficiently expanding the existing operational systems, or extending highly scalable cloud-based data hosting services. Also, process control tools are defined, utilizing experience from previous automated operational services by FMI and SYKE.

T2.3 - Processing lines for optical system [Months: 3-30]

VTT
This task will define, design, and implement the software needed for operational and automatic pre-processing of incoming Sentinel-2 and -3 data and automatic supply of the pre-processed data to the software modules of the Copernicus/GMES applications/services of the project. Metadata on pre-processing steps and image data quality are also generated to accompany the pre-processed Sentinel-2 and -3 data. Software from the pre-existing (background) Envimon system is used where viable.

T2.4 - Level 1 processing lines for SAR [Months: 3-30]

GAMMA
This task will develop and install the SAR processing line to provide L1 products from L0 Sentinel 1 data. The provided products include co-registered MLI and SLC data, as well as ortho-rectified backscatter images and repeat pass coherence and phase information.

T2.5 - Data ingestion tools [Months: 3-15]

FMI, VTT
Tools will be developed to request Sentinel-data from the ground segment, and to access the requested data from the ground segment. To support RT and NRT applications, data watch mechanism shall be developed; to enable immediate initiation of processing for any newly received raw data. Push-based data delivery from the PDGS to the SEN3APP system, with notification mechanism, will be utilized when available. The tools are developed to interface with the optical and SAR processing chains developed in WP2.3 and WP2.4 and the overall system control in WP2.2.

T2.6 - Product selection and access [Months: 12-30]

VTT, FMI, ENVEO
External system interfaces are defined and thoroughly documented, allowing integration of both the raw and processed products to external services with a near-real-time access. The interfaces support requesting customized products based on specific requirements on area, time, resolution etc. and features extracted by analysis, such as cloud or snow cover. Subscriptions to automatic repeated delivery of products are enabled, based on fine-grained user parameters. An accessible web-based user interface is offered, with search mechanisms that facilitate exact finding of the products of interest and product customization. The work of this task is aligned with and, if possible, builds on the concepts of ESA Multi Mission User Service (MMUS) environment and its Next Generation User Services for Earth Observation (ngEO) component. ENVEO will develop tools for accessing, automatic downloading and importing Sentinel-1, -2,-3 products into ENVEO software environment.

T2.7 - Interoperability and workflows for user controlled services [Months: 12-30]

SYKE, FMI
Methods and tools for executing user-customizable optical processing chains are defined. SYKE, in collaboration with FMI will define, design and implement methods of running externally developed software modules of the selected Copernicus/GMES products and services in FMI's premises for minimizing the need of data transfer between the organizations. Also methods and tools for executing user-customizable optical processing chains are defined.