Enabling Large-Scale Analytics for Avionics Flight Data

MetroStar partnered with the U.S. Air Force to develop a secure, classified, open-source platform to tame their data deluge.

Enabling Large-Scale Analytics for Avionics Flight Data

MetroStar partnered with the U.S. Air Force to develop a secure, classified, open-source platform to tame their data deluge.

using open source for defense missions

Empowering Data Analysts with Hardened Open-Source Tools

MetroStar unified open-source tools within a secure data science platform allowing defense agencies to unlock critical data for their mission goals. The platform was successfully deployed as part of a partnership between MetroStar, the U.S. Air Force’s 309th Software Engineering Group (SWEG), and the Extreme Digital Development Group Enterprise (EDDGE). Together, our solution empowers government-employed data analysts by helping them access and understand the possibilities that powerful and modern, military-hardened data science tools create.

Platform One is a collection of approved, hardened Cloud Native Computer Foundation (CNCF)-compliant Kubernetes distributions, infrastructure as code playbooks, and hardened containers. Platform One has continued to grow as a platform for data analysts to easily access modern tools and open-source resources in support of critical defense missions.

The 309th Software Engineering Group (SWEG)—a group of aggressive adaptors of open-source software data science as a core capability—is a group within the U.S. Air Force dedicated to supporting software for data analysts use at multiple hardened, secure bases. Their goal is to streamline data and communications across on-prem systems.

EDDGE is a team within the 309th SWEG, that collaborates closely with MetroStar to provide Free and Open-Source Software (FOSS) through Platform One.

overview

Data analysts are often challenged by a lack of access to hardened, open-source software, hindering their ability to modernize their data analysis. To combat this, the Air Force, with SWEG, sought to develop a secure, classified, open-source platform that would enable the analysis of large-scale analytics from avionics flight data.

MetroStar, in collaboration with the Air Force, was tasked with shifting flight data analytics and DevSecOps development efforts to a modern ‘big data’ framework that was consistent across varying development environments and weapons platforms.

What is Big Data? Copious amounts of data require more complex data processing software to manage and make sense of the available data.

Free and Open-Source Software (FOSS) data platforms provide access to certain functions on disconnected on-premises (on-prem) and virtual private clouds, allowing for stronger and more fluid data conversations. MetroStar is committed to making new tools open-source and as secure as possible, as well as flexible, maintainable, and capable of expansion because of a challenge called Vendor Locking.

What is Vendor Locking? When users become dependent on a singular product and service and lose out on the benefit of using another vendor because it costs too much to switch services or products. 

The Open Source Initiative (a group of analysts with the freedom to run, review, alter, and enhance code to further a project) builds community and provides toolsets and resources that reduce vendor lock-in. Open-source tools on Platform One—where defense analysts solve challenges together—allow agencies to build safe, secure communities to power innovation and save money and time. Furthermore, the size and variability of the data range within platforms are many petabytes.

Open Platform for Advanced Learning (OPAL) is a machine learning platform made in tight collaboration with industry (MetroStar) and government (Air Force 309th). OPAL is horizontally scalable and lightweight enough to deploy with relative ease. In addition to ease of deployment, MetroStar is training future platform users to maintain and expand OPAL. Data analysts are learning to use the platform to monitor the proper avionics analytics and leverage the data collected for insightful use.

measurable mission success

Data Driving Further Insights + Inspirations

0%
Accuracy in Model Insights
0+
Petabytes of Data from Multiple Weapons Platforms
0+
IL2 to IL6+ Platform Deployment Objectives
0+
Open Source Project Contributions

solutions + approach

Customized Free and Open-Source Accelerators:  Onyx  & OPAL

To tackle the data deluge—having more data than scientists can realistically manage and create meaningful insights with—agencies must receive access to collaborative, cloud-based, scalable platforms; open-source tools make this possible.

Inside look at Onyx & OPAL

MetroStar’s open-source Onyx platform is the predecessor of OPAL and contains a custom experiment tracking system that integrates with JupyterHub, allowing teams of data scientists to audit, track, and automatically evaluate experiments under any fairness and explainability criteria. In addition, Onyx is 100 percent sustainable and extensible, which helps agencies avoid vendor lock-ins.

OPAL is a transformed copy of Onyx. The transformation included the necessary changes for the platform to be made available on Platform One for government agencies to unlock critical data to reach their mission’s goals. In collaboration with the Air Force 309th, OPAL is a secure and classified, open-source platform, enabling large-scale analytics for avionics flight data. This FOSS product produces a lightweight cloud-native data platform that performs near real-time petabyte-scale data analysis, cutting quality assurance and regression time by several orders of magnitude.

OPAL is an approachable yet attainable set of machine learning capabilities that effectively integrates data workflows, enabling the release of mission-critical government products. MetroStar’s transformation of the Onyx platform led to a solution that gives the right people in government agencies the right resources, at the right time, to resolve data access challenges.

Mock Up of Open-Source tool Onyx PDF

Why Onyx & OPAL Matter

Onyx and OPAL leverage community-baked, open-source components, eliminating ongoing licensing costs by sharing the maintenance costs among the technology community.

The platforms also improve decision-making by allowing users to spend less time managing and tracking data and more time deriving actionable intelligence to make valuable mission decisions. Furthermore, Onyx and OPAL streamline data architecture to enable easier access across an organization or agency, which encourages transparent, accurate, and faster collaboration between users.

Access to open-source tools has historically been largely unavailable to the government. Government computers cannot readily download tools from the internet, but OPAL is an approved, stable, and secure open-source resource that can be downloaded by government tech professionals. OPAL, and the work of MetroStar with the Air Force 309th, is helping government employees finally access the best tools to accomplish their missions in speed.

request demo

Fully Open-Source

Community-Driven, Customer-Controlled

Massive Scale Data Processing & Visualization

Successfully hosted on USAF Platform One

we strive for great results

A lightweight, license-free, government-built data platform that we can use in a variety of environments with relative ease has been a game-changer in understanding and leveraging our vast caches of data. MetroStar has been a strong partner, has fully committed to our open vision, and continues to help us aggressively pursue advances in this important project.

Zayd L. Ma, EDDGE Technical Manager, CIV USAF AFMC 520 SWES/EDDGE

accelerate your mission with meaningful and targeted digital transformations

Learn more about how MetroStar’s playbooks, accelerators, and industry experts can help move your organization’s mission forward.

work with us