Enabling Large-Scale Analytics for Avionics Flight Data
MetroStar partnered with the U.S. Air Force to develop a secure, classified, open-source platform to tame their data deluge.
Enabling Large-Scale Analytics for Avionics Flight Data
MetroStar partnered with the U.S. Air Force to develop a secure, classified, open-source platform to tame their data deluge.
using open source for defense missions
Empowering Data Analysts with Hardened Open-Source Tools
MetroStar unified open-source tools within a secure data science platform allowing defense agencies to unlock critical data for their mission goals. The platform was successfully deployed as part of a partnership between MetroStar, the U.S. Air Force’s 309th Software Engineering Group (SWEG), and the Extreme Digital Development Group Enterprise (EDDGE). Together, our solution empowers government-employed data analysts by helping them access and understand the possibilities that powerful and modern, military-hardened data science tools create.
overview
Data analysts are often challenged by a lack of access to hardened, open-source software, hindering their ability to modernize their data analysis. To combat this, the Air Force, with SWEG, sought to develop a secure, classified, open-source platform that would enable the analysis of large-scale analytics from avionics flight data.
MetroStar, in collaboration with the Air Force, was tasked with shifting flight data analytics and DevSecOps development efforts to a modern ‘big data’ framework that was consistent across varying development environments and weapons platforms.
What is Big Data? Copious amounts of data require more complex data processing software to manage and make sense of the available data.
Free and Open-Source Software (FOSS) data platforms provide access to certain functions on disconnected on-premises (on-prem) and virtual private clouds, allowing for stronger and more fluid data conversations. MetroStar is committed to making new tools open-source and as secure as possible, as well as flexible, maintainable, and capable of expansion because of a challenge called Vendor Locking.
What is Vendor Locking? When users become dependent on a singular product and service and lose out on the benefit of using another vendor because it costs too much to switch services or products.
The Open Source Initiative (a group of analysts with the freedom to run, review, alter, and enhance code to further a project) builds community and provides toolsets and resources that reduce vendor lock-in. Open-source tools on Platform One—where defense analysts solve challenges together—allow agencies to build safe, secure communities to power innovation and save money and time. Furthermore, the size and variability of the data range within platforms are many petabytes.
Open Platform for Advanced Learning (OPAL) is a machine learning platform made in tight collaboration with industry (MetroStar) and government (Air Force 309th). OPAL is horizontally scalable and lightweight enough to deploy with relative ease. In addition to ease of deployment, MetroStar is training future platform users to maintain and expand OPAL. Data analysts are learning to use the platform to monitor the proper avionics analytics and leverage the data collected for insightful use.
measurable mission success
Data Driving Further Insights + Inspirations
solutions + approach
Customized Free and Open-Source Accelerators: Onyx & OPAL
To tackle the data deluge—having more data than scientists can realistically manage and create meaningful insights with—agencies must receive access to collaborative, cloud-based, scalable platforms; open-source tools make this possible.
Inside look at Onyx & OPAL
MetroStar’s open-source Onyx platform is the predecessor of OPAL and contains a custom experiment tracking system that integrates with JupyterHub, allowing teams of data scientists to audit, track, and automatically evaluate experiments under any fairness and explainability criteria. In addition, Onyx is 100 percent sustainable and extensible, which helps agencies avoid vendor lock-ins.
OPAL is a transformed copy of Onyx. The transformation included the necessary changes for the platform to be made available on Platform One for government agencies to unlock critical data to reach their mission’s goals. In collaboration with the Air Force 309th, OPAL is a secure and classified, open-source platform, enabling large-scale analytics for avionics flight data. This FOSS product produces a lightweight cloud-native data platform that performs near real-time petabyte-scale data analysis, cutting quality assurance and regression time by several orders of magnitude.
OPAL is an approachable yet attainable set of machine learning capabilities that effectively integrates data workflows, enabling the release of mission-critical government products. MetroStar’s transformation of the Onyx platform led to a solution that gives the right people in government agencies the right resources, at the right time, to resolve data access challenges.
Why Onyx & OPAL Matter
Onyx and OPAL leverage community-baked, open-source components, eliminating ongoing licensing costs by sharing the maintenance costs among the technology community.
The platforms also improve decision-making by allowing users to spend less time managing and tracking data and more time deriving actionable intelligence to make valuable mission decisions. Furthermore, Onyx and OPAL streamline data architecture to enable easier access across an organization or agency, which encourages transparent, accurate, and faster collaboration between users.
Access to open-source tools has historically been largely unavailable to the government. Government computers cannot readily download tools from the internet, but OPAL is an approved, stable, and secure open-source resource that can be downloaded by government tech professionals. OPAL, and the work of MetroStar with the Air Force 309th, is helping government employees finally access the best tools to accomplish their missions in speed.
Fully Open-Source
Community-Driven, Customer-Controlled
Massive Scale Data Processing & Visualization
Successfully hosted on USAF Platform One
we strive for great results
– Zayd L. Ma, EDDGE Technical Manager, CIV USAF AFMC 520 SWES/EDDGE
