Presented by Sam Bail at Airflow Summit 2021.
Data quality has become a much discussed topic in the fields of data engineering and data science, and it has become clear that data validation is absolutely crucial to ensuring the reliability of any data products and insights produced by an organization’s data pipelines. This session will outline patterns for combining three popular open source tools in the data ecosystem – dbt, Airflow, and Great Expectations – and use them to build a robust data pipeline with data validation at each critical step.
0:00 Welcome
3:20 Quick review of dbt
6:32 Overview of Great Expectations
14:00 Integrating dbt and Airflow
27:50 Testing with dbt and Great Expectations
41:40 Wrap-up Video Rating: / 5
Health Data is traditionally held and processed in large and complex mazes of hospital information systems. The market is dominated by vendors offering monolithic and proprietary software due to the critical nature of the supported processes and – in some cases – due to legal requirements. The “digital transformation”, “big data” and “artificial intelligence” are some of the hypes that demand for improved exchange of health care data in routine health care and medical research alike. Exchanging data at these scales requires open data formats and protocols, multi-stakeholder collaboration, and agile development. As an example, the de-facto messaging standard organization in medicine HL7 noticed a much more positive response from the medical research community regarding their openly available FHIR specification in comparison to the for-members-only and XML-based HL7v3 messaging standard specification.
While some past (or rather: ongoing) projects on a national scale in the German health care system have tried centralized, top-down specification and development approaches, more recent infrastructure projects embrace the competitive collaboration of a decentralized, bottom-up strategy. As a result, importance and recognition of free software increase in the Medical Informatics research community.
In a series of rapid spotlights, we present tools and frameworks that serve as cornerstones for the envisioned health data exchange infrastructure, including: Organization and collaboration tools; data extraction from clinical source systems, data transformation and de-identification; data management systems and long-term archival using persistent globally-unique object identifiers; federated queries across multiple independently managed clinical data integration centers.
We aim to encourage participants to actively add tools and frameworks within the discussion and highlight their experiences and challenges with using open systems in Medical Informatics.
❮h3❯Speaker bio:❮/h3❯
Marcel Parciak and Markus Suhr are research associates at the University Medical Center Göttingen (UMG), Department of Medical Informatics.
Marcel graduated the Göttingen Medical Informatics Master program in 2018 and is currently a PhD student, investigating the challenges of data provenance in medical research. He is a system architect for the HiGHmed project that facilitates innovative federated infrastructure for cross-organisational secondary use of health care data.
Markus started his professional career in 2014 as a system administrator and software developer at the UMG hospital data center. He joined the Department of Medical Informatics in 2017, becoming lead developer for a free software project and working on multiple biomedical research projects. Since 2019 he is technical lead for the newly created Medical Data Integration Center. Markus is a supporter of the Free Software Foundation Europe.
Room: AW1.126
Scheduled start: 2020-02-01 11:00:00 Video Rating: / 5
Intrinseque Health, is a, EN ISO 13485 certified, Quality Driven, Project Management-based, Global Drug Development Support organization, specialising in Clinical Supply Chain solutions.
Intrinseque Health is an expert in solving supply chain challenges like, product availability and lead time, selecting in-country sourcing or import, import regulations and IOR requirements, storage, installation, calibration, validation, servicing & maintenance of equipment and supplies. Intrinseque Health engages early with its clients to ensure each supply plan is pressure tested, as the smallest flaw in the strategy may result in study delay or even loss of patient. Every item, no matter how big or small, needs to be available at site at the time pf patient visit.
Importance of building a robust clinical supply plan for a clinical trial is often underestimated – Product Availability, Lead Times, Import Regulations, IQ, OQ, PQ, Equipment Maintenance, Importer of Records, Anti-Corruption Laws are only a few factors that need to be taken into account while building a successful supply plan.
Intrinseque Health is a Quality Driven, Project Management-based, Global Drug Development Support Organization that specializes in Clinical Supply Chain solutions.
We are experts in solving supply chain challenges like, product availability and lead time, selecting in-country sourcing or import, import regulations and IOR capabilities, installation, calibration, validation, servicing & maintenance of equipment and supplies.
At Intrinseque Health, we engage early with our clients to ensure each supply plan is pressure tested, as the smallest flaw in the strategy may result in study delay or even loss of patient. Every item, no matter how big or small, needs to be available at site at the time pf patient visit.
Maintenance of distribution data & records due to recent anti-corruption enforcement activities in developing countries had made Returns, Reclamation & Destruction of Unutilized Supplies and Equipment, more important than ever. Intrinseque Health ensures adherence to all local & international regulations and considers all dynamics & characteristics of every region & country involved in the study.
Restrictions in free flow of timely delivery of shipments and enrolled patient’s visits to sites due to the COVID-19 situation, have made it difficult for clinical supplies to reach the patients. Intrinseque Health offers Direct-to-Patient services. Video Rating: / 5
The coronavirus outbreak that originated in Wuhan in December 2019 has now spread to over 50 countries across 6 continents with more than 80,000 reported cases and the death toll having crossed 2800. To contain the spread of this infection, more than 20 Chinese cities have been placed under lockdown and other countries have issued travel advisories. Recognizing the severity of the infection, the World Health Organization (WHO) has declared this outbreak a global health emergency. With cases of coronavirus outbreak being confirmed in India, are we prepared to tackle it effectively? This workshop will examine the source of the infection, evaluate its relationship with other pre-existing coronaviruses, assess its potential to become a pandemic, and discuss India’s capacity to prepare and respond to such infections.
Discussants:
Shahid Jameel, chief executive officer of the Wellcome Trust/DBT India Alliance
Gautam Menon, professor of physics and biology at Ashoka University
Chitra Pattabiraman, fellow at the National Institute of Mental Health and Neuroscience Video Rating: / 5
Vicon’s next-generation video management system features a robust health dashboard designed to help streamline your surveillance operation with interactive mapping. The Dashboard has been redesigned to provide a better user experience and clearer graphical tools.
Learn more about Valerus and special promotional offers here: www.vicon-security.com/forms/valerus-offers/ Video Rating: / 5
