DQMS - The Data Quality Management System™

Case Studies

Already working in DQMS (3.5)? Get more details from "Best Practices: Tips & Workflow Ideas" in the help file.

Case 1: Full Processing of Solar & Meteorological Data

A weather station collects data at 1-minute intervals for: solar radiation, wind speed and direction, pressure, temperature, humidity, precipitation, and battery voltage. The station's data is recorded in UTC.

  1. Import data recorded in UTC to a station configured for a different time zone, data in DQMS is then based on local time.
  2. Test the data: Check that every time interval has a record (no missing timestamps). Check the values of individual data fields vs. a min/max value, test the difference between two fields, step size between consecutive timestamps, ratio of two fields, etc.
  3. Generate a list of missing timestamps and failed tests.
  4. Fill in missing data: copy from another table, copy from the same table (ex: from 2 days before or 1 week later), linear interpolation, do a user-entered calculation.
  5. Summarize data over different intervals getting different kinds of averages, totals, min/max, and standard deviations.
    • For ex: Take 1-minute data and make summaries on 1-hour, 1-day, 1-month or any other user-defined time interval.
  6. Make graphs of all the data and have DQMS print out the graphs as images using a time interval (ex: graph the weekly irradiance, then generate 10 weeks of images in one go).
  7. Export data in any of several formats (ex: Excel, tab-delimited text file, etc).

Case 2: Partial Data Processing Ideas

Here are examples of what your workflow might look like in conjunction with other programs:

  • Auto-generate your configured graph on a desired interval (ex: plot weekly plots of solar irradiance for every week of the year).
  • Use the Execute driver to do post-processing on the exported data files and assemble data into user-friendly report. This would be useful if you had separate code to compile DQMS outputs into one report file for your business.
  • Use DQMS with your own custom flags: devise your own test or modification process external to DQMS, passing data out of DQMS to an external file via Export, then after your custom test or processing, re-import the data along with externally generated flags back in to the DQMS data table. You can have DQMS pass control to your custom, external processing program by using the Execute driver operation.
  • Have DQMS estimate "clear sky" values for solar irradiance data for comparison purposes in tables and graphs.