Difference between revisions of "Data Warehouse"
| (7 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | Since HiveTool is open source/open notebook, the entire primary record is publicly available online as it is recorded. | + | Since HiveTool is open source/open notebook, the entire primary record is publicly available online as it is recorded. Two databases are used: |
| + | Input and Analysis. | ||
| − | In addition to the measured data, there are external factors that need to be systematically and consistently documented. | + | Each hive sends in data every five minutes, 288 times a day, inserting over 100,000 rows a year into the Input Database. One thousand hives would generate 100 million rows per year. In addition to the measured data, there are external factors that need to be systematically and consistently documented. This metadata includes hive genetics, manipulations, mite treatments, data conversion and calibration formulas, etc. |
| − | [[File: | + | [[File:Database_servers-2-1.jpg|thumb 640px|Operational and Research Databases]] |
| − | + | ||
| + | As the data is moved from the Input Database to the Analysis Database it should: | ||
| + | |||
| + | *Converted to other measurement systems (lb <=> kg, Fahrenheit <=> Celsius). | ||
| + | *Manipulation changes filtered out | ||
| + | *Weather data checked if recent, else fetched. | ||
| + | *Compared with limits and send alerts if the data exceeds limits. | ||
| + | *Transactional data processed | ||
| + | |||
| + | |||
| + | Additional Data Warehouse goals: | ||
*Structure the data so that it makes sense to the researcher. | *Structure the data so that it makes sense to the researcher. | ||
| − | *Structure the data to optimize query performance | + | *Structure the data to optimize query performance. |
*Make research and decision–support queries easier to write. | *Make research and decision–support queries easier to write. | ||
*Maintain data and conversion history. | *Maintain data and conversion history. | ||
| − | *Improve data quality | + | *Improve data quality by flagging and fixing bad data and assigning quality codes and descriptions. |
| − | + | *partitioned into yearly or seasonal periods, | |
| − | + | *summarized (daily weight changes), | |
| − | + | *cataloged and tied into the metadata (foundation type, hive orientation, mite treatment, etc.), | |
| − | + | *tracked and controlled with version control software, | |
| − | + | *released for use by researchers for data mining, online analytical processing, research and decision support. | |
| − | *partitioned into yearly or seasonal periods | ||
| − | * | ||
| − | * | ||
| − | * | ||
| − | * | ||
| − | [[File: | + | [[File:Database_servers-2-2.jpg|thumb 640px|Data Warehouse]] |
Latest revision as of 16:39, 11 June 2017
Since HiveTool is open source/open notebook, the entire primary record is publicly available online as it is recorded. Two databases are used: Input and Analysis.
Each hive sends in data every five minutes, 288 times a day, inserting over 100,000 rows a year into the Input Database. One thousand hives would generate 100 million rows per year. In addition to the measured data, there are external factors that need to be systematically and consistently documented. This metadata includes hive genetics, manipulations, mite treatments, data conversion and calibration formulas, etc.
As the data is moved from the Input Database to the Analysis Database it should:
- Converted to other measurement systems (lb <=> kg, Fahrenheit <=> Celsius).
- Manipulation changes filtered out
- Weather data checked if recent, else fetched.
- Compared with limits and send alerts if the data exceeds limits.
- Transactional data processed
Additional Data Warehouse goals:
- Structure the data so that it makes sense to the researcher.
- Structure the data to optimize query performance.
- Make research and decision–support queries easier to write.
- Maintain data and conversion history.
- Improve data quality by flagging and fixing bad data and assigning quality codes and descriptions.
- partitioned into yearly or seasonal periods,
- summarized (daily weight changes),
- cataloged and tied into the metadata (foundation type, hive orientation, mite treatment, etc.),
- tracked and controlled with version control software,
- released for use by researchers for data mining, online analytical processing, research and decision support.
