Comparison of noaa-stations with module-noaa-stations

This document describes differences between an implementation of similar functionality within a stand-alone command line tool written in Python and a Zimagi module.

Code required

This snapshot is NOAA-stations commits b910e2e and 76a0dae with commit 58d87e6 of module-noaa-stations. Lines of code used and features supported may change in future revisions.

Zimagi module [b] [c]:

module-noaa-stations % cloc .  # 58d87e6
      17 text files.
      17 unique files.
       9 files ignored.

github.com/AlDanial/cloc v 1.86  T=0.02 s (760.2 files/s, 36315.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
YAML                             8             69             66            272
Python                           4             26             11            110 (33)
Markdown                         1             23              0             44
-------------------------------------------------------------------------------
SUM:                            13            118             77            426
-------------------------------------------------------------------------------

Command-line tool [a] [b]:

NOAA-Stations % cloc .  # b910e2e1
       7 text files.
       7 unique files.
       3 files ignored.

github.com/AlDanial/cloc v 1.86  T=0.01 s (789.9 files/s, 30489.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           2             12             34             85
Markdown                         2             21              0             31
YAML                             1              0              0             10
-------------------------------------------------------------------------------
SUM:                             5             33             34            126
-------------------------------------------------------------------------------

Command-line tool (normalized tables) [a] [b]:

NOAA-Stations % cloc .  # 76a0daee
       7 text files.
       7 unique files.
       3 files ignored.

github.com/AlDanial/cloc v 1.86  T=0.01 s (680.8 files/s, 43300.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Markdown                         2             46              0            107
Python                           2             16             44             95
YAML                             1              0              0             10
-------------------------------------------------------------------------------
SUM:                             5             62             44            212
-------------------------------------------------------------------------------

[a] (1,2)

The small YAML file in NOAA-Stations is a conda environment configuration file. It is only indirectly related to the tool itself, but providing the necessary dependencies is reasonable to consider part of tool requirements. A pip requirements.txt would be similar length.

[b] (1,2,3)

The Markdown files in both repositories are entirely documentation and are not directly related to the functionality of either. In the main, the documentation is this file itself.

[c]

Three of the four Python files in module-noaa-stations are auto-generated. The only file written by hand contains 33 code lines (and some comments).

Feature comparisons

Feature description	Zimagi module	Command-line	CL Normalized
Exposes all source data columns	No [f]	Yes	Yes
Download by year range	Partial [d]	Yes	Yes
Download by station list	Partial [d]	Yes	Yes
Download of all stations	No [e]	Yes	Yes
Flexible querying of local DB	Yes	Yes	Yes
RESTful API to access local DB	Yes	No	Yes
Missing data cleaned	Partial	Yes	Yes
Performs good normalization	Yes	No [g]	Yes
Provisions for cloud deployment	Yes	No [h]	No [h]
Supports “pretty” output	Yes	Yes	Yes
Supports CSV export	Yes	Yes	Yes
Supports TSV export	No	Yes	Yes
Supports JSON export	Yes	Yes	Yes
“Code” lines [i]	305	95	105

[d] (1,2)

Only a test import subcommand defined currently, but data model supports parameters for min year, max year, and station list.

[e]

Logic for obtaining station list within year currently stubbed out but should follow identical logic to that used in command-line tool.

[f]

Data definitions could be created for columns not currently utilized. My estimate is that it would require about 150 additional lines of YAML and maybe 20 lines of Python.

[g]

The initial command-line tool simply used the same table structure as the source CSV files. Adding a child table with foreign key would require about 6 extra lines of Python, and 8 extra lines of SQL (which is currently defined as a Python string rather than separate file).

[h] (1,2)

No current code knows about any clouds, but the code that would need to be distributed to one is very minimal.

[i]

YAML or Python code that is functionally required for the system to operate. Documentation in Markdown or other formats is very desirable to have, but does not change functionality. Auto-generated Python code is excluded.