Usage#

Assuming that you’ve followed the installations steps, you’re now ready to use this package.

Please note that this is an early release, and the API may change in future versions.

Examples#

Preview the files to be processed:

flysight2csv source/files/or/directories/
flysight2csv source/files/or/directories/ --info=metadata

Copy just the TRACK.CSV files (unchanged) to a single directory, prepending the date and time to the filename:

flysight2csv source/files/or/directories/ -o output/path/ '--glob=**/TRACK.CSV'

Reformat all the FlySight 2 CSVs into a “flat” CSV (single header) for use with other tools:

flysight2csv source/files/or/directories/ -o output/path/ -f csv-flat

For more options, see the help:

flysight2csv --help

Supported Output Formats#

When the -o (--output-directory) directory is specified, the files will be copied to that directory. The files will be renamed to include the original directory names, and the file can be reformatted using the --format option.

The possible output formats (--format option) are:

  • unchanged (default)

    • NOTE: this disables merging, filtering, etc… and just copies the files

  • csv-flat - a CSV file with a single header row

  • json-lines-minimal - one JSON object per line, with only the fields relevant to each row

  • json-lines-header - one JSON object per line, with a header row at the top of all the fields set to null

  • json-lines-full - one JSON object per line, with the same fields in each object (set to null if not relevant)

Examples of each format are available in the formatted test data folder.

Compatible Tools#

The reformatted files can be analyzed with any tools that support CSV or JSONL file formats, such as:

All Command Line Options#

TODO: describe these options in more detail

Usage: flysight2csv [-h] [–version] [–glob PATTERN [PATTERN …]] [-i {none,path,metadata}] [–field-path-levels INT] [–metadata-only] [–offset-datetime DATETIME] [–continue-on-format-error] [–ignore-all-format-errors] [–ignored-format-errors MESSAGE [MESSAGE …]] [–output-directory PATH] [–output-path-levels INT] [–output-path-separator OUTPUT_PATH_SEPARATOR] [–no-merge] [–only-merge] [–merged-name NAME] [–format {unchanged,csv-flat,json-lines-minimal,json-lines-header,json-lines-full}] [–csv-dialect {excel,excel-tab,unix,flysight2csv}] [–sensors SENSOR [SENSOR …]] [–columns COLUMN [COLUMN …]] [–continue-on-error] [–stop-on-warning] [–log-level {debug,info,warning,error,critical}] [–no-color] [–tracebacks] [–dump-args] [–dump-config] FILE_OR_DIR [FILE_OR_DIR …]

Utility for working with FlySight 2 CSV files.

Options: -h, –help show this help message and exit –version display the version and exit. –dump-args Print parsed arguments and exit. (default: False) –dump-config Print parsed configuration (from args) and exit. (default: False)

File Discovery: FILE_OR_DIR Files or directories to process. –glob, –glob-patterns PATTERN [PATTERN …] Glob patterns to match. (default: [’/*TRACK.CSV’, ‘/*SENSOR.CSV’]) -i, –info, –info-type {none,path,metadata} The type of information to display about each discovered file. (default: path)

Parser Options: –field-path-levels INT The number of path levels to display in the file_path metadata field (added to the data). (default: 3) –metadata-only Stop parsing after the first data row. (default: False) –offset-datetime DATETIME Force this offset datetime instead of auto-detecting from $TIME columns. (default: None) –continue-on-format-error Continue attempting to parse the file even if there are format errors. (default: False) –ignore-all-format-errors Ignore all format errors. (default: False) –ignored-format-errors MESSAGE [MESSAGE …] Ignore these format error messages. (default: None)

Output Path: –output-directory, -o PATH Directory to copy files to. Required for the –format option. (default: None) –output-path-levels INT The number of path names to join into the new file name. (default: 3) –output-path-separator OUTPUT_PATH_SEPARATOR Join directories to file name with this separator. Use / to preserve directory structure. (default: -) –no-merge Do not also merge files from a single directory. Conflicts with –only-merge. (default: True) –only-merge Do not write non-merged files. Conflicts with –no-merge. (default: False) –merged-name NAME The name of the merged file. NOTE: also affected by –output-path-levels. (default: MERGED)

Reformatting: –format, -f {unchanged,csv-flat,json-lines-minimal,json-lines-header,json-lines-full} The output file format. (default: unchanged) –csv-dialect {excel,excel-tab,unix,flysight2csv} CSV dialect. (default: flysight2csv) –sensors SENSOR [SENSOR …] Filter data to just these sensors. (default: None) –columns COLUMN [COLUMN …] Only include these columns. (default: None)

General: –continue-on-error Continue processing files if a file cannot be processed. (default: False) –stop-on-warning Stop processing files if a warning is encountered. (default: False) –log-level {debug,info,warning,error,critical} Minimal log level to display. (default: info) –no-color Disable color output. (default: False) –tracebacks Show exception tracebacks. (default: False)