Skip to content

Commit

Permalink
Merge pull request #1 from Bugazelle/dev
Browse files Browse the repository at this point in the history
[Feat] 0.1.6
  • Loading branch information
Bugazelle authored Jul 12, 2019
2 parents 17aa131 + 404d307 commit 379f774
Show file tree
Hide file tree
Showing 4 changed files with 72 additions and 20 deletions.
62 changes: 49 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,68 +1,103 @@
Export CSV To Influx
====================

> Version 0.1.0
**Export CSV To Influx**: Process CSV data, and export the data to influx db

## Install

Use the pip to install the library. Then the binary **export_csv_to_influx** is ready.

```
pip install ExportCsvToInflux
```

## Features

1. Allow to use binary **export_csv_to_influx** to run exporter

2. Allow to check dozens of csv files in a folder

3. Auto convert csv data to int/float/string in Influx

4. Allow to limit string length in Influx

5. Allow to judge the csv has new data or not

6. Allow to use the latest file modify time as time column

7. Auto Create database if not exist

8. Allow to drop database before inserting data

9. Allow to drop measurements before inserting data

10. Allow to match or filter the data by using string or regex.

11. Allow to count, and generate count measurement

## Command Arguments

You could use `export_csv_to_influx -h` to see the help guide.

Here are the details:
```bash
-c, --csv, Input CSV file path, or the folder path. **Mandatory**

-d, --delimiter, CSV delimiter. Default: ','.

```
-c, --csv, Input CSV file path, or the folder path. **Mandotory**
-d, --delimiter, CSV delimiter. Default: ','.
-lt, --lineterminator, CSV lineterminator. Default: '\n'.
-s, --server, InfluxDB Server address. Default: localhost:8086
-u, --user, InfluxDB User name. **Mandotory**
-p, --password, InfluxDB Password. **Mandotory**
-db, --dbname, InfluxDB Database name. **Mandotory**
-m, --measurement, Metric column name. **Mandotory**

-s, --server, InfluxDB Server address. Default: localhost:8086.

-u, --user, InfluxDB User name.

-p, --password, InfluxDB Password.

-db, --dbname, InfluxDB Database name. **Mandatory**

-m, --measurement, Measurement name. **Mandatory**

-t, --time_column, Timestamp column name. Default: timestamp. If no timestamp column, the timestamp is set to the last file modify time for whole csv rows.

-tf, --time_format, Timestamp format. Default: '%Y-%m-%d %H:%M:%S' e.g.: 1970-01-01 00:00:00.

-tz, --time_zone, Timezone of supplied data. Default: UTC.
-fc, --field_columns, List of csv columns to use as fields, separated by comma. **Mandotory**
-tc, --tag_columns, List of csv columns to use as tags, separated by comma. **Mandotory**

-fc, --field_columns, List of csv columns to use as fields, separated by comma. **Mandatory**

-tc, --tag_columns, List of csv columns to use as tags, separated by comma. **Mandatory**

-b, --batch_size, Batch size when inserting data to influx. Default: 500.

-lslc, --limit_string_length_columns, Limit string length column. Default: None.

-ls, --limit_length, Limit length. Default: 20.

-dd, --drop_database, Drop database before inserting data.

-dm, --drop_measurement, Drop measurement before inserting data.

-mc, --match_columns, Match the data you want to get for certain columns, separated by comma.

-mbs, --match_by_string, Match by string, separated by comma.

-mbr, --match_by_regex, Match by regex, separated by comma.

-fic, --filter_columns, Filter the data you want to filter for certain columns, separated by comma.

-fibs, --filter_by_string, Filter by string, separated by comma.

-fibr, --filter_by_regex, Filter by regex, separated by comma.

-ecm, --enable_count_measurement, Enable count measurement.

-fi, --force_insert_even_csv_no_update, Force insert data to influx, even csv no update.
```

> **Note 1:** You could use the library programmablly.
```
from ExportCsvToInflux import ExporterObject
exporter = ExporterObject()
exporter.export_csv_to_influx(...)
```
Expand Down Expand Up @@ -151,4 +186,5 @@ timestamp,url,response_time
```
## Special Thanks
The lib is inspired by: [https://github.com/fabio-miranda/csv-to-influxdb](https://github.com/fabio-miranda/csv-to-influxdb)
26 changes: 21 additions & 5 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,37 +3,53 @@
import re

CURDIR = os.path.dirname(os.path.abspath(__file__))
url = 'https://github.com/Bugazelle/export-csv-to-inlfux'

with open(os.path.join(CURDIR, 'requirements.txt')) as f:
REQUIRES = f.read().splitlines()

with open(os.path.join(CURDIR, 'src', 'ExportCsvToInflux', '__version__.py')) as f:
VERSION = re.search("__version__ = '(.*)'", f.read()).group(1)
download_url = '{0}/archive/v{1}.tar.gz'.format(url, VERSION)

this_directory = os.path.abspath(os.path.dirname(__file__))
with open(os.path.join(this_directory, 'README.md')) as f:
long_description = f.read()
index = long_description.find('''```\n\n> **Note''')
new_long_description = long_description[:index].replace('```bash', '')\
.replace('## Install', '**Install**')\
.replace('## Features', '**Features**')\
.replace('## Command Arguments', '**Command Arguments**')
suffix = '\nFor more info, please refer to the {0}'.format(url)
long_description = new_long_description + suffix

setup(
name='ExportCsvToInflux',
package_dir={'': 'src'},
packages=find_packages('src'),
version=VERSION,
license='bsd-3-clause-clear',
description='Export',
zip_safe=False,
include_package_data=True,
description='ExportCsvToInflux: A Solution to export csv to influx db',
long_description=long_description,
long_description_content_type='text/markdown',
author='Bugazelle',
author_email='[email protected]',
keywords=['python', 'csv', 'influx'],
install_requires=REQUIRES,
download_url='',
url='https://github.com/Bugazelle/export-csv-to-inlfux',
download_url=download_url,
url=url,
classifiers=(
'Development Status :: Production/Stable',
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'Natural Language :: English',
'Programming Language :: Python',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 237',
'Programming Language :: Python :: 3.7',
),
entry_points={
'console_scripts': [
Expand Down
2 changes: 1 addition & 1 deletion src/ExportCsvToInflux/__version__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '0.1.0'
__version__ = '0.1.6'
2 changes: 1 addition & 1 deletion src/ExportCsvToInflux/exporter_object.py
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,7 @@ def export_csv_to_influx():
parser.add_argument('-db', '--dbname', nargs='?', required=True,
help='InfluxDB Database name.')
parser.add_argument('-m', '--measurement', nargs='?', required=True,
help='Metric column name.')
help='Measurement name.')
parser.add_argument('-t', '--time_column', nargs='?', default='timestamp',
help='Timestamp column name. Default: timestamp. '
'If no timestamp column, '
Expand Down

0 comments on commit 379f774

Please sign in to comment.