Skip to content

Commit

Permalink
改写
Browse files Browse the repository at this point in the history
  • Loading branch information
nigo81 committed Mar 27, 2022
1 parent 26b9afc commit 2c7559c
Show file tree
Hide file tree
Showing 7 changed files with 201 additions and 57 deletions.
60 changes: 58 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,60 @@
# import-csv-to-mysql
takes csv data and write into mysql
import csv or txt files to mysql.

[中文说明](./README_CN.md)

[视频介绍](https://www.bilibili.com/video/BV1HS4y1m748/)

## how to install

clone the project:

```bash
git clone https://github.com/nigo81/import-csv-to-mysql.git
```

prompt the `csv2mysql` execute privileges

```bash
chmod +x csv2mysql
```

cp `csv2mysql` to `~/.local/bin`:

```bash
cp csv2mysql ~/.local/bin
```

at first you shoule install `fzf` which is a file search tool in linux.

## how to use

change to the directory where your csv files are, and execute `csv2mysql`


![fzf](pictures/fzf.png)

your can use `fzf` to search file, type `tab` to select your data files.

finished with `enter`.

![shell](pictures/shell.png)

```bash
   ~/tmp  csv2mysql
input some database parameters

Enter your host ip,(default is 127.0.0.1)🔗:
Enter your database name 📚:book
Enter your account name 🥷:root
Enter password 🔑:\n
table name 📑:vip
do your wan't to drop table if exists? 0:don't drop, 1:drop :1
data ignore lines num,usually as 0 or 1:1
input character set(default is utf8):
```

your should input your mysql datase info, such as `ip`,`database name`,`account`,`password`.

you should input `table name`, which the table your data will store in.(it will auto create table,if not exists)

not finished
59 changes: 59 additions & 0 deletions README_CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# import-csv-to-mysql

批量导入csv文件或者txt文件到mysql数据库。

[视频介绍](https://www.bilibili.com/video/BV1HS4y1m748/)

## 如何安装

克隆本项目:
```bash
git clone https://github.com/nigo81/import-csv-to-mysql.git
```

`csv2mysql`文件添加可执行权限:

```bash
chmod +x csv2mysql
```

`csv2mysql`复制到`~/.local/bin`:

```bash
cp csv2mysql ~/.local/bin
```

你需要安装`fzf`搜索工具,因为脚本里调用它来选择文件。

## 如何使用

进入你csv文件所在目录,然后执行`csv2mysql`


![fzf](pictures/fzf.png)

你可以使用`fzf`的搜索功能,按下`tab`键可以选中文件。

选择完成后,按下回车确认。

![shell](pictures/shell.png)

```bash
   ~/tmp  csv2mysql
input some database parameters

Enter your host ip,(default is 127.0.0.1)🔗:
Enter your database name 📚:book
Enter your account name 🥷:root
Enter password 🔑:\n
table name 📑:vip
do your wan't to drop table if exists? 0:don't drop, 1:drop :1
data ignore lines num,usually as 0 or 1:1
input character set(default is utf8):
```
你需要填写mysql数据库的一些信息,如ip,数据库名称,账号,密码。

你需要在`table name`处输入你将导入的数据保存的表的名称,如果数据库中没有该表,会自动创建。



64 changes: 64 additions & 0 deletions csv2mysql
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#!/bin/sh

# show commands being executed, per debug
#set -x

_csv_files=$(ls $(pwd)/*.{csv,txt} 2>/dev/null | fzf -m)
[[ $? -eq 0 ]] || { echo "you must chose some csv or txt files";exit 1; }
# define database connectivity
echo -e "input some database parameters\n"
read -p "Enter your host ip,(default is 127.0.0.1)🔗:" _db_host
if [ ! -n "$_db_host" ];then
_db_host="127.0.0.1"
fi
read -p "Enter your database name 📚:" _db
read -p "Enter your account name 🥷:" _db_user
read -p "Enter password 🔑:" -s _db_password
echo '\n'
read -p "table name 📑:" _table_name
read -p "do your wan't to drop table if exists? 0:don't drop, 1:drop :" _drop_table
# read file start num
read -p "data ignore lines num,usually as 0 or 1:" _ignore_line
read -p "input character set(default is utf8):" _character_set
if [ ! -n "$_character_set" ];then
_character_set="utf8"
fi
# loop through csv files
i=1
for _csv_file in $_csv_files ;
do
if [ $i -eq 1 ];then
# get header columns from CSV file
_header_columns=(`head -1 $_csv_file | tr ',' '\n' | sed 's/^"//' | sed 's/"$//' | sed 's/ /_/g'`)
_header_columns_string=`head -1 $_csv_file | sed 's/ /_/g' | sed 's/"//g'`
# ensure table exists
sql_drop_table="drop table if exists \`$_table_name\`;"
if [ $_drop_table -eq 1 ];then
sql_create_table="${sql_drop_table}create table \`$_table_name\` \n (\n"
else
sql_create_table="create table \`$_table_name\` \n (\n"
fi
## loop through header columns
length=${#_header_columns[@]}
for i in ${!_header_columns[@]};do
if [ "$i" -ne $[$length-1] ];then
sql_create_table="$sql_create_table \`${_header_columns[i]}\` varchar(255),\n"
else
sql_create_table="$sql_create_table \`${_header_columns[i]}\` varchar(255)\n"
fi
done
sql_create_table="$sql_create_table);"

mysql -h$_db_host -u$_db_user -p$_db_password $_db --execute="$sql_create_table" >/dev/null
fi
echo "开始导入表$_csv_file"
mysql -h$_db_host -u$_db_user -p$_db_password $_db << eof
set global local_infile = 1;
load data local infile "$_csv_file"
into table \`$_table_name\`
character set $_character_set
fields terminated by ',' optionally enclosed by '"'
lines terminated by '\n' ignore $_ignore_line lines;
eof
done
exit
20 changes: 20 additions & 0 deletions excel2csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/bin/sh
files=$(ls *.xls?)
for i in $files;
do
filename="${i%.*}"
extention="${i#*.}"
outext=".csv"
if [ "$extention" == "xlsx" ];then
xlsx2csv "$i" > "$filename$outext"
elif [ "$extention" == "xls" ];then
xls2csv "$i" > "$filename$outext"
else
echo "${i}不是xlsx或xls文件"
fi
if [ $? -ne 0 ];then
echo "${i}转换失败"
else
echo "${i}转换成功"
fi
done
55 changes: 0 additions & 55 deletions mysql import

This file was deleted.

Binary file added pictures/fzf.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added pictures/shell.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 2c7559c

Please sign in to comment.