forked from Kaskere/import-csv-to-mysql
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
7 changed files
with
201 additions
and
57 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,60 @@ | ||
# import-csv-to-mysql | ||
takes csv data and write into mysql | ||
import csv or txt files to mysql. | ||
|
||
[中文说明](./README_CN.md) | ||
|
||
[视频介绍](https://www.bilibili.com/video/BV1HS4y1m748/) | ||
|
||
## how to install | ||
|
||
clone the project: | ||
|
||
```bash | ||
git clone https://github.com/nigo81/import-csv-to-mysql.git | ||
``` | ||
|
||
prompt the `csv2mysql` execute privileges | ||
|
||
```bash | ||
chmod +x csv2mysql | ||
``` | ||
|
||
cp `csv2mysql` to `~/.local/bin`: | ||
|
||
```bash | ||
cp csv2mysql ~/.local/bin | ||
``` | ||
|
||
at first you shoule install `fzf` which is a file search tool in linux. | ||
|
||
## how to use | ||
|
||
change to the directory where your csv files are, and execute `csv2mysql` | ||
|
||
|
||
![fzf](pictures/fzf.png) | ||
|
||
your can use `fzf` to search file, type `tab` to select your data files. | ||
|
||
finished with `enter`. | ||
|
||
![shell](pictures/shell.png) | ||
|
||
```bash | ||
~/tmp csv2mysql | ||
input some database parameters | ||
|
||
Enter your host ip,(default is 127.0.0.1)🔗: | ||
Enter your database name 📚:book | ||
Enter your account name 🥷:root | ||
Enter password 🔑:\n | ||
table name 📑:vip | ||
do your wan't to drop table if exists? 0:don't drop, 1:drop :1 | ||
data ignore lines num,usually as 0 or 1:1 | ||
input character set(default is utf8): | ||
``` | ||
|
||
your should input your mysql datase info, such as `ip`,`database name`,`account`,`password`. | ||
|
||
you should input `table name`, which the table your data will store in.(it will auto create table,if not exists) | ||
|
||
not finished |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
# import-csv-to-mysql | ||
|
||
批量导入csv文件或者txt文件到mysql数据库。 | ||
|
||
[视频介绍](https://www.bilibili.com/video/BV1HS4y1m748/) | ||
|
||
## 如何安装 | ||
|
||
克隆本项目: | ||
```bash | ||
git clone https://github.com/nigo81/import-csv-to-mysql.git | ||
``` | ||
|
||
给`csv2mysql`文件添加可执行权限: | ||
|
||
```bash | ||
chmod +x csv2mysql | ||
``` | ||
|
||
将`csv2mysql`复制到`~/.local/bin`: | ||
|
||
```bash | ||
cp csv2mysql ~/.local/bin | ||
``` | ||
|
||
你需要安装`fzf`搜索工具,因为脚本里调用它来选择文件。 | ||
|
||
## 如何使用 | ||
|
||
进入你csv文件所在目录,然后执行`csv2mysql` | ||
|
||
|
||
![fzf](pictures/fzf.png) | ||
|
||
你可以使用`fzf`的搜索功能,按下`tab`键可以选中文件。 | ||
|
||
选择完成后,按下回车确认。 | ||
|
||
![shell](pictures/shell.png) | ||
|
||
```bash | ||
~/tmp csv2mysql | ||
input some database parameters | ||
|
||
Enter your host ip,(default is 127.0.0.1)🔗: | ||
Enter your database name 📚:book | ||
Enter your account name 🥷:root | ||
Enter password 🔑:\n | ||
table name 📑:vip | ||
do your wan't to drop table if exists? 0:don't drop, 1:drop :1 | ||
data ignore lines num,usually as 0 or 1:1 | ||
input character set(default is utf8): | ||
``` | ||
你需要填写mysql数据库的一些信息,如ip,数据库名称,账号,密码。 | ||
|
||
你需要在`table name`处输入你将导入的数据保存的表的名称,如果数据库中没有该表,会自动创建。 | ||
|
||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
#!/bin/sh | ||
|
||
# show commands being executed, per debug | ||
#set -x | ||
|
||
_csv_files=$(ls $(pwd)/*.{csv,txt} 2>/dev/null | fzf -m) | ||
[[ $? -eq 0 ]] || { echo "you must chose some csv or txt files";exit 1; } | ||
# define database connectivity | ||
echo -e "input some database parameters\n" | ||
read -p "Enter your host ip,(default is 127.0.0.1)🔗:" _db_host | ||
if [ ! -n "$_db_host" ];then | ||
_db_host="127.0.0.1" | ||
fi | ||
read -p "Enter your database name 📚:" _db | ||
read -p "Enter your account name 🥷:" _db_user | ||
read -p "Enter password 🔑:" -s _db_password | ||
echo '\n' | ||
read -p "table name 📑:" _table_name | ||
read -p "do your wan't to drop table if exists? 0:don't drop, 1:drop :" _drop_table | ||
# read file start num | ||
read -p "data ignore lines num,usually as 0 or 1:" _ignore_line | ||
read -p "input character set(default is utf8):" _character_set | ||
if [ ! -n "$_character_set" ];then | ||
_character_set="utf8" | ||
fi | ||
# loop through csv files | ||
i=1 | ||
for _csv_file in $_csv_files ; | ||
do | ||
if [ $i -eq 1 ];then | ||
# get header columns from CSV file | ||
_header_columns=(`head -1 $_csv_file | tr ',' '\n' | sed 's/^"//' | sed 's/"$//' | sed 's/ /_/g'`) | ||
_header_columns_string=`head -1 $_csv_file | sed 's/ /_/g' | sed 's/"//g'` | ||
# ensure table exists | ||
sql_drop_table="drop table if exists \`$_table_name\`;" | ||
if [ $_drop_table -eq 1 ];then | ||
sql_create_table="${sql_drop_table}create table \`$_table_name\` \n (\n" | ||
else | ||
sql_create_table="create table \`$_table_name\` \n (\n" | ||
fi | ||
## loop through header columns | ||
length=${#_header_columns[@]} | ||
for i in ${!_header_columns[@]};do | ||
if [ "$i" -ne $[$length-1] ];then | ||
sql_create_table="$sql_create_table \`${_header_columns[i]}\` varchar(255),\n" | ||
else | ||
sql_create_table="$sql_create_table \`${_header_columns[i]}\` varchar(255)\n" | ||
fi | ||
done | ||
sql_create_table="$sql_create_table);" | ||
|
||
mysql -h$_db_host -u$_db_user -p$_db_password $_db --execute="$sql_create_table" >/dev/null | ||
fi | ||
echo "开始导入表$_csv_file" | ||
mysql -h$_db_host -u$_db_user -p$_db_password $_db << eof | ||
set global local_infile = 1; | ||
load data local infile "$_csv_file" | ||
into table \`$_table_name\` | ||
character set $_character_set | ||
fields terminated by ',' optionally enclosed by '"' | ||
lines terminated by '\n' ignore $_ignore_line lines; | ||
eof | ||
done | ||
exit |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
#!/bin/sh | ||
files=$(ls *.xls?) | ||
for i in $files; | ||
do | ||
filename="${i%.*}" | ||
extention="${i#*.}" | ||
outext=".csv" | ||
if [ "$extention" == "xlsx" ];then | ||
xlsx2csv "$i" > "$filename$outext" | ||
elif [ "$extention" == "xls" ];then | ||
xls2csv "$i" > "$filename$outext" | ||
else | ||
echo "${i}不是xlsx或xls文件" | ||
fi | ||
if [ $? -ne 0 ];then | ||
echo "${i}转换失败" | ||
else | ||
echo "${i}转换成功" | ||
fi | ||
done |
This file was deleted.
Oops, something went wrong.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.