Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Opencv error and Array dimension error when train cityscapes datasets from scratch #72

Open
liuzhenboo opened this issue Nov 22, 2020 · 0 comments
Open

Opencv error and Array dimension error when train cityscapes datasets from scratch #72

liuzhenboo opened this issue Nov 22, 2020 · 0 comments

Comments

@liuzhenboo
Copy link

Hello,I want to train cityscapes datasets from scratch, But when I run:

developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./log

some opencv errors occurs:

`developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./log
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.

INTERFACE:
data yaml: cfg/cityscapes/data.yaml
net yaml: cfg/cityscapes/net_bonnet.yaml
train yaml: cfg/cityscapes/train_bonnet.yaml
log dir ./log
model path None
model type iou

Commit hash (training version): b'7bf03b0'

Opening desired data file cfg/cityscapes/data.yaml
Opening desired net file cfg/cityscapes/net_bonnet.yaml
Opening desired train file cfg/cityscapes/train_bonnet.yaml
Copying files to ./log for further reference.
Training from scratch
Fetching dataset
Training with 1 GPU's
Training with batch size 2
DEVICE AVAIL: /device:CPU:0
DEVICE AVAIL: /device:GPU:0
Number of GPU's available is 1
This means 2 images per GPU
Data depth: 3
Parsing directory /shared/datasets/cityscapes/bonnet_data/train
Total number of pixels: 6239027200
Total number of pixels of class road: 2036049408
Total number of pixels of class train: 12863940
Total number of pixels of class car: 386502976
Total number of pixels of class sky: 221459584
Total number of pixels of class trafficsign: 30521320
Total number of pixels of class fence: 48487192
Total number of pixels of class sidewalk: 336031712
Total number of pixels of class truck: 14775009
Total number of pixels of class person: 67202392
Total number of pixels of class vegetation: 878734336
Total number of pixels of class pole: 67768880
Total number of pixels of class building: 1259774336
Total number of pixels of class bus: 12995807
Total number of pixels of class rider: 7444910
Total number of pixels of class bicycle: 22849672
Total number of pixels of class terrain: 63965240
Total number of pixels of class trafficlight: 11509943
Total number of pixels of class motorcycle: 5445904
Total number of pixels of class wall: 36211256
Total number of pixels of class crap: 718432320
Content percentage of class road in dataset: 0.326341
Content percentage of class car in dataset: 0.061949
Content percentage of class trafficsign in dataset: 0.004892
Content percentage of class sidewalk in dataset: 0.053860
Content percentage of class truck in dataset: 0.002368
Content percentage of class vegetation in dataset: 0.140845
Content percentage of class building in dataset: 0.201918
Content percentage of class bus in dataset: 0.002083
Content percentage of class terrain in dataset: 0.010252
Content percentage of class wall in dataset: 0.005804
Content percentage of class train in dataset: 0.002062
Content percentage of class sky in dataset: 0.035496
Content percentage of class fence in dataset: 0.007772
Content percentage of class motorcycle in dataset: 0.000873
Content percentage of class person in dataset: 0.010771
Content percentage of class pole in dataset: 0.010862
Content percentage of class bicycle in dataset: 0.003662
Content percentage of class rider in dataset: 0.001193
Content percentage of class trafficlight in dataset: 0.001845
Content percentage of class crap in dataset: 0.115151
Total amount of images: 2975

OpenCV Error: Assertion failed (dims <= 2 && step[0] > 0) in locateROI, file /io/opencv/modules/core/src/matrix.cpp, line 991
Exception in thread ImgBufftrain:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/shared/bonnet/train_py/dataset/abstract_dataset.py", line 84, in run
img, lbl = self.augment(img, lbl)
File "/shared/bonnet/train_py/dataset/abstract_dataset.py", line 68, in augment
img = cv2.blur(img,(ksize,ksize))
cv2.error: /io/opencv/modules/core/src/matrix.cpp:991: error: (-215) dims <= 2 && step[0] > 0 in function locateROI

Parsing directory /shared/datasets/cityscapes/bonnet_data/valid
Total number of pixels: 1048576000
Total number of pixels of class road: 345222080
Total number of pixels of class train: 1032100
Total number of pixels of class car: 59759312
Total number of pixels of class sky: 30708080
Total number of pixels of class trafficsign: 6110454
Total number of pixels of class fence: 7527026
Total number of pixels of class sidewalk: 49559052
Total number of pixels of class truck: 2760469
Total number of pixels of class person: 11890229
Total number of pixels of class vegetation: 158682896
Total number of pixels of class pole: 13564731
Total number of pixels of class building: 200895344
Total number of pixels of class bus: 3564221
Total number of pixels of class rider: 1970543
Total number of pixels of class bicycle: 6500852
Total number of pixels of class terrain: 7625936
Total number of pixels of class trafficlight: 1813749
Total number of pixels of class motorcycle: 728922
Total number of pixels of class wall: 6720678
Total number of pixels of class crap: 131939472
Content percentage of class road in dataset: 0.329229
Content percentage of class car in dataset: 0.056991
Content percentage of class trafficsign in dataset: 0.005827
Content percentage of class sidewalk in dataset: 0.047263
Content percentage of class truck in dataset: 0.002633
Content percentage of class vegetation in dataset: 0.151332
Content percentage of class building in dataset: 0.191589
Content percentage of class bus in dataset: 0.003399
Content percentage of class terrain in dataset: 0.007273
Content percentage of class wall in dataset: 0.006409
Content percentage of class train in dataset: 0.000984
Content percentage of class sky in dataset: 0.029286
Content percentage of class fence in dataset: 0.007178
Content percentage of class motorcycle in dataset: 0.000695
Content percentage of class person in dataset: 0.011339
Content percentage of class pole in dataset: 0.012936
Content percentage of class bicycle in dataset: 0.006200
Content percentage of class rider in dataset: 0.001879
Content percentage of class trafficlight in dataset: 0.001730
Content percentage of class crap in dataset: 0.125827
Total amount of images: 500

SPECIFIC TO CITYSCAPES
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
SPECIFIC TO CITYSCAPES
Parsing directory /shared/datasets/cityscapes/bonnet_data/test
Total number of pixels: 15250000
Total number of pixels of class road: 0
Total number of pixels of class train: 0
Total number of pixels of class car: 0
Total number of pixels of class sky: 0
Total number of pixels of class trafficsign: 0
Total number of pixels of class fence: 0
Total number of pixels of class sidewalk: 0
Total number of pixels of class truck: 0
Total number of pixels of class person: 0
Total number of pixels of class vegetation: 0
Total number of pixels of class pole: 0
Total number of pixels of class building: 0
Total number of pixels of class bus: 0
Total number of pixels of class rider: 0
Total number of pixels of class bicycle: 0
Total number of pixels of class terrain: 0
Total number of pixels of class trafficlight: 0
Total number of pixels of class motorcycle: 0
Total number of pixels of class wall: 0
Total number of pixels of class crap: 15250000
Content percentage of class road in dataset: 0.000000
Content percentage of class car in dataset: 0.000000
Content percentage of class trafficsign in dataset: 0.000000
Content percentage of class sidewalk in dataset: 0.000000
Content percentage of class truck in dataset: 0.000000
Content percentage of class vegetation in dataset: 0.000000
Content percentage of class building in dataset: 0.000000
Content percentage of class bus in dataset: 0.000000
Content percentage of class terrain in dataset: 0.000000
Content percentage of class wall in dataset: 0.000000
Content percentage of class train in dataset: 0.000000
Content percentage of class sky in dataset: 0.000000
Content percentage of class fence in dataset: 0.000000
Content percentage of class motorcycle in dataset: 0.000000
Content percentage of class person in dataset: 0.000000
Content percentage of class pole in dataset: 0.000000
Content percentage of class bicycle in dataset: 0.000000
Content percentage of class rider in dataset: 0.000000
Content percentage of class trafficlight in dataset: 0.000000
Content percentage of class crap in dataset: 1.000000
Total amount of images: 1525

SPECIFIC TO CITYSCAPES
Don't weigh the 'crap' class (key 255)
SPECIFIC TO CITYSCAPES
Successfully imported datasets
Train data samples: 2975
Validation data samples: 500
Test data samples: 1525
Initializing network
TRAINING GRAPH
Building graph
encoder
downsample1
W: [5, 5, 3, 13] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
downsample2
W: [5, 5, 16, 32] Train: True
non-bt-1
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-2
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-3
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-4
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
downsample3
W: [5, 5, 48, 32] Train: True
non-bt-1
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-2
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-3
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-4
W: [7, 1, 80, 80] Train: True
`

`Saving this iteration of training in ./log/lr_0.0001
Training model
Training network 100000 epochs (14875000 iterations at batch size 20)
Decaying learn rate by 1.010000 every 1 epochs (148 steps)
Traceback (most recent call last):

File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped)
developer@linux:/shared/bonnet/train_py$
`

I think it is because a thread is break!
so I comment augment function as below:

`
# if self.name == "ImgBufftrain":

  #   img, lbl = self.augment(img, lbl)

`

But! new error occurs!!!!!!!!!!! some information is below：

`Total number of parameters in network: 1871287

Reporting accuracy every 10 epochs

Saving this iteration of training in ./log/lr_0.0001

Training model
Training network 100000 epochs (14875000 iterations at batch size 20)
Decaying learn rate by 1.010000 every 1 epochs (148 steps)

Traceback (most recent call last):
File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)

ValueError: setting an array element with a sequence.

terminate called without an active exception

terminate called recursively

Aborted (core dumped)
`

a detail error infomation is:

`developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./log
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
中文

INTERFACE:
data yaml: cfg/cityscapes/data.yaml
net yaml: cfg/cityscapes/net_bonnet.yaml
train yaml: cfg/cityscapes/train_bonnet.yaml
log dir ./log
model path None
model type iou

Commit hash (training version): b'7bf03b0'

Opening desired data file cfg/cityscapes/data.yaml
Opening desired net file cfg/cityscapes/net_bonnet.yaml
Opening desired train file cfg/cityscapes/train_bonnet.yaml
Copying files to ./log for further reference.
Training from scratch
Fetching dataset
Training with 1 GPU's
Training with batch size 20
DEVICE AVAIL: /device:CPU:0
DEVICE AVAIL: /device:GPU:0
Number of GPU's available is 1
This means 20 images per GPU
Data depth: 3

Parsing directory /shared/datasets/cityscapes/bonnet_data/train
Total number of pixels: 6239027200
Total number of pixels of class road: 2036049408
Total number of pixels of class train: 12863940
Total number of pixels of class car: 386502976
Total number of pixels of class sky: 221459584
Total number of pixels of class trafficsign: 30521320
Total number of pixels of class fence: 48487192
Total number of pixels of class sidewalk: 336031712
Total number of pixels of class truck: 14775009
Total number of pixels of class person: 67202392
Total number of pixels of class vegetation: 878734336
Total number of pixels of class pole: 67768880
Total number of pixels of class building: 1259774336
Total number of pixels of class bus: 12995807
Total number of pixels of class rider: 7444910
Total number of pixels of class bicycle: 22849672
Total number of pixels of class terrain: 63965240
Total number of pixels of class trafficlight: 11509943
Total number of pixels of class motorcycle: 5445904
Total number of pixels of class wall: 36211256
Total number of pixels of class crap: 718432320
Content percentage of class road in dataset: 0.326341
Content percentage of class car in dataset: 0.061949
Content percentage of class trafficsign in dataset: 0.004892
Content percentage of class sidewalk in dataset: 0.053860
Content percentage of class truck in dataset: 0.002368
Content percentage of class vegetation in dataset: 0.140845
Content percentage of class building in dataset: 0.201918
Content percentage of class bus in dataset: 0.002083
Content percentage of class terrain in dataset: 0.010252
Content percentage of class wall in dataset: 0.005804
Content percentage of class train in dataset: 0.002062
Content percentage of class sky in dataset: 0.035496
Content percentage of class fence in dataset: 0.007772
Content percentage of class motorcycle in dataset: 0.000873
Content percentage of class person in dataset: 0.010771
Content percentage of class pole in dataset: 0.010862
Content percentage of class bicycle in dataset: 0.003662
Content percentage of class rider in dataset: 0.001193
Content percentage of class trafficlight in dataset: 0.001845
Content percentage of class crap in dataset: 0.115151
Total amount of images: 2975

**************************** SPECIFIC TO CITYSCAPES ****************************
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
**************************** SPECIFIC TO CITYSCAPES ****************************
imhfetcher

Parsing directory /shared/datasets/cityscapes/bonnet_data/valid
Total number of pixels: 1048576000
Total number of pixels of class road: 345222080
Total number of pixels of class train: 1032100
Total number of pixels of class car: 59759312
Total number of pixels of class sky: 30708080
Total number of pixels of class trafficsign: 6110454
Total number of pixels of class fence: 7527026
Total number of pixels of class sidewalk: 49559052
Total number of pixels of class truck: 2760469
Total number of pixels of class person: 11890229
Total number of pixels of class vegetation: 158682896
Total number of pixels of class pole: 13564731
Total number of pixels of class building: 200895344
Total number of pixels of class bus: 3564221
Total number of pixels of class rider: 1970543
Total number of pixels of class bicycle: 6500852
Total number of pixels of class terrain: 7625936
Total number of pixels of class trafficlight: 1813749
Total number of pixels of class motorcycle: 728922
Total number of pixels of class wall: 6720678
Total number of pixels of class crap: 131939472
Content percentage of class road in dataset: 0.329229
Content percentage of class car in dataset: 0.056991
Content percentage of class trafficsign in dataset: 0.005827
Content percentage of class sidewalk in dataset: 0.047263
Content percentage of class truck in dataset: 0.002633
Content percentage of class vegetation in dataset: 0.151332
Content percentage of class building in dataset: 0.191589
Content percentage of class bus in dataset: 0.003399
Content percentage of class terrain in dataset: 0.007273
Content percentage of class wall in dataset: 0.006409
Content percentage of class train in dataset: 0.000984
Content percentage of class sky in dataset: 0.029286
Content percentage of class fence in dataset: 0.007178
Content percentage of class motorcycle in dataset: 0.000695
Content percentage of class person in dataset: 0.011339
Content percentage of class pole in dataset: 0.012936
Content percentage of class bicycle in dataset: 0.006200
Content percentage of class rider in dataset: 0.001879
Content percentage of class trafficlight in dataset: 0.001730
Content percentage of class crap in dataset: 0.125827
Total amount of images: 500

**************************** SPECIFIC TO CITYSCAPES ****************************
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
**************************** SPECIFIC TO CITYSCAPES ****************************

Parsing directory /shared/datasets/cityscapes/bonnet_data/test
Total number of pixels: 15250000
Total number of pixels of class road: 0
Total number of pixels of class train: 0
Total number of pixels of class car: 0
Total number of pixels of class sky: 0
Total number of pixels of class trafficsign: 0
Total number of pixels of class fence: 0
Total number of pixels of class sidewalk: 0
Total number of pixels of class truck: 0
Total number of pixels of class person: 0
Total number of pixels of class vegetation: 0
Total number of pixels of class pole: 0
Total number of pixels of class building: 0
Total number of pixels of class bus: 0
Total number of pixels of class rider: 0
Total number of pixels of class bicycle: 0
Total number of pixels of class terrain: 0
Total number of pixels of class trafficlight: 0
Total number of pixels of class motorcycle: 0
Total number of pixels of class wall: 0
Total number of pixels of class crap: 15250000
Content percentage of class road in dataset: 0.000000
Content percentage of class car in dataset: 0.000000
Content percentage of class trafficsign in dataset: 0.000000
Content percentage of class sidewalk in dataset: 0.000000
Content percentage of class truck in dataset: 0.000000
Content percentage of class vegetation in dataset: 0.000000
Content percentage of class building in dataset: 0.000000
Content percentage of class bus in dataset: 0.000000
Content percentage of class terrain in dataset: 0.000000
Content percentage of class wall in dataset: 0.000000
Content percentage of class train in dataset: 0.000000
Content percentage of class sky in dataset: 0.000000
Content percentage of class fence in dataset: 0.000000
Content percentage of class motorcycle in dataset: 0.000000
Content percentage of class person in dataset: 0.000000
Content percentage of class pole in dataset: 0.000000
Content percentage of class bicycle in dataset: 0.000000
Content percentage of class rider in dataset: 0.000000
Content percentage of class trafficlight in dataset: 0.000000
Content percentage of class crap in dataset: 1.000000
Total amount of images: 1525

**************************** SPECIFIC TO CITYSCAPES ****************************
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
**************************** SPECIFIC TO CITYSCAPES ****************************
Successfully imported datasets
Train data samples: 2975
Validation data samples: 500
Test data samples: 1525
Initializing network
******************************** TRAINING GRAPH ********************************
GRAPH GPU:0
Building graph
encoder
downsample1
W: [5, 5, 3, 13] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
downsample2
W: [5, 5, 16, 32] Train: True
non-bt-1
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-2
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-3
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-4
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
downsample3
W: [5, 5, 48, 32] Train: True
non-bt-1
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-2
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-3
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-4
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
godeep
non-bt-1
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-2
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-3
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-4
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
============= End of encoder ===============
size of code: [20, 80, 48, 96]
=========== Beginning of decoder============
decoder
upsample
unpool1
W: [2, 2, 80, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
unpool2
W: [2, 2, 48, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
unpool3
W: [2, 2, 32, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [1, 1, 16, 20] Train: True
b: [20] Train: True
Defining loss function

Weights for loss function (1/log(frec(c)+e)):
[ 3.36258268 14.03326797 4.98947144 39.25167084 36.5057373 32.89967728
46.27554703 40.67154694 6.70474911 33.55268478 18.51486206 32.99533463
47.68305206 12.69611359 45.20449448 45.78203583 45.8253479 48.40744781
42.75931931 0. ]
Weight decay: 1e-06
Using tensorflow gradients

******************************** TESTING GRAPH *********************************
GRAPH GPU:0
Building graph
encoder
downsample1
W: [5, 5, 3, 13] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
downsample2
W: [5, 5, 16, 32] Train: False
non-bt-1
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
non-bt-2
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
non-bt-3
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
non-bt-4
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
downsample3
W: [5, 5, 48, 32] Train: False
non-bt-1
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-2
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-3
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-4
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
godeep
non-bt-1
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-2
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-3
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-4
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
============= End of encoder ===============
size of code: [20, 80, 48, 96]
=========== Beginning of decoder============
decoder
upsample
unpool1
W: [2, 2, 80, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
unpool2
W: [2, 2, 48, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
unpool3
W: [2, 2, 32, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [1, 1, 16, 20] Train: False
b: [20] Train: False

Total number of parameters in network: 1871287
Reporting accuracy every 10 epochs
Saving this iteration of training in ./log/lr_0.0001
Training model
Training network 100000 epochs (14875000 iterations at batch size 20)
Decaying learn rate by 1.010000 every 1 epochs (148 steps)
Traceback (most recent call last):
File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped)`

I hope you can help me, sincerely！
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@liuzhenboo