File upload via portal

[jolabrenz]

Hello I am Johannes from Kiel,

since we have this discussion place here I would like to report about a problem I have with file uploads via the portal.
I wanted to increase the upload limit for the portal from 100MB to 300MB and ran into the problem that for files greater than 132120551 Byte the upload fails with message "FAILED".
Davrods and iCommands do not show this, just the portal.

The rodsLog shows me the error:
Authenticated
Nov 30 11:49:49 pid:17949 NOTICE: writeLine: inString = Agent process started for puser=johannes and cuser=johannes from 10.0.2.116
Nov 30 11:49:49 pid:17949 remote addresses: 10.0.2.116, 10.0.2.84 ERROR: [-] /irods/server/core/src/rsApiHandler.cpp:540:int
readAndProcClientMsg(rsComm_t *, int) : status [SYS_HEADER_READ_LEN_ERR] errno [] -- message [failed
to call 'read header']
[-] /irods/lib/core/src/sockComm.cpp:201:irods::error readMsgHeader(irods::network_object_ptr, msgHeader_t *, struct timeval
*) : status [SYS_HEADER_READ_LEN_ERR] errno [] -- message [failed to call 'read header']
[-] /irods/plugins/network/tcp/libtcp.cpp:194:irods::error tcp_read_msg_header(irods::plugin_context &, void *, struct
timeval *) : status [SYS_HEADER_READ_LEN_ERR] errno [] -- message [only read [0] of [4]]

For files smaller than 132120551 Byte the upload works fine.

I looked into the file libtcp.cpp and found that the function "tcp_socket_read" returns (bytes_read != sizeof(header_length)) and thus gives out an error.
https://github.com/irods/irods/blob/abb809040485326855ffffbda1883dc99a432379/plugins/network/tcp/libtcp.cpp#L193

Since the number of 132120551 doesn't ring any bells and I haven't any clue why this happens for certain file sizes , I was wondering if anyone else have seen this behaviour and if yes has a fix for that.

Best regards
Johannes

[lwesterhof]

Hi Johannes,

To increase to upload limit in v1.7.x (v1.7.2 already has an increased upload limit of 300MB) changes are needed in several places. In the portal (UtrechtUniversity/yoda-portal-research@c094e6f ) and in the PHP.ini configuration (72a79b6) We have seen similar behaviour when the changes are made in the portal, but not in the PHP.ini configuration.

In the upcoming v1.8 release we completely reimplemented the portal upload functionality. The new upload does not have an upload limit and should have the same behaviour as Davrods and the iCommands.

[jolabrenz]

Hi Lazlo,

thanks for the answer. The limit was already set to 300MB at all places you mentioned. I will set it back to 100MB so the user is not wondering about it and the wait for the next release. Thanks for all the work.

Nevertheless I'm still super curious why it fails for greater than this exact file size. For failed uploads with sizes below 300MB there are no entries in the logs of the webserver.
The error in the rodsLog of the ICAT tells me that the function tcp_socket_read in the libtcp.cpp is running but returns bytes_read=0.
https://github.com/irods/irods/blob/abb809040485326855ffffbda1883dc99a432379/plugins/network/tcp/libtcp.cpp#L23

Thanks again and best regards
Johannes

[jolabrenz]

Hi Lazlo,

I found the reason for the mysterious behavior ....
The memory_limit value in the php.ini was still set to 128M.
After setting it to 300M everything was as expected.

Thanks for all you work again
Johannes