Squashed commit of the following

commit 18f198a3 
Author: gregmagdits <[email protected]> 
Date: Thu Apr 11 2024 11:33:46 GMT-0400 (Eastern Daylight Time) 

    version bump and added to changelog for release


commit ffd5c681 
Author: gregmagdits <[email protected]> 
Date: Thu Mar 07 2024 12:30:09 GMT-0500 (Eastern Standard Time) 

    version bump


commit e3278647 
Author: [email protected] <[email protected]> 
Date: Thu Mar 07 2024 11:58:19 GMT-0500 (Eastern Standard Time) 

    Conflicts resolved in the console.

commit 77a6278c 
Author: Roger Lee <[email protected]> 
Date: Thu Mar 07 2024 09:48:18 GMT-0500 (Eastern Standard Time) 

    Squashed commit of the following 

commit 0e0f563b 
Author: Roger Lee <[email protected]> 
Date: Wed Mar 06 2024 13:38:16 GMT-0500 (Eastern Standard Time) 

    URL encode all but drive name in the Windows directories/filenames



commit 6b93f414 
Author: gregmagdits <[email protected]> 
Date: Wed Mar 06 2024 16:22:18 GMT-0500 (Eastern Standard Time) 

    python linting


commit 33d4201a 
Author: gregmagdits <[email protected]> 
Date: Wed Mar 06 2024 10:37:38 GMT-0500 (Eastern Standard Time) 

    Merge branch 'release/version_2_27' into release/REV-643-v0-2-27-release-for-nda-tools

# Conflicts:
#	NDATools/BuildPackage.py
#	NDATools/__init__.py
#	NDATools/clientscripts/downloadcmd.py
#	NDATools/clientscripts/vtcmd.py


commit 613fbc28 
Author: gregmagdits <[email protected]> 
Date: Wed Mar 06 2024 15:50:40 GMT-0500 (Eastern Standard Time) 

    started code cleanup


commit 0e476f74 
Author: Roger Lee <[email protected]> 
Date: Wed Feb 28 2024 11:02:42 GMT-0500 (Eastern Standard Time) 

    Squashed commit of the following 

commit 14ecf81c 
Author: [email protected] <[email protected]> 
Date: Wed Feb 28 2024 10:32:40 GMT-0500 (Eastern Standard Time) 

    Conflicts resolved in the console.

commit 576593d5 
Author: Roger Lee <[email protected]> 
Date: Wed Feb 28 2024 10:11:17 GMT-0500 (Eastern Standard Time) 

    Lowercase username flag


commit 8e85d983 
Author: Roger Lee <[email protected]> 
Date: Tue Feb 27 2024 16:42:28 GMT-0500 (Eastern Standard Time) 

    Increment version


commit 650e1217 
Author: Roger Lee <[email protected]> 
Date: Tue Feb 27 2024 16:41:54 GMT-0500 (Eastern Standard Time) 

    Lowercase the username prompt if username in settings.cfg is empty


commit 3a146b76 
Author: Roger Lee <[email protected]> 
Date: Mon Feb 26 2024 17:11:00 GMT-0500 (Eastern Standard Time) 

    Correct version


commit 3957a99d 
Author: Roger Lee <[email protected]> 
Date: Mon Feb 26 2024 17:09:55 GMT-0500 (Eastern Standard Time) 

    Correct version


commit b4c3f8ae 
Author: [email protected] <[email protected]> 
Date: Mon Feb 26 2024 17:05:39 GMT-0500 (Eastern Standard Time) 

    Conflicts resolved in the console.

commit 2b1539f8 
Author: Roger Lee <[email protected]> 
Date: Mon Feb 26 2024 16:30:16 GMT-0500 (Eastern Standard Time) 

    Set username to lower when reading settings.cfg. Update changelog. Increment version



commit 18ad2b99 
Author: gregmagdits <[email protected]> 
Date: Thu Dec 28 2023 11:48:17 GMT-0500 (Eastern Standard Time) 

    fix for resubmission workflow bug


commit 6add169f 
Author: gregmagdits <[email protected]> 
Date: Wed Mar 06 2024 15:04:44 GMT-0500 (Eastern Standard Time) 

    fixed typo


commit f6c0a476 
Author: Roger Lee <[email protected]> 
Date: Wed Feb 28 2024 09:23:24 GMT-0500 (Eastern Standard Time) 

    Squashed commit of the following 

commit 84ef1fbd 
Author: [email protected] <[email protected]> 
Date: Tue Feb 27 2024 16:43:49 GMT-0500 (Eastern Standard Time) 

    Conflicts resolved in the console.

commit 8e85d983 
Author: Roger Lee <[email protected]> 
Date: Tue Feb 27 2024 16:42:28 GMT-0500 (Eastern Standard Time) 

    Increment version


commit 650e1217 
Author: Roger Lee <[email protected]> 
Date: Tue Feb 27 2024 16:41:54 GMT-0500 (Eastern Standard Time) 

    Lowercase the username prompt if username in settings.cfg is empty


commit 3a146b76 
Author: Roger Lee <[email protected]> 
Date: Mon Feb 26 2024 17:11:00 GMT-0500 (Eastern Standard Time) 

    Correct version


commit 3957a99d 
Author: Roger Lee <[email protected]> 
Date: Mon Feb 26 2024 17:09:55 GMT-0500 (Eastern Standard Time) 

    Correct version


commit b4c3f8ae 
Author: [email protected] <[email protected]> 
Date: Mon Feb 26 2024 17:05:39 GMT-0500 (Eastern Standard Time) 

    Conflicts resolved in the console.

commit 2b1539f8 
Author: Roger Lee <[email protected]> 
Date: Mon Feb 26 2024 16:30:16 GMT-0500 (Eastern Standard Time) 

    Set username to lower when reading settings.cfg. Update changelog. Increment version



commit 1ba9987d 
Author: gregmagdits <[email protected]> 
Date: Thu Dec 14 2023 12:36:45 GMT-0500 (Eastern Standard Time) 

    Merge branch 'feature/REV-640-nda-tools-python-remove-alt-endpoint-as-option-for-upload' into release/version_2_26


commit a9556a91 
Author: gregmagdits <[email protected]> 
Date: Wed Mar 06 2024 14:29:18 GMT-0500 (Eastern Standard Time) 

    fixed bugs found during testing


commit 8b60cfab 
Author: Nancy Schroepfer <[email protected]> 
Date: Tue Feb 27 2024 16:15:17 GMT-0500 (Eastern Standard Time) 

    REV-178 Allow log location configuration



commit fe0aaf1a 
Author: gregmagdits <[email protected]> 
Date: Thu Dec 14 2023 12:27:58 GMT-0500 (Eastern Standard Time) 

    Merge branch 'main' of ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/nda-tools-python into release/version_2_26

# Conflicts:
#	NDATools/__init__.py
#	buildspec.yml


commit 6888f34 
Author: gregmagdits <[email protected]> 
Date: Thu Nov 30 2023 17:24:19 GMT-0500 (Eastern Standard Time) 

    fixed additional errors found during testing


commit 37c72654 
Author: gregmagdits <[email protected]> 
Date: Wed Mar 06 2024 10:54:41 GMT-0500 (Eastern Standard Time) 

    Merge branch 'release/REV-643-v0-2-27-release-for-nda-tools' into hd/REV-688-504-pending-changes-xxl

# Conflicts:
#	NDATools/Submission.py
#	NDATools/Utils.py
#	NDATools/__init__.py
#	NDATools/clientscripts/vtcmd.py


commit 99a896b8 
Author: Roger Lee <[email protected]> 
Date: Mon Feb 26 2024 18:31:24 GMT-0500 (Eastern Standard Time) 

    Squashed commit of the following 

commit 3a146b76 
Author: Roger Lee <[email protected]> 
Date: Mon Feb 26 2024 17:11:00 GMT-0500 (Eastern Standard Time) 

    Correct version


commit 3957a99d 
Author: Roger Lee <[email protected]> 
Date: Mon Feb 26 2024 17:09:55 GMT-0500 (Eastern Standard Time) 

    Correct version


commit b4c3f8ae 
Author: [email protected] <[email protected]> 
Date: Mon Feb 26 2024 17:05:39 GMT-0500 (Eastern Standard Time) 

    Conflicts resolved in the console.

commit 2b1539f8 
Author: Roger Lee <[email protected]> 
Date: Mon Feb 26 2024 16:30:16 GMT-0500 (Eastern Standard Time) 

    Set username to lower when reading settings.cfg. Update changelog. Increment version



commit 083d5402 
Author: alkohlanyrf <[email protected]> 
Date: Fri Dec 01 2023 10:44:12 GMT-0500 (Eastern Standard Time) 

    Added buildspec.yml


commit 9ed0737 
Author: gregmagdits <[email protected]> 
Date: Thu Nov 30 2023 16:04:22 GMT-0500 (Eastern Standard Time) 

    added some changes to fix bug found during testing


commit c5fdd255 
Author: Reem Al-kohlany <[email protected]> 
Date: Thu Feb 08 2024 14:47:26 GMT-0500 (Eastern Standard Time) 

    version bump

commit f0635475 
Author: Nancy Schroepfer <[email protected]> 
Date: Mon Feb 26 2024 11:51:39 GMT-0500 (Eastern Standard Time) 

    REV-124 Update nda-tools to display missing data structure msg

commit 51a6b91 
Author: gregmagdits <[email protected]> 
Date: Wed Nov 29 2023 13:50:09 GMT-0500 (Eastern Standard Time) 

    fixed issues found during testing


commit 9eaa208 
Author: gregmagdits <[email protected]> 
Date: Thu Nov 30 2023 13:05:15 GMT-0500 (Eastern Standard Time) 

    first commit to remove alt endpoint from tool


commit 44b77f1c 
Author: gregmagdits <[email protected]> 
Date: Fri Feb 02 2024 15:09:56 GMT-0500 (Eastern Standard Time) 

    version bump


commit 9caaf31e 
Author: Debapriya Sarkar <[email protected]> 
Date: Wed Feb 21 2024 15:12:57 GMT-0500 (Eastern Standard Time) 

    REV-747 nda-tools should not allow submission with blank dataset description
Switched to testpypi URL


commit acecabb 
Author: gregmagdits <[email protected]> 
Date: Tue Nov 28 2023 16:57:04 GMT-0500 (Eastern Standard Time) 

    version bump


commit 6a8ac15f 
Author: gregmagdits <[email protected]> 
Date: Wed Jan 31 2024 06:10:59 GMT-0500 (Eastern Standard Time) 

    comitting small changes that are present during successful tests


commit d93a8299 
Author: Debapriya Sarkar <[email protected]> 
Date: Wed Feb 21 2024 15:08:41 GMT-0500 (Eastern Standard Time) 

     REV-747 nda-tools should not allow submission with blank dataset description
Merged conflicts



commit cce30c6 
Author: gregmagdits <[email protected]> 
Date: Tue Nov 28 2023 16:53:55 GMT-0500 (Eastern Standard Time) 

    updated code to initalize files in .NDATools dir the same way as the files in downloadcmd are initialized


commit e4c4ff1c 
Author: gregmagdits <[email protected]> 
Date: Sun Jan 28 2024 12:17:44 GMT-0500 (Eastern Standard Time) 

    async refactor seems successful


commit 4efce99 
Author: gregmagdits <[email protected]> 
Date: Tue Nov 28 2023 15:08:13 GMT-0500 (Eastern Standard Time) 

    changed option name


commit 7319066a 
Author: gregmagdits <[email protected]> 
Date: Wed Jan 24 2024 13:00:21 GMT-0500 (Eastern Standard Time) 

    this might be done


commit 0855f35 
Author: Reem Al-kohlany <[email protected]> 
Date: Mon Nov 27 2023 12:04:27 GMT-0500 (Eastern Standard Time) 

    Squashed commit of the following 

commit c8850129 
Author: alkohlanyrf <[email protected]> 
Date: Mon Nov 27 2023 11:30:39 GMT-0500 (Eastern Standard Time) 

     Removed --password as parser option for vtcmd and downloadcmd



commit 23b4034a 
Author: gregmagdits <[email protected]> 
Date: Wed Jan 24 2024 12:20:04 GMT-0500 (Eastern Standard Time) 

    code cleanup and fixed bug affecting the traversal of files while uploading

While uploading, files statuses are updated. We need to traverse backwars from the last page to the first page because going the other way will mean we will navigate past the last page sooner than intended and not all files will get uploaded.


commit 6fd6fe4 
Author: gregmagdits <[email protected]> 
Date: Wed Nov 22 2023 18:07:06 GMT-0500 (Eastern Standard Time) 

    add step to download metadata file if its not already present


commit 4dda180c 
Author: gregmagdits <[email protected]> 
Date: Wed Jan 24 2024 11:42:56 GMT-0500 (Eastern Standard Time) 

    got resume working


commit e9b2e54 
Author: gregmagdits <[email protected]> 
Date: Wed Nov 22 2023 10:07:27 GMT-0500 (Eastern Standard Time) 

    updated this to not require requests library when installing


commit 162807bd 
Author: gregmagdits <[email protected]> 
Date: Wed Jan 24 2024 10:24:32 GMT-0500 (Eastern Standard Time) 

    looking good


commit 49e10a1 
Author: gregmagdits <[email protected]> 
Date: Wed Nov 22 2023 07:40:59 GMT-0500 (Eastern Standard Time) 

    removed botocore as direct dependency


commit 2d11019a 
Author: gregmagdits <[email protected]> 
Date: Wed Jan 24 2024 10:14:47 GMT-0500 (Eastern Standard Time) 

    manually tested loop after detecting non-existant files...works


commit ae371d6 
Author: gregmagdits <[email protected]> 
Date: Wed Nov 22 2023 06:28:48 GMT-0500 (Eastern Standard Time) 

    moved mock to extra-require


commit 338570c0 
Author: gregmagdits <[email protected]> 
Date: Tue Jan 23 2024 17:24:14 GMT-0500 (Eastern Standard Time) 

    second round of refactoring to try to stream everything to avoid loading large number of files into memory


commit 03a8c8d 
Author: gregmagdits <[email protected]> 
Date: Tue Nov 21 2023 14:30:49 GMT-0500 (Eastern Standard Time) 

    change error logic to help with tests


commit be244db7 
Author: gregmagdits <[email protected]> 
Date: Tue Jan 23 2024 14:14:37 GMT-0500 (Eastern Standard Time) 

    fixed progress bar


commit 01a7b08 
Author: gregmagdits <[email protected]> 
Date: Thu Nov 16 2023 13:04:49 GMT-0500 (Eastern Standard Time) 

    fixed regression affecting -d flag of downloadcmd. moved package-metadata to metadata directory to avoid potentially downloading it multiple times


commit 07336ff6 
Author: gregmagdits <[email protected]> 
Date: Tue Jan 23 2024 13:30:38 GMT-0500 (Eastern Standard Time) 

    commit after working test with regular submission. (progress bar still needs work)


commit 2f583ed 
Author: gregmagdits <[email protected]> 
Date: Thu Nov 16 2023 11:55:35 GMT-0500 (Eastern Standard Time) 

    removed sleeps and bumped version


commit 5e9e65c3 
Author: gregmagdits <[email protected]> 
Date: Mon Jan 22 2024 16:31:27 GMT-0500 (Eastern Standard Time) 

    before attempting to remove full_file_path


commit bc4fbd7 
Author: alkohlanyrf <[email protected]> 
Date: Tue Nov 14 2023 14:10:13 GMT-0500 (Eastern Standard Time) 

    Fixed validation warning incorrect path output

CHANGELOG.md

@@ -1,5 +1,12 @@
 
 # CHANGELOG
+## 0.2.27 - 2024-04-11
+* [`nda-tools`] Username is now case-insensitive.
+* [`nda-tools`] added --log-dir command line arg to configure location of generated log file
+* [`vtcmd`] added validation for dataset description
+* [`vtcmd`] updated error handling when structure missing from data-expected
+* [`vtcmd`] code refactor to reduce memory footprint when handling large submissions. scroll through associated files instead of requesting credentials all up front
+
 ## 0.2.26 - 2023-12-04
 * [`nda-tools`] Added --verbose parser option. Enables debug logging.
 * [`nda-tools`] Removed obsolete --password and -qa as parser option.

NDATools/Configuration.py

@@ -1,30 +1,24 @@
 from __future__ import absolute_import, with_statement
 
+import configparser
 import getpass
 import json
 import logging
 import logging.config
+import os
 import platform
-import sys
 import time
 
 import keyring
 import requests
 import yaml
+from pkg_resources import resource_filename
 from requests import HTTPError
 
 import NDATools
 from NDATools import NDA_TOOLS_LOGGING_YML_FILE, Utils
 from NDATools.Utils import exit_error, HttpErrorHandlingStrategy
 
-if sys.version_info[0] < 3:
-    import ConfigParser as configparser
-    input = raw_input
-else:
-    import configparser
-import os
-from pkg_resources import resource_filename
-
 logger = logging.getLogger(__name__)
 
 
@@ -34,25 +28,28 @@ def __init__(self):
         pass
 
     @staticmethod
-    def load_config(logs_directory, verbose=False):
+    def load_config(default_log_directory, verbose=False, log_dir=None):
 
         with open(NDA_TOOLS_LOGGING_YML_FILE, 'r') as stream:
             config = yaml.load(stream, Loader=yaml.FullLoader)
-        log_file = os.path.join(logs_directory, "debug_log_{}.txt").format(time.strftime("%Y%m%dT%H%M%S"))
-        config['handlers']['file']['filename']=log_file
+        if log_dir and os.path.exists(log_dir):
+            log_file = os.path.join(log_dir, "debug_log_{}.txt").format(time.strftime("%Y%m%dT%H%M%S"))
+        else:
+            log_file = os.path.join(default_log_directory, "debug_log_{}.txt").format(time.strftime("%Y%m%dT%H%M%S"))
+        config['handlers']['file']['filename'] = log_file
         if verbose:
-            config['loggers']['NDATools']['level']='DEBUG'
-            config['handlers']['console']['formatter']='detailed'
+            config['loggers']['NDATools']['level'] = 'DEBUG'
+            config['handlers']['console']['formatter'] = 'detailed'
         logging.config.dictConfig(config)
 
-class ClientConfiguration:
 
+class ClientConfiguration:
     SERVICE_NAME = 'nda-tools'
 
-    def __init__(self, username=None, access_key=None, secret_key=None):
+    def __init__(self, args):
         self.config = configparser.ConfigParser()
         logger.info('Using configuration file from {}'.format(NDATools.NDA_TOOLS_SETTINGS_CFG_FILE))
-        user_settings = self.config.read(NDATools.NDA_TOOLS_SETTINGS_CFG_FILE)
+        self.config.read(NDATools.NDA_TOOLS_SETTINGS_CFG_FILE)
         self._check_and_fix_missing_options()
         self.validation_api = self.config.get("Endpoints", "validation")
         self.submission_package_api = self.config.get("Endpoints", "submission_package")
@@ -61,34 +58,41 @@ def __init__(self, username=None, access_key=None, secret_key=None):
         self.package_creation_api = self.config.get("Endpoints", "package_creation")
         self.package_api = self.config.get("Endpoints", "package")
         self.datadictionary_api = self.config.get("Endpoints", "datadictionary")
+        self.collection_api = self.config.get("Endpoints", "collection")
         self.user_api = self.config.get("Endpoints", "user")
         self.aws_access_key = self.config.get("User", "access_key")
         self.aws_secret_key = self.config.get("User", "secret_key")
         self.aws_session_token = self.config.get('User', 'session_token')
-        self.username = self.config.get("User", "username")
+        self.username = self.config.get("User", "username").lower()
+
+        # options that appear in both vtcmd and downloadcmd
+        self.workerThreads = args.workerThreads
 
-        if username:
-            self.username = username
+        if args.username:
+            self.username = args.username
         elif self.username:
             logger.warning("-u/--username argument not provided. Using default value of '%s' which was saved in %s",
                            self.username, NDATools.NDA_TOOLS_SETTINGS_CFG_FILE)
+        self.password = None
 
-        if access_key:
-            self.aws_access_key = access_key
-        if secret_key:
-            self.aws_secret_key = secret_key
-        self.collection_id = None
-        self.endpoint_title = None
-        self.scope = None
-        self.directory_list = None
-        self.manifest_path = None
-        self.source_bucket = None
-        self.source_prefix = None
-        self.title = None
-        self.description = None
-        self.JSON = False
-        self.hideProgress = False
-        self.skip_local_file_check = False
+        is_vtcmd = 'collectionID' in args
+        if is_vtcmd:
+            self.aws_access_key = args.accessKey
+            self.aws_secret_key = args.secretKey
+            self.hideProgress = args.hideProgress
+            self.force = True if args.force else False
+            self.collection_id = args.collectionID
+            self.directory_list = args.listDir
+            self.manifest_path = args.manifestPath
+            self.source_bucket = args.s3Bucket
+            self.source_prefix = args.s3Prefix
+            self.validation_timeout = args.validation_timeout
+            self.title = args.title
+            self.description = args.description
+            self.scope = args.scope
+            self.JSON = args.JSON
+            self.skip_local_file_check = args.skipLocalAssocFileCheck
+            self.replace_submission = args.replace_submission
         logger.info('proceeding as nda user: {}'.format(self.username))
 
     def _check_and_fix_missing_options(self):
@@ -97,7 +101,7 @@ def _check_and_fix_missing_options(self):
         default_config.read(default_file_path)
         change_detected = False
         for section in default_config.sections():
-            if not section in self.config.sections():
+            if section not in self.config.sections():
                 logger.debug(f'adding {section} to settings.cfg')
                 self.config.add_section(section)
                 change_detected = True
@@ -115,12 +119,11 @@ def _check_and_fix_missing_options(self):
 
     def read_user_credentials(self, auth_req=True):
 
-        self.password = None
         if auth_req:
             while True:
                 try:
                     while not self.username:
-                        self.username = input('Enter your NIMH Data Archives username:')
+                        self.username = str(input('Enter your NIMH Data Archives username:')).lower()
                         with open(NDATools.NDA_TOOLS_SETTINGS_CFG_FILE, 'w') as configfile:
                             self.config.set('User', 'username', self.username)
                             self.config.write(configfile)
@@ -144,24 +147,24 @@ def read_user_credentials(self, auth_req=True):
                     raise e
 
         # Only ask for access-key/secret-key when needed (which is only when a user is creating a submission
-        # and the files are stored in an s3 bucket.
-        if self.source_bucket:
+        # and the files are stored in a s3 bucket)
+        if hasattr(self, 'source_bucket') and self.source_bucket:
             self.read_aws_credentials()
 
     def read_aws_credentials(self):
         if not self.aws_access_key:
-            self.aws_access_key = getpass.getpass("Enter your aws_access_key (must have read access to the %s bucket): " % self.source_bucket)
+            self.aws_access_key = getpass.getpass(
+                "Enter your aws_access_key (must have read access to the %s bucket): " % self.source_bucket)
 
         if not self.aws_secret_key:
             self.aws_secret_key = getpass.getpass('Enter your aws_secret_key:')
 
-
     def is_valid_nda_credentials(self):
         try:
             # will raise HTTP error 401 if invalid creds
-            tmp = Utils.get_request(self.user_api, headers={'content-type': 'application/json'},
-                                    auth=requests.auth.HTTPBasicAuth(self.username, self.password),
-                                    error_handler=HttpErrorHandlingStrategy.reraise_status)
+            Utils.get_request(self.user_api, headers={'content-type': 'application/json'},
+                              auth=requests.auth.HTTPBasicAuth(self.username, self.password),
+                              error_handler=HttpErrorHandlingStrategy.reraise_status)
             return True
         except HTTPError as e:
             if e.response.status_code == 401:

NDATools/Download.py

@@ -3,9 +3,11 @@
 import copy
 import gzip
 import pathlib
+import platform
 import shutil
 import sys
 import tempfile
+import urllib.parse
 import uuid
 from shutil import copyfile
 
@@ -82,10 +84,16 @@ def wait_completion(self):
 class DownloadRequest():
 
     def __init__(self,  package_file, presigned_url, package_id, download_dir):
+        operating_system = platform.system()
+        if operating_system == 'Windows':
+            download_dir = Utils.sanitize_windows_download_filename(download_dir)
         self.presigned_url = presigned_url
         self.package_file_id = str(package_file['package_file_id'])
         self.package_file_relative_path = package_file['download_alias']
-        self.completed_download_abs_path = os.path.normpath(Utils.convert_to_abs_path(os.path.join(download_dir, self.package_file_relative_path)))
+        self.completed_download_abs_path = os.path.normpath(Utils.convert_to_abs_path(os.path.join(download_dir,
+                         Utils.sanitize_windows_download_filename(self.package_file_relative_path)
+                         if operating_system == 'Windows' else self.package_file_relative_path)
+        ))
         self.package_download_directory = Utils.convert_to_abs_path(os.path.join(NDATools.NDA_TOOLS_DOWNLOADS_FOLDER, str(package_id)))
         self.nda_s3_url = None
         self.exists = False
@@ -244,7 +252,8 @@ def start(self):
                                                          fieldnames=self.download_job_progress_report_column_defs, extrasaction='ignore')
         failed_s3_links_file = tempfile.NamedTemporaryFile(mode='a',
                                                            delete=False,
-                                                           prefix='failed_s3_links_file_{}.txt'.format(time.strftime("%Y%m%dT%H%M%S")),
+                                                           prefix='failed_s3_links_file_{}'.format(time.strftime("%Y%m%dT%H%M%S")),
+                                                           suffix='.csv',
                                                            dir=NDATools.NDA_TOOLS_DOWNLOADCMD_LOGS_FOLDER
                                                            )
 
@@ -452,6 +461,7 @@ def mk_dir_ignore_err(dir):
                     for chunk in response.iter_content(chunk_size=1024 * 1024 * 5):  # iterate 5MB chunks
                         if chunk:
                             bytes_written += download_file.write(chunk)
+        # TODO - this doesnt work when using s3fs...add ticket to make it easy to download using s3fs
         os.rename(download_request.partial_download_abs_path, download_request.completed_download_abs_path)
         logger.info('Completed download {}'.format(download_request.completed_download_abs_path))
         download_request.actual_file_size = bytes_written
@@ -539,7 +549,7 @@ def handle_download_exception(self, download_request, e, download_local, err_if_
             #if we are using expired credentials, regenerate and resume the download
             credentials_are_expired = False
             if download_local:
-                if isinstance(e, requests.exceptions.HTTPError)and 'Request has expired' in e.response.text:
+                if isinstance(e, requests.exceptions.HTTPError) and 'Request has expired' in e.response.text:
                     credentials_are_expired = True
 
             if credentials_are_expired: