Colab

GDown Download file (works on colab)

!pip install --upgrade --no-cache-dir gdown
import gdown
url = "gdrive file link"
gdown.download(url=url, output="filename", quiet=False, fuzzy=True)

Reload Modules (colab)

!pip install Ipython --upgrade
%load_ext autoreload
%autoreload 2

Audio

Cut all mp3 files in a folder

!pip install pydub
from os import listdir
from os.path import isfile, join
from pydub import AudioSegment

mypath = "/content/full_audio_files/"
all_music_filenames = [f for f in listdir(mypath) if isfile(join(mypath, f))]

#output folder
!mkdir cut_music
  
for m in all_music_filenames:
    song = AudioSegment.from_mp3(mypath+m)
    first_20_seconds = song[:20000]
    first_20_seconds.export("/content/cut_music/"+m, format="mp3")

Convert all audio files from a folder into mp3, and save to another folder

import os
from pydub import AudioSegment
from pydub.exceptions import CouldntDecodeError

!mkdir mp3_converted

def convert_files_to_mp3(source_folder, destination_folder):
    if not os.path.exists(destination_folder):
        os.makedirs(destination_folder)

    files = os.listdir(source_folder)

    for file_name in files:
        if not file_name.lower().endswith(('.wav', '.mp3', '.ogg', '.flac', '.aac', '.m4a')):
            continue
            
        source_path = os.path.join(source_folder, file_name)
        destination_path = os.path.join(destination_folder, os.path.splitext(file_name)[0] + '.mp3')

        try:
            # Load the audio file
            audio = AudioSegment.from_file(source_path)
        except CouldntDecodeError:
            print(f"Skipping {file_name} - Unable to decode the file.")
            continue

        audio.export(destination_path, format='mp3')

    print("Conversion complete!")

source_folder = '/content/all_music'
destination_folder = '/content/mp3_converted'
convert_files_to_mp3(source_folder, destination_folder)

Conda

Create new env

conda create --name myenv

Activate env

conda activate myenv

Fix wrong python, even when in conda env

conda create --name env_name python=3.10

export conda env info

conda env export --no-builds > env.yml

to set up: conda create env_name conda activate env_name conda env update --file env.yml

Zip

Extract contents of a zip into a destination foler

unzip /content/foo.zip -d /content/destination_folder

Zip a folder

zip -r compressed_filename.zip foldername

File

List all files in a folder

from os import listdir
from os.path import isfile, join
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

Git

Set up username and email on current repo:

git config user.name "Mainakdeb"

git config user.email "[email protected]"

check username or email before pushing

git config user.name

git config user.email

which branch am i on?

git branch

switch to existing branch

git switch <existing_branch>

create new branch and switch to it

git switch -c <non_existing_branch>

in case you commit using the wrong username:

git rebase -r <some commit before all of your bad commits> \

--exec 'git commit --amend --no-edit --reset-author'

git clone from a private repo using personal access token

git clone https://<PERSONAL_ACCESS_TOKEN>@github.com/username/your_repo.git

pull changes from remote repo and rebase

git pull --rebase origin main <or any branch>

rename / move a file

git mv sourceLocation targetLocation

Hugo Blog

1. clone

git clone https://github.com/Mainakdeb/blog.git

2. Create and activate env for hugo dev

conda create --name hugo-local-env
conda activate hugo-local-env

3. install hugo if needed

sudo apt install hugo

4. serve locally

hugo serve

5. Build Site

hugo

Upload contents from ./blog/public/ into a github repo named as your_username.github.io

SLURM

Allocate resources to yourself from login node (openmind)

srun -t 12:00:00 --ntasks-per-node=8 --mem=32G --gres=gpu:a100:1 --pty bash

Allocate resources (PACE) (gpu v100)

salloc -A gts-rmurty7 -q inferno -N1 --ntasks-per-node=1 --gres=gpu:1 -t4:00:00

Show running jobs

watch squeue -u <username>

Cancel job by name

scancel --name <your-job-name>

Cancel all running jobs

scancel -u <username>

SCP (Secure Copy Protocol)

scp <username>@ssh-host:/path-to-source ./path-to-destination/

Tmux

when in a terminal, and you need another within

tmux

tmux new-session -s code-tunnel

inside tmux, now to get a terminal (dont't forget to activate your conda env after this if needed)

bash

start your stuff, then press "ctrl+b, d" to exit tmux, "ctrl+b, x" will kill

check back whats going on in tmux

tmux attach -t 0

VSCode port forwarding

in case default port forwarding fails, use ngrok (useful for pycortex or jupyter lab)

ngrok http <PORT>

when running jupyter lab in vscode inside a slurm cluster (phoenix)

jupyter lab --NotebookApp.allow_origin='*' --NotebookApp.ip='0.0.0.0'

Python Package

install a package to edit (from local repo)

python3 setup.py develop

Mounting Drive (Linux)

find device to mount:

df -h

mount:

sudo mount /dev/sda /home/penfield/research/mega_storage

(You can unmount using umount)

check if mounted:

sudo df -a -T -h

give permissions to mounted device:

sudo chmod -R g+w /home/penfield/research/mega_storage
sudo chmod -R g+w /home/penfield/research/mega_storage

Publishing to pypi

python setup.py bdist_wheel sdist twine check dist/* twine upload --repository-url https://test.pypi.org/legacy/ dist/*

if you're using api key, type username token and paste token as password.