Simulatori neuronali e tools per l'analisi
Posted: Thu Oct 03, 2019 9:11 am
Cura questo topic Thierry Nieus
forum INDACO
https://forum.indaco.unimi.it/phpBB3/
Such an error might occur because the script you want to run is copied to a Slurm specific directory of another node and run from there.ModuleNotFoundError: No module named 'mymodule'
Code: Select all
import sys,os
sys.path.append(os.getcwd())
Code: Select all
import sys,os
sys.path.append(os.getcwd()) # append the path of the original folder to the system folders
import mymodule # now the module can be retrieved
Code: Select all
'''
Prototypical example that generates all scripts needed to launch an analysis on a set of files.
The procedure batch (below) will generate a set of files in the folder <<path_slurm_scripts_laptop>> of your computer.
For any problem/doubts on this please write to: thierry.nieus@unimi.it
'''
import os # module for calls to the operating system
# change the following variables according to your run
# from here <----
path_scripts_laptop='/home/myuser/scripts/' # path to the scripts on your laptop
path_scripts_cluster='/home/users/name.surname/my_project/' # path to the scripts on the cluster
path_data_cluster='/home/users/name.surname/data/' # path to the folder of your data
account='my_account' # name of the project you registered
sTIME='06:00:00' # how much time the process is supposed to run (keep it slightly higher than the max time you expect)
ram_memory=8000 # ram allocated to the process, 8000 MB here
# to here ---->
name_process='first_batch' # name of the process, it is a fancy and easy name to remember what your are doing
fname_execall='execall.sh' # file to manually execute the batch process (e.g. on a shell "sh execall.sh")
def core_procedure(file_to_process,count,path_pyslurm_local):
'''
file_to_process file name to process
count file number to process
path_pyslurm_local path of the .py and .slurm files
note:
The file "main_file.py" (below) should contain all variables and procedures to let "my_procedure" properly run on the cluster.
This can normally tested in advance on your computer before moving to the cluster.
'''
# names of the python and the slurm files
fnPY=os.path.join(path_scripts_laptop,path_pyslurm_local,'run_%d.py'%count) # the python file to run
fnSLURM=fnPY.replace('.py','.slurm') # the slurm file to run
fnPYcluster=fnPY.replace(path_scripts_laptop,path_scripts_cluster) # the full path of the python file to run on the cluster
# generates the Python file to execute
f = open(fnPY, 'w')
f.write('import sys,os\n')
f.write('sys.path.append(os.getcwd())\n') # the folder of the main_file.py is appended to the global path (see my other post)
f.write('exec(open(\'main_file.py\').read())\n') # load the main_file containing the procedures to launch
f.write('my_procedure(\'%s\')\n'%file_to_process)
f.close()
# generates the SLURM file to execute
f = open(fnSLURM, 'w')
f.write('#!/bin/bash \n')
# *.out - the log files to keep trace of the outcome of the runs
f.write('#SBATCH -o %s/%s%s.%s.%s.out \n'%(path_scripts_cluster,path_pyslurm_local,'out.out','%j','%N'))
f.write('#SBATCH -D %s/%s \n'%(path_scripts_cluster,path_pyslurm_local))
f.write('#SBATCH -J %s \n'%name_process)
f.write('#SBATCH --get-user-env \n')
f.write('#SBATCH -p light\n')
f.write('#SBATCH --nodes=1\n')
f.write('#SBATCH -c 1\n')
f.write('#SBATCH --mem-per-cpu %d\n'%ram_memory)
f.write('#SBATCH --account=%s\n'%account)
f.write('#SBATCH --time=%s \n'%sTIME)
f.write('module load python3/intel/2019 \n')
f.write('cd %s \n'%path_scripts_cluster)
f.write('python3 /gpfs%s \n'%fnPYcluster)
f.write('seff $SLURM_JOBID \n')
f.close()
return fnSLURM
def batch(file_list_name,path_pyslurm_local='slurm/batch1/'):
'''
path_pyslurm_local subfolder of path_scripts_laptop to move to the cluster
'''
# load the list of files to process
g = open(file_list_name,'r')
file_list=g.read().splitlines()
g.close()
num_processes=len(file_list) # number of files/processes to execute
path_slurm_scripts_laptop = os.path.join(path_scripts_laptop,path_pyslurm_local) # execall.sh will be in the folder of the PY and SLURM files
if not(os.path.exists(path_slurm_scripts_laptop)): os.makedirs(path_slurm_scripts_laptop)
fname_execall_fullpath = os.path.join(path_slurm_scripts_laptop,fname_execall)
g = open(fname_execall_fullpath, 'w')
for count in range(num_processes):
file2process=os.path.join(path_data_cluster,file_list[count])
fnSLURM=core_procedure(file2process,count,path_pyslurm_local)
g.write('sbatch %s\n'%fnSLURM.split(os.sep)[-1]) # os.sep yields the separator used for the path / (linux) \ (windows)
g.close()
# the files are now ready
print('')
print('Everything is ready now to be executed!')
print('')
print('1. copy the folder << %s >> and its content '%path_pyslurm_local.split('/')[0])
print(' to: %s'%path_scripts_cluster)
print('2. go to the folder %s'%os.path.join(path_scripts_cluster,path_pyslurm_local))
print(' run the command: sh %s'%fname_execall)
silvia.casarotto wrote: ↑Mon Dec 20, 2021 1:48 pm Hello! I would like to explore the possibility of running a specific analysis using INDACO facility.
I have a large set (about 400 recordings) of electroencephalographic (EEG) potentials evoked by transcranial magnetic stimulation (TMS). These data, which have been collected from about 60 EEG electrodes, have been used to compute a synthetic index of brain complexity (PCI) that found useful clinical applications.
In order to possibly simplify the experimental setup, I would like to evaluate whether PCI can be successfully computed also from a smaller number of electrodes.
However, the performance of the pre-processing steps (e.g., independent component analysis – ICA) required to reduce common artifacts of ocular and muscular origin before the computation of PCI may vary with the number of channels. Thus, I need to re-run the same analysis previously performed on 60-channels, also on 32 and 19 channels. In particular, I would like to apply the same ICA decomposition, i.e. runica as implemented in the EEGLAB software package (https://sccn.ucsd.edu/eeglab/index.php) that runs within the Matlab® environment. Then, for each dataset, I would like to perform a manual selection of artifact-contaminated independent components by visual inspection of their time course and scalp topography.
I kindly ask whether this project can be performed exploiting the INDACO features and how to implement this analysis pipeline.
Thank you in advance and best regards,
Silvia Casarotto
Dept. Biomedical and Clinical Sciences "L. Sacco"