ftract_dev / ImaGIN2 · Issues

For 0023REN, files in the /Raw and /Electrodes folders are from 13/07/2021 but files in the /Crop folder are from 29/08/2020 and are thus outdated (do not benefit from the re-processing of the previous steps). Regarding the issue, the commit from 10/05/2021 (8883f903) modified the BadChannel step so it manages CRFs <2009. Since this commit, the cropped files need to be updated/overwritten (rollback=True for the cropping step). However, to not lose the results of the quality control (you might want to do a test on a single CRF) you should not rollback the BadChannel step so the newly created/overwritten cropped files should be re-added to the database and then the corresponding _bIdxChecked.txt files already existing in the /BadChannels folder will be re-used to update the database as seen in execBadChannels.py.

Reassigned to @chantelb

The thing is that the rollback() function for the Cropping step deletes the CheckedCrop object in which the statuses of the Crops and of the BadChannels are stored...

I made a test on 0023REN.

Cropping step rollback=True

CorrectArtPulse rollback=True

BadChannels rollback=False

The statuses are forced to be "OK" if a _bIdxChecked.txt file is found. The content of this file is not written in the listBC field in the CheckedCrop object in the database. I don't know if it is OK or not...

This is an issue we already have to deal with in the past i think. As far as i can remember, doing a rollback on the cropping step will delete all crops (we want this) and remove them from the db (remove corresponding ftqualitycontrol_checkedcrop objects in the db). Re-doing the cropping step will re-generate the crops and add them back to the db. Then, the BadChannels step will detect that the "new" crops already have a corresponding _bIdxChecked.txt file existing in the BadChannels folder and will just update the ftqualitycontrol_checkedcrop object with an OK status (see execBadChannels.py at line 122). As i said in my first message it is important to not rollback the BadChannels step so you don't erase the bIdxChecked.txt files. I'm not 100% sure of all this but i think you arleady did some testing and it worked as intended. Maybe you can verify on 1 CRF. Doing a rollback from convert to cropping and then running the BadChannels step without rollback.

I'm just wondering if it matters if the content of the listBC field in the CheckedCrop object in the database is not the same as the content of the _bIdxChecked.txt file. I did not check it in my previous tests...

I think this is how it is designed. ListBC contains the result of manual validation on the webpage and not the list of all bad channels. After deleting the checkedcrop object you lose those results but the actual complete list of bad channels was saved in the bIdxChecked.txt file. I don't think there was a plan to use the bIdxChecked.txt to update ListBC or even check if both ListBC and bIdxChecked.txt have equal values. It is a one-way process as far as i can tell. When statusBC is set to OK the BadChannels step will just use the existing bIdxChecked.txt to update the SPM objects and the BadChannelQualityControl step will basically do nothing.

OK. Thank you very much for all these information :)

Another little question: do I have to reprocess this CRF from the BadChannelQualityControl to the end or not (as the files have been changed)?

I'm not sure. The files have changed but in a minor way (for the new BadChannels step that uses a new color scheme on screenshots). But you said "J'ai voulu retraiter des CRFs de REN suite à un changement des notes dans les fichiers sources en nov 2020" So maybe there are new crops ? The data inside the crops should be the same so if there are no new crops it is maybe not worth it to reprocess to the end.

OK thanks. I will close this issue.

Status changed to closed