Moderation classifier
Installation local
python -m venv pp_env
source pp_env/bin/activate
pip install -r requirements.txt
Installation Euler
Tensorflow
PyTorch
Activation of environment
Local
source pp_env/bin/activate
On Euler
PyTorch
srun --pty --mem-per-cpu=3g --gpus=1 --gres=gpumem:12g bash
module load gcc/8.2.0 python_gpu/3.11.2 eth_proxy
source pp_env_torch/bin/activate
TensorFlow
srun --pty --mem-per-cpu=3g --gpus=1 --gres=gpumem:12g bash
module load gcc/8.2.0 python_gpu/3.10.4 eth_proxy
source pp_env_tf_python310/bin/activate
Usage
1. Preprocessing of dataframe (adding language field)
moderation_classifier --prepare_data path_to_csv
2. Model training
PyTorch
moderation_classifier --train_bert_torch data/tamedia_for_classifier_v2_preproc.csv
TensorFlow
moderation_classifier --train_bert data/tamedia_for_classifier_v2_preproc.csv