Skip to content
Snippets Groups Projects
user avatar
Franziska Oschmann authored
36d1cc06
History

Moderation classifier

Installation local

python -m venv pp_env
source pp_env/bin/activate
pip install -r requirements.txt

Installation Euler

Tensorflow

PyTorch

Activation of environment

Local

source pp_env/bin/activate

On Euler

PyTorch

srun --pty --mem-per-cpu=3g --gpus=1 --gres=gpumem:12g bash
module load gcc/8.2.0 python_gpu/3.11.2 eth_proxy
source pp_env_torch/bin/activate

TensorFlow

srun --pty --mem-per-cpu=3g --gpus=1 --gres=gpumem:12g bash
module load gcc/8.2.0 python_gpu/3.10.4 eth_proxy
source pp_env_tf_python310/bin/activate

Usage

1. Preprocessing of dataframe (adding language field)

moderation_classifier --prepare_data path_to_csv

2. Model training

PyTorch

moderation_classifier --train_bert_torch data/tamedia_for_classifier_v2_preproc.csv

TensorFlow

moderation_classifier --train_bert data/tamedia_for_classifier_v2_preproc.csv