You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I read the paper and downloaded the AG news dataset,and tested PET model on it,but there is a great margin between my results and the author's results. I set parameters as below. To be specifically, I just used 10 examples for train(10 shot),model type is roberta,model_name_or_path is roberta-large,and I used all patterns for AG news. I did not change other parameters. And here is my results:
==============my results===============
acc-p0: 0.6450877192982456 +- 0.053859095516898825
acc-p1: 0.7874561403508772 +- 0.01841603941808791
acc-p2: 0.5642543859649123 +- 0.06912621607498706
acc-p3: 0.6119298245614034 +- 0.09528808314997761
acc-p4: 0.7537719298245614 +- 0.07473549078343446
acc-all-p: 0.6725 +- 0.10462149351553651
===============parameters setting===========
parser.add_argument("--train_examples", default=10, type=int,
help="The total number of train examples to use, where -1 equals all examples.")
parser.add_argument("--method", required=False, default='pet', choices=['pet', 'ipet', 'sequence_classifier'],
help="The training method to use. Either regular sequence classification, PET or iPET.")
parser.add_argument("--data_dir", default="./agnews/", type=str, required=False,
help="The input data dir. Should contain the data files for the task.")
parser.add_argument("--model_type", default="roberta", type=str, required=False, choices=MODEL_CLASSES.keys(),
help="The type of the pretrained language model to use")
parser.add_argument("--model_name_or_path", default="roberta-large", type=str, required=False,
help="Path to the pre-trained model or shortcut name")
parser.add_argument("--task_name", default="agnews", type=str, required=False, choices=PROCESSORS.keys(),
help="The name of the task to train/evaluate on")
parser.add_argument("--output_dir", default="./output/", type=str, required=False,
help="The output directory where the model predictions and checkpoints will be written")
# PET-specific optional parameters
parser.add_argument("--wrapper_type", default="mlm", choices=WRAPPER_TYPES,
help="The wrapper type. Set this to 'mlm' for a masked language model like BERT or to 'plm' "
"for a permuted language model like XLNet (only for PET)")
parser.add_argument("--pattern_ids", default=[0,1,2,3,4], type=int, nargs='+',
help="The ids of the PVPs to be used (only for PET)")
The text was updated successfully, but these errors were encountered:
Sorry,I missed the appendix. According to the details in appendixD, I retrain the model with some parameters changed,and here is the results, there is still a margin, but very close, maybe I still missed somethings.
acc-p0: 0.818859649122807 +- 0.02375440378086349
acc-p1: 0.8622368421052632 +- 0.011842105263157876
acc-p2: 0.8199122807017544 +- 0.044753157499628125
acc-p3: 0.7920175438596491 +- 0.01292178509375676
acc-p4: 0.8303508771929825 +- 0.032106970762756155
acc-all-p: 0.8246754385964913 +- 0.03328311578066454
I read the paper and downloaded the AG news dataset,and tested PET model on it,but there is a great margin between my results and the author's results. I set parameters as below. To be specifically, I just used 10 examples for train(10 shot),model type is roberta,model_name_or_path is roberta-large,and I used all patterns for AG news. I did not change other parameters. And here is my results:
==============my results===============
acc-p0: 0.6450877192982456 +- 0.053859095516898825
acc-p1: 0.7874561403508772 +- 0.01841603941808791
acc-p2: 0.5642543859649123 +- 0.06912621607498706
acc-p3: 0.6119298245614034 +- 0.09528808314997761
acc-p4: 0.7537719298245614 +- 0.07473549078343446
acc-all-p: 0.6725 +- 0.10462149351553651
===============parameters setting===========
parser.add_argument("--train_examples", default=10, type=int,
help="The total number of train examples to use, where -1 equals all examples.")
parser.add_argument("--method", required=False, default='pet', choices=['pet', 'ipet', 'sequence_classifier'],
help="The training method to use. Either regular sequence classification, PET or iPET.")
parser.add_argument("--data_dir", default="./agnews/", type=str, required=False,
help="The input data dir. Should contain the data files for the task.")
parser.add_argument("--model_type", default="roberta", type=str, required=False, choices=MODEL_CLASSES.keys(),
help="The type of the pretrained language model to use")
parser.add_argument("--model_name_or_path", default="roberta-large", type=str, required=False,
help="Path to the pre-trained model or shortcut name")
parser.add_argument("--task_name", default="agnews", type=str, required=False, choices=PROCESSORS.keys(),
help="The name of the task to train/evaluate on")
parser.add_argument("--output_dir", default="./output/", type=str, required=False,
help="The output directory where the model predictions and checkpoints will be written")
The text was updated successfully, but these errors were encountered: