diff --git a/docs/_posts/Cabir40/2023-10-03-multiclassifierdl_respiratory_disease_en.md b/docs/_posts/Cabir40/2023-10-03-multiclassifierdl_respiratory_disease_en.md index 97877b8826..fc4265674f 100644 --- a/docs/_posts/Cabir40/2023-10-03-multiclassifierdl_respiratory_disease_en.md +++ b/docs/_posts/Cabir40/2023-10-03-multiclassifierdl_respiratory_disease_en.md @@ -84,7 +84,6 @@ data = spark.createDataFrame([ ["""The patient was on Metformin for DM2, mood stabilizers for Bipolar II Disorder, and inhaled corticosteroids for Asthma."""], ["""The patient was diagnosed with Chronic Bronchitis after a series of pulmonary function tests."""], ["""Chest CT imaging revealed significant bullae and airspace enlargement, consistent with a diagnosis of emphysema."""], - ]).toDF("text") @@ -188,6 +187,6 @@ COPD 267 27 52 0.90816325 0.8369906 0.8711256 No 55 8 19 0.8730159 0.7432432 0.8029197 Chronic bronchitis 241 27 25 0.8992537 0.90601504 0.9026217 Asthma 104 15 25 0.8739496 0.8062016 0.83870965 -Macro-average 823 110 193 0.83017427 0.7265169 0.7748944 -Micro-average 823 110 193 0.88210076 0.8100393 0.84453565 -``` \ No newline at end of file +Macro-average 823 110 193 0.83017427 0.7265169 0.7748944 +Micro-average 823 110 193 0.88210076 0.8100393 0.84453565 +``` diff --git a/docs/_posts/Cabir40/2023-10-16-multiclassifierdl_heart_disease_en.md b/docs/_posts/Cabir40/2023-10-16-multiclassifierdl_heart_disease_en.md index 08d7b7620c..e51c8248bb 100644 --- a/docs/_posts/Cabir40/2023-10-16-multiclassifierdl_heart_disease_en.md +++ b/docs/_posts/Cabir40/2023-10-16-multiclassifierdl_heart_disease_en.md @@ -177,4 +177,4 @@ Hypertension 184 30 41 0.8598131 0.8177778 0.8382688 CAD 191 13 14 0.9362745 0.9317073 0.9339853 Macro-average 917 105 136 0.8870357 0.8389225 0.8623085 Micro-average 917 105 136 0.89726025 0.8708452 0.88385534 -``` \ No newline at end of file +``` diff --git a/docs/_posts/Cabir40/2023-10-16-multiclassifierdl_mental_disorder_en.md b/docs/_posts/Cabir40/2023-10-16-multiclassifierdl_mental_disorder_en.md index 047daa03be..bbbf3b5e48 100644 --- a/docs/_posts/Cabir40/2023-10-16-multiclassifierdl_mental_disorder_en.md +++ b/docs/_posts/Cabir40/2023-10-16-multiclassifierdl_mental_disorder_en.md @@ -35,6 +35,7 @@ The Mental Disorder Classifier Model is a specialized text classification system `Anxiety Disorder`, `Bipolar disorder`, `No`, `Schizophrenia`, `Depression`, `Other/Unknown` + {:.btn-box} [Live Demo](https://demo.johnsnowlabs.com/healthcare/CLASSIFICATION_MENTAL_DISORDER/){:.button.button-orange} [Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/streamlit_notebooks/healthcare/MENTAL_HEALTH.ipynb){:.button.button-orange.button-orange-trans.co.button-icon} @@ -133,7 +134,7 @@ val data = Seq(Array( """Commenced CBT to equip the patient with coping strategies for persistent irrational fears and nervousness stemming from their anxiety disorder.""", """Michael Anderson takes medications for schizophrenia, hypertension, hyperlipidemia, and migraines.""", """With a clinical presentation of persistent wheezing and respiratory discomfort, the patient has been conclusively diagnosed with Asthma, necessitating immediate commencement of anti-inflammatory medications.""", - )).toDS.toDF("text") +)).toDS.toDF("text") val result = clf_pipeline.fit(data).transform(data) @@ -204,4 +205,4 @@ Bipolar_disorder 101 5 10 0.9528302 0.9099099 0.9308756 Schizophrenia 66 4 23 0.94285715 0.74157304 0.8301887 Macro-average 750 73 119 0.89897865 0.8252565 0.8605415 Micro-average 750 73 119 0.9113001 0.863061 0.886524 -``` \ No newline at end of file +``` diff --git a/docs/_posts/ahmet-mesut/2023-11-05-date_of_birth_parser_en.md b/docs/_posts/ahmet-mesut/2023-11-05-date_of_birth_parser_en.md new file mode 100644 index 0000000000..3a7cee0882 --- /dev/null +++ b/docs/_posts/ahmet-mesut/2023-11-05-date_of_birth_parser_en.md @@ -0,0 +1,163 @@ +--- +layout: model +title: Date of Brith Contextual Parser Model +author: John Snow Labs +name: date_of_birth_parser +date: 2023-11-05 +tags: [en, licensed, contextual_parser, date_of_birth, dob, clinical] +task: Contextual Parser +language: en +edition: Healthcare NLP 5.0.2 +spark_version: 3.0 +supported: true +annotator: ContextualParserModel +article_header: + type: cover +use_language_switcher: "Python-Scala-Java" +--- + +## Description + +This model is a `ContextualParserModel` that can extract date-of-birth (DOB) entities in clinical texts. + +## Predicted Entities + +`DOB` + +{:.btn-box} + + +[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/date_of_birth_parser_en_5.0.2_3.0_1699191960551.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden} +[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/date_of_birth_parser_en_5.0.2_3.0_1699191960551.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3} + +## How to use + + + +
+{% include programmingLanguageSelectScalaPythonNLU.html %} + +```python +document_assembler = DocumentAssembler() \ + .setInputCol("text") \ + .setOutputCol("document") + +sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\ + .setInputCols(["document"])\ + .setOutputCol("sentence") + +tokenizer = Tokenizer() \ + .setInputCols(["sentence"]) \ + .setOutputCol("token") + +dob_contextual_parser = ContextualParserModel.pretrained("date_of_birth_parser", "en", "clinical/models") \ + .setInputCols(["sentence", "token"]) \ + .setOutputCol("chunk_dob") + +chunk_converter = ChunkConverter() \ + .setInputCols(["chunk_dob"]) \ + .setOutputCol("ner_chunk") + +parserPipeline = Pipeline(stages=[ + document_assembler, + sentence_detector, + tokenizer, + dob_contextual_parser, + chunk_converter + ]) + +model = parserPipeline.fit(spark.createDataFrame([[""]]).toDF("text")) + +sample_text = """ +Record date : 2081-01-04 +DB : 11.04.1962 +DT : 12-03-1978 +DOD : 10.25.23 + +SOCIAL HISTORY: +She was born on Nov 04, 1962 in London and got married on 04/05/1979. When she got pregnant on 15 May 1079, the doctor wanted to verify her DOB was November 4, 1962. Her date of birth was confirmed to be 11-04-1962, the patient is 45 years old on 25 Sep 2007. + +PROCEDURES: +Patient was evaluated on 1988-03-15 for allergies. She was seen by the endocrinology service and she was discharged on 9/23/1988. + +MEDICATIONS +1. Coumadin 1 mg daily. Last INR was on August 14, 2007, and her INR was 2.3.""" + +result = model.transform(spark.createDataFrame([[sample_text]]).toDF("text")) +``` +```scala +val document_assembler = new DocumentAssembler() + .setInputCol("text") + .setOutputCol("document") + +val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models") + .setInputCols("document") + .setOutputCol("sentence") + +val tokenizer = new Tokenizer() + .setInputCols("sentence") + .setOutputCol("token") + +val dob_contextual_parser = ContextualParserModel.pretrained("date_of_birth_parser", "en", "clinical/models") + .setInputCols(Array("sentence", "token")) + .setOutputCol("chunk_dob") + +val chunk_converter = new ChunkConverter() + .setInputCols(Array("chunk_dob")) + .setOutputCol("ner_chunk") + +val parserPipeline = new Pipeline().setStages(Array( + document_assembler, + sentence_detector, + tokenizer, + dob_contextual_parser, + chunk_converter + )) + +val data = Seq(Array(""" +Record date : 2081-01-04 +DB : 11.04.1962 +DT : 12-03-1978 +DOD : 10.25.23 + +SOCIAL HISTORY: +She was born on Nov 04, 1962 in London and got married on 04/05/1979. When she got pregnant on 15 May 1079, the doctor wanted to verify her DOB was November 4, 1962. Her date of birth was confirmed to be 11-04-1962, the patient is 45 years old on 25 Sep 2007. + +PROCEDURES: +Patient was evaluated on 1988-03-15 for allergies. She was seen by the endocrinology service and she was discharged on 9/23/1988. + +MEDICATIONS +1. Coumadin 1 mg daily. Last INR was on August 14, 2007, and her INR was 2.3.""" +)).toDS.toDF("text") + +val result = parserPipeline.fit(data).transform(data) +``` +
+ +## Results + +```bash ++-----------+----------------+-----+---+---------+ +|sentence_id|chunk |begin|end|ner_label| ++-----------+----------------+-----+---+---------+ +|1 |11.04.1962 |31 |40 |DOB | +|3 |Nov 04, 1962 |107 |118|DOB | +|4 |November 4, 1962|239 |254|DOB | +|5 |11-04-1962 |295 |304|DOB | ++-----------+----------------+-----+---+---------+ +``` + +{:.model-param} +## Model Information + +{:.table-model} +|---|---| +|Model Name:|date_of_birth_parser| +|Compatibility:|Healthcare NLP 5.0.2+| +|License:|Licensed| +|Edition:|Official| +|Input Labels:|[sentence, token]| +|Output Labels:|[chunk_dob]| +|Language:|en| +|Size:|4.9 KB| +|Case sensitive:|false| diff --git a/docs/_posts/ahmet-mesut/2023-11-05-date_of_death_parser_en.md b/docs/_posts/ahmet-mesut/2023-11-05-date_of_death_parser_en.md new file mode 100644 index 0000000000..22bef9c79b --- /dev/null +++ b/docs/_posts/ahmet-mesut/2023-11-05-date_of_death_parser_en.md @@ -0,0 +1,166 @@ +--- +layout: model +title: Date of Death Contextual Parser Model +author: John Snow Labs +name: date_of_death_parser +date: 2023-11-05 +tags: [en, clinical, date_of_death, contextual_parser, dod, licensed] +task: Contextual Parser +language: en +edition: Healthcare NLP 5.1.2 +spark_version: 3.0 +supported: true +annotator: ContextualParserModel +article_header: + type: cover +use_language_switcher: "Python-Scala-Java" +--- + +## Description + +This model is a `ContextualParserModel` that can extract date-of-death (DOD) entities in clinical texts. + +## Predicted Entities + +`DOD` + +{:.btn-box} + + +[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/date_of_death_parser_en_5.1.2_3.0_1699192712408.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden} +[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/date_of_death_parser_en_5.1.2_3.0_1699192712408.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3} + +## How to use + + + +
+{% include programmingLanguageSelectScalaPythonNLU.html %} + +```python +document_assembler = DocumentAssembler() \ + .setInputCol("text") \ + .setOutputCol("document") + +sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\ + .setInputCols(["document"])\ + .setOutputCol("sentence") + +tokenizer = Tokenizer() \ + .setInputCols(["sentence"]) \ + .setOutputCol("token") + +dod_contextual_parser = ContextualParserModel.pretrained("date_of_death_parser", "en", "clinical/models") \ + .setInputCols(["sentence", "token"]) \ + .setOutputCol("chunk_dod") + +chunk_converter = ChunkConverter() \ + .setInputCols(["chunk_dod"]) \ + .setOutputCol("ner_chunk") + +parserPipeline = Pipeline(stages=[ + document_assembler, + sentence_detector, + tokenizer, + dod_contextual_parser, + chunk_converter + ]) + +model = parserPipeline.fit(spark.createDataFrame([[""]]).toDF("text")) + +sample_text = """ +Record date : 2081-01-04 +DB : 11.04.1962 +DT : 12-03-1978 +DOD : 10.25.23 + +SOCIAL HISTORY: +Jane Doe was born on November 4, 1962, in London, and she got married on April 5, 1979. +When she got pregnant on May 15, 1979, the doctor wanted to verify her date of birth, which was confirmed to be November 4, 1962. +Jane was 45 years old when she sadly passed away on September 25, 2007. + +PROCEDURES: +Patient Jane Doe was evaluated on March 15, 1988, for allergies. She was seen by the endocrinology service and was discharged on September 23, 1988. + +MEDICATIONS: +1. Coumadin 1 mg daily. Jane's last INR was measured on August 14, 2007, and it was 2.3.""" + +result = model.transform(spark.createDataFrame([[sample_text]]).toDF("text")) + +``` +```scala +val document_assembler = new DocumentAssembler() + .setInputCol("text") + .setOutputCol("document") + +val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models") + .setInputCols("document") + .setOutputCol("sentence") + +val tokenizer = new Tokenizer() + .setInputCols("sentence") + .setOutputCol("token") + +val dod_contextual_parser = ContextualParserModel.pretrained("date_of_death_parser", "en", "clinical/models") + .setInputCols(Array("sentence", "token")) + .setOutputCol("chunk_dod") + +val chunk_converter = new ChunkConverter() + .setInputCols(Array("chunk_dod")) + .setOutputCol("ner_chunk") + +val parserPipeline = new Pipeline().setStages(Array( + document_assembler, + sentence_detector, + tokenizer, + dod_contextual_parser, + chunk_converter + )) + +val data = Seq(Array(""" +Record date : 2081-01-04 +DB : 11.04.1962 +DT : 12-03-1978 +DOD : 10.25.23 + +SOCIAL HISTORY: +Jane Doe was born on November 4, 1962, in London, and she got married on April 5, 1979. +When she got pregnant on May 15, 1979, the doctor wanted to verify her date of birth, which was confirmed to be November 4, 1962. +Jane was 45 years old when she sadly passed away on September 25, 2007. + +PROCEDURES: +Patient Jane Doe was evaluated on March 15, 1988, for allergies. She was seen by the endocrinology service and was discharged on September 23, 1988. + +MEDICATIONS: +1. Coumadin 1 mg daily. Jane's last INR was measured on August 14, 2007, and it was 2.3.""" +)).toDS.toDF("text") + +val result = parserPipeline.fit(data).transform(data) +``` +
+ +## Results + +```bash ++-----------+------------------+-----+---+---------+ +|sentence_id|chunk |begin|end|ner_label| ++-----------+------------------+-----+---+---------+ +|3 |10.25.23 |64 |71 |DOD | +|5 |September 25, 2007|360 |377|DOD | ++-----------+------------------+-----+---+---------+ +``` + +{:.model-param} +## Model Information + +{:.table-model} +|---|---| +|Model Name:|date_of_death_parser| +|Compatibility:|Healthcare NLP 5.1.2+| +|License:|Licensed| +|Edition:|Official| +|Input Labels:|[sentence, token]| +|Output Labels:|[chunk_dod]| +|Language:|en| +|Size:|5.0 KB| +|Case sensitive:|false| diff --git a/docs/_posts/akrztrk/2023-10-23-ner_clinical_he.md b/docs/_posts/akrztrk/2023-10-23-ner_clinical_he.md new file mode 100644 index 0000000000..ad1e594fcf --- /dev/null +++ b/docs/_posts/akrztrk/2023-10-23-ner_clinical_he.md @@ -0,0 +1,162 @@ +--- +layout: model +title: Detect Problems, Tests and Treatments (ner_clinical) in Hebrew +author: John Snow Labs +name: ner_clinical +date: 2023-10-23 +tags: [ner, clinical, licensed, he] +task: Named Entity Recognition +language: he +edition: Healthcare NLP 5.1.1 +spark_version: 3.0 +supported: true +annotator: MedicalNerModel +article_header: + type: cover +use_language_switcher: "Python-Scala-Java" +--- + +## Description + +Pretrained named entity recognition deep learning model for clinical terms in Hebrew. The SparkNLP deep learning model (MedicalNerModel) is inspired by a former state of the art model for NER: Chiu & Nicols, Named Entity Recognition with Bidirectional LSTM-CNN. + +## Predicted Entities + +`PROBLEM`, `TEST`, `TREATMENT` + +{:.btn-box} +[Live Demo](https://demo.johnsnowlabs.com/healthcare/NER_CLINICAL_MULTI/){:.button.button-orange} +[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/streamlit_notebooks/healthcare/NER_CLINICAL_MULTI.ipynb){:.button.button-orange.button-orange-trans.co.button-icon} +[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/ner_clinical_he_5.1.1_3.0_1698090197863.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden} +[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/ner_clinical_he_5.1.1_3.0_1698090197863.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3} + +## How to use + + + +
+{% include programmingLanguageSelectScalaPythonNLU.html %} +```python +document_assembler = DocumentAssembler()\ + .setInputCol("text")\ + .setOutputCol("document") + +sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "xx")\ + .setInputCols(["document"])\ + .setOutputCol("sentence") + +tokenizer = Tokenizer()\ + .setInputCols(["sentence"])\ + .setOutputCol("token") + +embeddings = BertEmbeddings.pretrained("alephbertgimmel_base_512","he") \ + .setInputCols(["sentence", "token"]) \ + .setOutputCol("embeddings") + +ner_model = MedicalNerModel.pretrained("ner_clinical", "he", "clinical/models") \ + .setInputCols(["sentence", "token", "embeddings"])\ + .setOutputCol("ner") + +ner_converter = NerConverterInternal()\ + .setInputCols(["sentence", "token", "ner"])\ + .setOutputCol("ner_chunk") + +pipeline = Pipeline(stages=[ + document_assembler, + sentence_detector, + tokenizer, + embeddings, + ner_model, + ner_converter + ]) + +sample_text = """פה רגיל צוואר רגיל תרואיד תקין שדיים רגילים ללא גושים בולטים פטילות רגילות הפוכות [ב] , מתפשטות עם גירוי חזה רגיל LCTA COR תקין RRR בטן רגילה מעוברת רגליים רגילות עור רגיל צמיגים רגילים פרטים רגילים ללא גידולים פלפים רגילים ללא תסמין נמוך רגיל פרטים לבנים דקים כמות קטנה של פ 4.5 , קוה + אמין , NS + רמז , טריש - רגיל רחם רגיל 1/100/0 SROM ברור סגור נפשות רגילות ללא גידולים פלפים NT רגיל רחם רגיל גודל בשבועות תקין מעי רגיל ללא גידולים חיצוניים.""" + + +data = spark.createDataFrame([[sample_text]]).toDF("text") + +result = pipeline.fit(data).transform(data) +``` +```scala +val document_assembler = new DocumentAssembler() + .setInputCol("text") + .setOutputCol("document") + +val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "xx") + .setInputCols("document") + .setOutputCol("sentence") + +val tokenizer = new Tokenizer() + .setInputCols("sentence") + .setOutputCol("token") + +val embeddings = BertEmbeddings.pretrained("alephbertgimmel_base_512","he") + .setInputCols(Array("sentence", "token")) + .setOutputCol("embeddings") + +val ner_model = MedicalNerModel.pretrained("ner_clinical", "he", "clinical/models") + .setInputCols(Array("sentence", "token", "embeddings")) + .setOutputCol("ner") + +val ner_converter = new NerConverterInternal() + .setInputCols(Array("sentence", "token", "ner")) + .setOutputCol("ner_chunk") + +val pipeline = new Pipeline().setStages(Array( + document_assembler, + sentence_detector, + tokenizer, + embeddings, + ner_model, + ner_converter +)) + +sample_data = Seq("""פה רגיל צוואר רגיל תרואיד תקין שדיים רגילים ללא גושים בולטים פטילות רגילות הפוכות [ב] , מתפשטות עם גירוי חזה רגיל LCTA COR תקין RRR בטן רגילה מעוברת רגליים רגילות עור רגיל צמיגים רגילים פרטים רגילים ללא גידולים פלפים רגילים ללא תסמין נמוך רגיל פרטים לבנים דקים כמות קטנה של פ 4.5 , קוה + אמין , NS + רמז , טריש - רגיל רחם רגיל 1/100/0 SROM ברור סגור נפשות רגילות ללא גידולים פלפים NT רגיל רחם רגיל גודל בשבועות תקין מעי רגיל ללא גידולים חיצוניים.""").toDS.toDF("text") + + +val result = pipeline.fit(sample_data).transform(sample_data) +``` +
+ +## Results + +```bash ++--------------------------+-----------+ +|chunk | ner_label | ++--------------------------+-----------+ +| גושים בולטים | PROBLEM | +| הפוכות | PROBLEM | +| מתפשטות עם גירוי חזה | TREATMENT | +| כמות קטנה של פ | TEST | +| קוה | TEST | +| NS | TEST | +| טריש | TEST | +| SROM | PROBLEM | ++--------------------------+-----------+ +``` + +{:.model-param} +## Model Information + +{:.table-model} +|---|---| +|Model Name:|ner_clinical| +|Compatibility:|Healthcare NLP 5.1.1+| +|License:|Licensed| +|Edition:|Official| +|Input Labels:|[sentence, token, embeddings]| +|Output Labels:|[ner]| +|Language:|he| +|Size:|3.4 MB| + +## Benchmarking + +```bash + label precision recall f1-score support + PROBLEM 0.80 0.78 0.79 607 + TREATMENT 0.80 0.64 0.71 280 + TEST 0.85 0.85 0.85 354 + micro-avg 0.81 0.77 0.79 1241 + macro-avg 0.82 0.75 0.78 1241 +weighted-avg 0.81 0.77 0.79 1241 +``` diff --git a/docs/_posts/akrztrk/2023-11-05-bert_sequence_classifier_sdoh_frailty_en.md b/docs/_posts/akrztrk/2023-11-05-bert_sequence_classifier_sdoh_frailty_en.md new file mode 100644 index 0000000000..319ab6373b --- /dev/null +++ b/docs/_posts/akrztrk/2023-11-05-bert_sequence_classifier_sdoh_frailty_en.md @@ -0,0 +1,144 @@ +--- +layout: model +title: Social Determinants of Healthcare for Frailty Classifier +author: John Snow Labs +name: bert_sequence_classifier_sdoh_frailty +date: 2023-11-05 +tags: [licenced, en, sdoh, clinical, social_determinants_of_heathcare, public_health, frailty, licensed, tensorflow] +task: Text Classification +language: en +edition: Healthcare NLP 5.1.2 +spark_version: 3.0 +supported: true +engine: tensorflow +annotator: MedicalBertForSequenceClassification +article_header: + type: cover +use_language_switcher: "Python-Scala-Java" +--- + +## Description + +The Frailty classifier employs [BioBERT](https://arxiv.org/abs/1901.08746v2) within a robust classifier architecture. Trained on a diverse dataset, this model provides accurate label assignments and confidence scores for its predictions. The primary goal of this model is to categorize text into two key labels: 'High_or_Low_Frailty' and 'No_Frailty_or_Unknown.' + +`High_or_Low_Frailty`: This category encompasses statements that indicate the presence of significant concerns or symptoms related to frailty conditions, suggesting either high or low risks. This label reflects substantial worries about frailty and the need for relevant medical measures. + +`No_Frailty_or_Unknown`: This category includes statements that do not contain clear concerns related to frailty or where the frailty condition is unknown. This label indicates the absence of frailty-related issues or insufficient information in the text. + +## Predicted Entities + +`High_or_Low_Frailty`, `No_Frailty_or_Unknown` + +{:.btn-box} + + +[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/bert_sequence_classifier_sdoh_frailty_en_5.1.2_3.0_1699190332760.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden} +[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/bert_sequence_classifier_sdoh_frailty_en_5.1.2_3.0_1699190332760.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3} + +## How to use + + + +
+{% include programmingLanguageSelectScalaPythonNLU.html %} +```python +document_assembler = DocumentAssembler()\ + .setInputCol('text')\ + .setOutputCol('document') + +tokenizer = Tokenizer()\ + .setInputCols(['document'])\ + .setOutputCol('token') + +sequenceClassifier = MedicalBertForSequenceClassification.pretrained('bert_sequence_classifier_sdoh_frailty', 'en', 'clinical/models')\ + .setInputCols(["document",'token'])\ + .setOutputCol("prediction") + +pipeline = Pipeline(stages=[ + document_assembler, + tokenizer, + sequenceClassifier +]) + +sample_texts=[ "Patient B is a 40-year-old female who was diagnosed with breast cancer. She has received a treatment plan that includes surgery, chemotherapy, and radiation therapy.", + "Post-chemotherapy, the patient was under regular surveillance for osteosarcoma. Recent imaging showed no signs of local recurrence or distant metastasis. Whereas the recovery was challenging, current evaluation confirms patient is in remission.", + "The patient was diagnosed with stage II colon cancer and will be undergoing a treatment regimen that includes both chemotherapy and radiation therapy.", + "Thyroid nodules detected during routine examination; fine-needle aspiration was conducted. Cytology results indicated no malignancy, consistent with a benign thyroid adenoma. However, patient is advised for a follow-up ultrasound in 12 months to monitor nodule size.", + "The patient's persistent lymphadenopathy led to further tests, which confirmed a diagnosis of AIDS.", + "Female patient presented with pelvic discomfort. Ovarian cysts were found during ultrasound; however, CA-125 levels are within normal range, and repeat imaging has shown consistent cyst size. No features of ovarian cancer were present, and a follow-up is scheduled in six months." + ] + +sample_data = spark.createDataFrame(sample_texts, StringType()).toDF("text") + +result = pipeline.fit(sample_data).transform(sample_data) + +result.select("text", "prediction.result").show(30,truncate=100) +``` +```scala +val documenter = new DocumentAssembler() + .setInputCol("text") + .setOutputCol("document") + +val tokenizer = new Tokenizer() + .setInputCols(Array("document")) + .setOutputCol("token") + +val sequenceClassifier = MedicalBertForSequenceClassification.pretrained("bert_sequence_classifier_sdoh_frailty", "en", "clinical/models") + .setInputCols(Array("document","token")) + .setOutputCol("prediction") + +val pipeline = new Pipeline().setStages(Array(documenter, tokenizer, sequenceClassifier)) + +val data = Seq(Array("Patient B is a 40-year-old female who was diagnosed with breast cancer. She has received a treatment plan that includes surgery, chemotherapy, and radiation therapy.", + "Post-chemotherapy, the patient was under regular surveillance for osteosarcoma. Recent imaging showed no signs of local recurrence or distant metastasis. Whereas the recovery was challenging, current evaluation confirms patient is in remission.", + "The patient was diagnosed with stage II colon cancer and will be undergoing a treatment regimen that includes both chemotherapy and radiation therapy.", + "Thyroid nodules detected during routine examination; fine-needle aspiration was conducted. Cytology results indicated no malignancy, consistent with a benign thyroid adenoma. However, patient is advised for a follow-up ultrasound in 12 months to monitor nodule size.", + "The patient's persistent lymphadenopathy led to further tests, which confirmed a diagnosis of AIDS.", + "Female patient presented with pelvic discomfort. Ovarian cysts were found during ultrasound; however, CA-125 levels are within normal range, and repeat imaging has shown consistent cyst size. No features of ovarian cancer were present, and a follow-up is scheduled in six months." +)).toDS.toDF("text") + +val result = pipeline.fit(data).transform(data) +``` +
+ +## Results + +```bash ++----------------------------------------------------------------------------------------------------+-----------------------+ +| text| result| ++----------------------------------------------------------------------------------------------------+-----------------------+ +|Patient B is a 40-year-old female who was diagnosed with breast cancer. She has received a treatm...| [High_or_Low_Frailty]| +|Post-chemotherapy, the patient was under regular surveillance for osteosarcoma. Recent imaging sh...|[No_Frailty_or_Unknown]| +|The patient was diagnosed with stage II colon cancer and will be undergoing a treatment regimen t...| [High_or_Low_Frailty]| +|Thyroid nodules detected during routine examination; fine-needle aspiration was conducted. Cytolo...|[No_Frailty_or_Unknown]| +| The patient's persistent lymphadenopathy led to further tests, which confirmed a diagnosis of AIDS.| [High_or_Low_Frailty]| +|Female patient presented with pelvic discomfort. Ovarian cysts were found during ultrasound; howe...|[No_Frailty_or_Unknown]| ++----------------------------------------------------------------------------------------------------+-----------------------+ +``` + +{:.model-param} +## Model Information + +{:.table-model} +|---|---| +|Model Name:|bert_sequence_classifier_sdoh_frailty| +|Compatibility:|Healthcare NLP 5.1.2+| +|License:|Licensed| +|Edition:|Official| +|Input Labels:|[document, token]| +|Output Labels:|[prediction]| +|Language:|en| +|Size:|406.4 MB| +|Case sensitive:|true| +|Max sentence length:|512| + +## Benchmarking + +```bash + label precision recall f1-score support + High_or_Low_Frailty 0.898067 0.891798 0.894921 573 +No_Frailty_or_Unknown 0.908148 0.913562 0.910847 671 + accuracy - - 0.903537 1244 + macro-avg 0.903107 0.902680 0.902884 1244 + weighted-avg 0.903505 0.903537 0.903511 1244 +``` \ No newline at end of file diff --git a/docs/_posts/bunyamin-polat/2023-11-04-ner_human_phenotype_gene_clinical_langtest_en.md b/docs/_posts/bunyamin-polat/2023-11-04-ner_human_phenotype_gene_clinical_langtest_en.md new file mode 100644 index 0000000000..427a5da9cc --- /dev/null +++ b/docs/_posts/bunyamin-polat/2023-11-04-ner_human_phenotype_gene_clinical_langtest_en.md @@ -0,0 +1,153 @@ +--- +layout: model +title: Detect Genes and Human Phenotypes (LangTest) +author: John Snow Labs +name: ner_human_phenotype_gene_clinical_langtest +date: 2023-11-04 +tags: [en, ner, clinical, licensed, langtest, human, phenotype, gene] +task: Named Entity Recognition +language: en +edition: Healthcare NLP 5.1.1 +spark_version: 3.0 +supported: true +annotator: MedicalNerModel +article_header: + type: cover +use_language_switcher: "Python-Scala-Java" +--- + +## Description + +This model detects mentions of genes and human phenotypes (hp) in medical text. It is the version of [ner_human_phenotype_gene_clinical](https://nlp.johnsnowlabs.com/2021/03/31/ner_human_phenotype_gene_clinical_en.html) model augmented with `langtest` library. + +| **test_type** | **before fail_count** | **after fail_count** | **before pass_count** | **after pass_count** | **minimum pass_rate** | **before pass_rate** | **after pass_rate** | +|----------------------|-----------------------|----------------------|-----------------------|----------------------|-----------------------|----------------------|---------------------| +| **add_ocr_typo** | 175 | 120 | 606 | 661 | 85% | 78% | 85% | +| **lowercase** | 262 | 110 | 521 | 673 | 85% | 67% | 86% | +| **swap_entities** | 123 | 112 | 600 | 614 | 80% | 83% | 85% | +| **titlecase** | 704 | 155 | 79 | 628 | 75% | 10% | 80% | +| **uppercase** | 709 | 174 | 74 | 609 | 75% | 9% | 78% | +| **weighted average** | **1973** | **671** | **1880** | **3185** | **80%** | **48.79%** | **82.60%** | + +## Predicted Entities + +`GENE`, `HP` + +{:.btn-box} + + +[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/ner_human_phenotype_gene_clinical_langtest_en_5.1.1_3.0_1699104573201.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden} +[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/ner_human_phenotype_gene_clinical_langtest_en_5.1.1_3.0_1699104573201.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3} + +## How to use + + + +
+{% include programmingLanguageSelectScalaPythonNLU.html %} + +```python +document_assembler = DocumentAssembler()\ + .setInputCol("text")\ + .setOutputCol("document") + +sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\ + .setInputCols(["document"])\ + .setOutputCol("sentence") + +tokenizer = Tokenizer()\ + .setInputCols(["sentence"])\ + .setOutputCol("token") + +word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")\ + .setInputCols(["sentence", "token"])\ + .setOutputCol("embeddings") + +clinical_ner = MedicalNerModel.pretrained("ner_human_phenotype_gene_clinical_langtest", "en", "clinical/models") \ + .setInputCols(["sentence", "token", "embeddings"]) \ + .setOutputCol("ner") + +ner_converter = NerConverter()\ + .setInputCols(["sentence", "token", "ner"])\ + .setOutputCol("ner_chunk") + +nlp_pipeline = Pipeline(stages=[document_assembler, sentence_detector, tokenizer, word_embeddings, clinical_ner, ner_converter]) + +model = nlp_pipeline.fit(spark.createDataFrame([['']]).toDF("text")) + +result = model.transform(spark.createDataFrame([["We will systematically examine seven genes (CHN1, MDH1, PCP4, RTN1, SLC14A1, SNAP25 and VSNL1) that are altered in the three neurodegenerative diseases."]]).toDF("text")) +``` +```scala +val document_assembler = new DocumentAssembler() + .setInputCol("text") + .setOutputCol("document") + +val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models") + .setInputCols("document") + .setOutputCol("sentence") + +val tokenizer = new Tokenizer() + .setInputCols("sentence") + .setOutputCol("token") + +val word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models") + .setInputCols(Array("sentence", "token")) + .setOutputCol("embeddings") + +val ner = MedicalNerModel.pretrained("ner_human_phenotype_gene_clinical_langtest", "en", "clinical/models") + .setInputCols(Array("sentence", "token", "embeddings")) + .setOutputCol("ner") + +val ner_converter = new NerConverter() + .setInputCols(Array("sentence", "token", "ner")) + .setOutputCol("ner_chunk") + +val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, tokenizer, word_embeddings, ner, ner_converter)) + +val data = Seq("""We will systematically examine seven genes (CHN1, MDH1, PCP4, RTN1, SLC14A1, SNAP25 and VSNL1) that are altered in the three neurodegenerative diseases.""").toDS().toDF("text") + +val result = pipeline.fit(data).transform(data) +``` +
+ +## Results + +```bash ++--------------------------+---------+ +|chunk |ner_label| ++--------------------------+---------+ +|CHN1 |GENE | +|MDH1 |GENE | +|PCP4 |GENE | +|RTN1 |GENE | +|SLC14A1 |GENE | +|SNAP25 |GENE | +|VSNL1 |GENE | +|neurodegenerative diseases|HP | ++--------------------------+---------+ +``` + +{:.model-param} +## Model Information + +{:.table-model} +|---|---| +|Model Name:|ner_human_phenotype_gene_clinical_langtest| +|Compatibility:|Healthcare NLP 5.1.1+| +|License:|Licensed| +|Edition:|Official| +|Input Labels:|[sentence, token, embeddings]| +|Output Labels:|[ner]| +|Language:|en| +|Size:|14.7 MB| + +## Benchmarking + +```bash +label precision recall f1-score support +GENE 0.85 0.89 0.87 1082 +HP 0.89 0.88 0.88 878 +micro-avg 0.87 0.88 0.87 1960 +macro-avg 0.87 0.88 0.87 1960 +weighted-avg 0.87 0.88 0.87 1960 +``` diff --git a/docs/_posts/bunyamin-polat/2023-11-04-ner_human_phenotype_go_clinical_langtest_en.md b/docs/_posts/bunyamin-polat/2023-11-04-ner_human_phenotype_go_clinical_langtest_en.md new file mode 100644 index 0000000000..8778e592ef --- /dev/null +++ b/docs/_posts/bunyamin-polat/2023-11-04-ner_human_phenotype_go_clinical_langtest_en.md @@ -0,0 +1,153 @@ +--- +layout: model +title: Detect Normalized Genes and Human Phenotypes (LangTest) +author: John Snow Labs +name: ner_human_phenotype_go_clinical_langtest +date: 2023-11-04 +tags: [en, ner, clinical, licensed, langtest, human, phenotype] +task: Named Entity Recognition +language: en +edition: Healthcare NLP 5.1.1 +spark_version: 3.0 +supported: true +annotator: MedicalNerModel +article_header: + type: cover +use_language_switcher: "Python-Scala-Java" +--- + +## Description + +This model can be used to detect normalized mentions of genes (go) and human phenotypes (hp) in medical text. It is the version of [ner_human_phenotype_go_clinical](https://nlp.johnsnowlabs.com/2021/03/31/ner_human_phenotype_go_clinical_en.html) model augmented with `langtest` library. + +| **test_type** | **before fail_count** | **after fail_count** | **before pass_count** | **after pass_count** | **minimum pass_rate** | **before pass_rate** | **after pass_rate** | +|----------------------|-----------------------|----------------------|-----------------------|----------------------|-----------------------|----------------------|---------------------| +| **add_abbreviation** | 116 | 53 | 475 | 538 | 85% | 80% | 91% | +| **add_ocr_typo** | 517 | 93 | 192 | 616 | 70% | 27% | 87% | +| **add_typo** | 201 | 92 | 480 | 594 | 75% | 70% | 87% | +| **lowercase** | 71 | 44 | 622 | 649 | 90% | 90% | 94% | +| **titlecase** | 701 | 123 | 8 | 586 | 70% | 1% | 83% | +| **uppercase** | 707 | 186 | 2 | 523 | 70% | 0% | 74% | +| **weighted average** | **2313** | **591** | **1779** | **3506** | **77%** | **43.48%** | **85.57%** | + +## Predicted Entities + +`GO`, `HP` + +{:.btn-box} + + +[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/ner_human_phenotype_go_clinical_langtest_en_5.1.1_3.0_1699113634845.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden} +[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/ner_human_phenotype_go_clinical_langtest_en_5.1.1_3.0_1699113634845.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3} + +## How to use + + + +
+{% include programmingLanguageSelectScalaPythonNLU.html %} + +```python +document_assembler = DocumentAssembler()\ + .setInputCol("text")\ + .setOutputCol("document") + +sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models") \ + .setInputCols(["document"]) \ + .setOutputCol("sentence") + +tokenizer = Tokenizer()\ + .setInputCols(["sentence"])\ + .setOutputCol("token") + +word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")\ + .setInputCols(["sentence", "token"])\ + .setOutputCol("embeddings") + +clinical_ner = MedicalNerModel.pretrained("ner_human_phenotype_go_clinical_langtest", "en", "clinical/models") \ + .setInputCols(["sentence", "token", "embeddings"]) \ + .setOutputCol("ner") + +ner_converter = NerConverter() \ + .setInputCols(["sentence", "token", "ner"]) \ + .setOutputCol("ner_chunk") + +pipeline = Pipeline(stages=[document_assembler, + sentence_detector, + tokenizer, + word_embeddings, + clinical_ner, + ner_converter]) + +data = spark.createDataFrame([["""Another disease that shares two of the tumor components of CT, namely GIST and the tricarboxylic acid cycle is the Carney-Stratakis syndrome (CSS) or dyad."""]]).toDF("text") + +result = pipeline.fit(data).transform(data) +``` +```scala +val document_assembler = new DocumentAssembler() + .setInputCol("text") + .setOutputCol("document") + +val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models") + .setInputCols("document") + .setOutputCol("sentence") + +val tokenizer = new Tokenizer() + .setInputCols("sentence") + .setOutputCol("token") + +val word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models") + .setInputCols(Array("sentence", "token")) + .setOutputCol("embeddings") + +val ner = MedicalNerModel.pretrained("ner_human_phenotype_go_clinical_langtest", "en", "clinical/models") + .setInputCols(Array("sentence", "token", "embeddings")) + .setOutputCol("ner") + +val ner_converter = new NerConverter() + .setInputCols(Array("sentence", "token", "ner")) + .setOutputCol("entities") + +val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, tokenizer, word_embeddings, ner, ner_converter)) + +val data = Seq("Another disease that shares two of the tumor components of CT, namely GIST and tricarboxylic acid cycle is the Carney-Stratakis syndrome (CSS) or dyad.").toDF("text") + +val result = pipeline.fit(data).transform(data) +``` +
+ +## Results + +```bash ++------------------------+---------+ +|chunk |ner_label| ++------------------------+---------+ +|tumor |HP | +|tricarboxylic acid cycle|GO | ++------------------------+---------+ +``` + +{:.model-param} +## Model Information + +{:.table-model} +|---|---| +|Model Name:|ner_human_phenotype_go_clinical_langtest| +|Compatibility:|Healthcare NLP 5.1.1+| +|License:|Licensed| +|Edition:|Official| +|Input Labels:|[sentence, token, embeddings]| +|Output Labels:|[ner]| +|Language:|en| +|Size:|14.6 MB| + +## Benchmarking + +```bash +label precision recall f1-score support +GO 0.89 0.81 0.85 1363 +HP 0.84 0.85 0.85 762 +micro-avg 0.87 0.82 0.85 2125 +macro-avg 0.86 0.83 0.85 2125 +weighted-avg 0.87 0.82 0.85 2125 +``` diff --git a/docs/_posts/bunyamin-polat/2023-11-06-ner_risk_factors_langtest_en.md b/docs/_posts/bunyamin-polat/2023-11-06-ner_risk_factors_langtest_en.md new file mode 100644 index 0000000000..e2188fc51d --- /dev/null +++ b/docs/_posts/bunyamin-polat/2023-11-06-ner_risk_factors_langtest_en.md @@ -0,0 +1,184 @@ +--- +layout: model +title: Detect Risk Factors (LangTest) +author: John Snow Labs +name: ner_risk_factors_langtest +date: 2023-11-06 +tags: [en, ner, clinical, licensed, langtest] +task: Named Entity Recognition +language: en +edition: Healthcare NLP 5.1.1 +spark_version: 3.0 +supported: true +annotator: MedicalNerModel +article_header: + type: cover +use_language_switcher: "Python-Scala-Java" +--- + +## Description + +Pretrained named entity recognition deep learning model for Heart Disease Risk Factors and Personal Health Information. It is the version of [ner_risk_factors](https://nlp.johnsnowlabs.com/2021/03/31/ner_risk_factors_en.html) model augmented with `langtest` library. + +| **test_type** | **before fail_count** | **after fail_count** | **before pass_count** | **after pass_count** | **minimum pass_rate** | **before pass_rate** | **after pass_rate** | +|----------------------|-----------------------|----------------------|-----------------------|----------------------|-----------------------|----------------------|---------------------| +| **add_ocr_typo** | 376 | 255 | 5287 | 5408 | 95% | 93% | 95% | +| **lowercase** | 616 | 403 | 5957 | 6170 | 92% | 91% | 94% | +| **titlecase** | 603 | 360 | 6083 | 6326 | 94% | 91% | 95% | +| **uppercase** | 1054 | 587 | 5715 | 6182 | 90% | 84% | 91% | +| **weighted average** | **2649** | **1605** | **23042** | **24086** | **93%** | **89.69%** | **93.75%** | + +## Predicted Entities + +`CAD`, `DIABETES`, `FAMILY_HIST`, `HYPERLIPIDEMIA`, `HYPERTENSION`, `MEDICATION`, `OBESE`, `PHI`, `SMOKER` + +{:.btn-box} + + +[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/ner_risk_factors_langtest_en_5.1.1_3.0_1699259077185.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden} +[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/ner_risk_factors_langtest_en_5.1.1_3.0_1699259077185.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3} + +## How to use + + + +
+{% include programmingLanguageSelectScalaPythonNLU.html %} + +```python +document_assembler = DocumentAssembler()\ + .setInputCol("text")\ + .setOutputCol("document") + +sentence_detector = SentenceDetector()\ + .setInputCols(["document"])\ + .setOutputCol("sentence") + +tokenizer = Tokenizer()\ + .setInputCols(["sentence"])\ + .setOutputCol("token") + +word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")\ + .setInputCols(["sentence", "token"])\ + .setOutputCol("embeddings") + +clinical_ner = MedicalNerModel.pretrained("ner_risk_factors_langtest", "en", "clinical/models") \ + .setInputCols(["sentence", "token", "embeddings"]) \ + .setOutputCol("ner") + +ner_converter = NerConverter()\ + .setInputCols(["sentence", "token", "ner"])\ + .setOutputCol("ner_chunk") + +nlp_pipeline = Pipeline(stages=[document_assembler, sentence_detector, tokenizer, word_embeddings, clinical_ner, ner_converter]) + +model = nlp_pipeline.fit(spark.createDataFrame([[""]]).toDF("text")) + +result = model.transform(spark.createDataFrame([["""PAST SURGICAL HISTORY: She has also had a hysterectomy, salpingoophorectomy, appendectomy, tonsillectomy, two carpal tunnel releases. She also has had a parathyroidectomy but still has had some borderline elevated calcium. Also, hypertension, hyperlipidemia, as well as diabetes. She also has osteoporosis. + +SOCIAL HISTORY: The patient still smokes about a third of a pack a day, also drinks only occasional alcoholic drinks. The patient is married. She has three grown sons, all of which are very successful in professional positions. One son is a gastroenterologist in San Diego, California. + +MEDICATIONS: Nifedipine-XR 90 mg daily, furosemide 20 mg half tablet b.i.d., lisinopril 20 mg daily, gemfibrozil 600 mg b.i.d., Synthroid 0.1 mg daily, Miacalcin one spray in alternate nostrils daily, Ogen 0.625 mg daily, Daypro 600 mg t.i.d., also Lortab 7.5 two or three a day, also Flexeril occasionally, also other vitamin. + +ALLERGIES: She had some adverse reactions to penicillin, sulfa, perhaps contrast medium, and some mycins."""]], ["text"])) +``` +```scala +val document_assembler = new DocumentAssembler() + .setInputCol("text") + .setOutputCol("document") + +val sentence_detector = new SentenceDetector() + .setInputCols("document") + .setOutputCol("sentence") + +val tokenizer = new Tokenizer() + .setInputCols("sentence") + .setOutputCol("token") + +val word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models") + .setInputCols(Array("sentence", "token")) + .setOutputCol("embeddings") + +val ner = MedicalNerModel.pretrained("ner_risk_factors_langtest", "en", "clinical/models") + .setInputCols(Array("sentence", "token", "embeddings")) + .setOutputCol("ner") + +val ner_converter = new NerConverter() + .setInputCols(Array("sentence", "token", "ner")) + .setOutputCol("ner_chunk") + +val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, tokenizer, word_embeddings, ner, ner_converter)) + +val data = Seq("""PAST SURGICAL HISTORY: She has also had a hysterectomy, salpingoophorectomy, appendectomy, tonsillectomy, two carpal tunnel releases. She also has had a parathyroidectomy but still has had some borderline elevated calcium. Also, hypertension, hyperlipidemia, as well as diabetes. She also has osteoporosis. + +SOCIAL HISTORY: The patient still smokes about a third of a pack a day, also drinks only occasional alcoholic drinks. The patient is married. She has three grown sons, all of which are very successful in professional positions. One son is a gastroenterologist in San Diego, California. + +MEDICATIONS: Nifedipine-XR 90 mg daily, furosemide 20 mg half tablet b.i.d., lisinopril 20 mg daily, gemfibrozil 600 mg b.i.d., Synthroid 0.1 mg daily, Miacalcin one spray in alternate nostrils daily, Ogen 0.625 mg daily, Daypro 600 mg t.i.d., also Lortab 7.5 two or three a day, also Flexeril occasionally, also other vitamin. + +ALLERGIES: She had some adverse reactions to penicillin, sulfa, perhaps contrast medium, and some mycins.""").toDS().toDF("text") + +val result = pipeline.fit(data).transform(data) +``` +
+ +## Results + +```bash ++------------------------------------------+--------------+ +|chunk |ner_label | ++------------------------------------------+--------------+ +|hypertension |HYPERTENSION | +|hyperlipidemia |HYPERLIPIDEMIA| +|diabetes |DIABETES | +|still smokes about a third of a pack a day|SMOKER | +|San Diego |PHI | +|California |PHI | +|lisinopril |MEDICATION | +|gemfibrozil |MEDICATION | ++------------------------------------------+--------------+ +``` + +{:.model-param} +## Model Information + +{:.table-model} +|---|---| +|Model Name:|ner_risk_factors_langtest| +|Compatibility:|Healthcare NLP 5.1.1+| +|License:|Licensed| +|Edition:|Official| +|Input Labels:|[sentence, token, embeddings]| +|Output Labels:|[ner]| +|Language:|en| +|Size:|14.7 MB| + +## References + +Trained on plain n2c2 2014: De-identification and Heart Disease Risk Factors Challenge datasets with `embeddings_clinical`. https://portal.dbmi.hms.harvard.edu/projects/n2c2-2014/ + +## Benchmarking + +```bash +label precision recall f1-score support +B-CAD 0.55 0.48 0.51 183 +B-DIABETES 0.74 0.80 0.77 246 +B-FAMILY_HIST 0.00 0.00 0.00 3 +B-HYPERLIPIDEMIA 0.85 0.89 0.87 95 +B-HYPERTENSION 0.73 0.81 0.77 233 +B-MEDICATION 0.84 0.85 0.84 894 +B-OBESE 0.69 0.82 0.75 49 +B-PHI 0.81 0.80 0.81 2106 +B-SMOKER 0.66 0.19 0.30 118 +I-CAD 0.38 0.36 0.37 476 +I-DIABETES 0.59 0.59 0.59 214 +I-FAMILY_HIST 0.83 0.29 0.43 17 +I-HYPERLIPIDEMIA 0.55 0.35 0.43 17 +I-HYPERTENSION 0.24 0.31 0.27 108 +I-MEDICATION 0.73 0.20 0.31 347 +I-OBESE 0.23 0.43 0.30 7 +I-PHI 0.79 0.72 0.75 618 +I-SMOKER 0.74 0.30 0.43 220 +micro-avg 0.73 0.67 0.70 5951 +macro-avg 0.61 0.51 0.53 5951 +weighted-avg 0.73 0.67 0.69 5951 +```