Open ExchangeThank you community for translating an earlier article into Portuguese.
Am returning the favor with a new release of Pattern Match Workbench demo app.
Added support for Portuguese.
The labels, buttons, feedback messages and help-text for user interface are updated.
Pattern Descriptions can be requested for the new language.
The single AI Model for transforming user prompt into Pattern match code was fully retrained.
.png)
Values to Pattern Code Model also retrained
The separate AI model for generating Pattern match code from a sample list of values has been retrained.
The previous model release exhibited loss of generalized pattern identification behaviors in order to support new delimiter style patterns.
The new training approach was evolved to better suit both types of challenge.
Post training benchmarking of the model stages was used to quantify the better candidate to preserve generalized pattern solving.
The principle to quantify 100% success for each generated pattern was to attempt to satisfy all its respective candidate sample records.
Overview benchmark report:
| Total benchmark tests used | 3895 |
| Mean success across all matches | 91.75% |
| Complete pattern match success | 81.98% |
| Partial pattern match success | 15.74% |
| Unsuccessful match records | 2.28% |
The following table gives examples from benchmark candidates demonstrating partial success.
It shows the percentage of sample records successfully matched to generated pattern code.
| Item | Sample Size | Context window | Rows matched | % Match | Actual Generated Pattern | Pattern Template |
| 1 | 31 | 31 | 28 | 90.3 | 4UN5AN2.3(4UNP2"/"1UP3.6LP1UP,2"/"1"Ç")5.11N | 4UN5AN2(4PUN2"/"1PU3.6LP1PU,2"/"1"Ç")5.12N |
| 2 | 31 | 31 | 17 | 54.8 | 5N5ANP3"7Æ6N1ŃS8"1(2"{{{",4"Û4"3"ĪĪċ")3P | 5.6N5NPA3"7Æ6N1ŃS8"1(3"{{",4"Û4"3"ĪĪċ")3P |
| 3 | 31 | 24 | 15 | 62.5 | 5.8"6Ã02"3(4"51704833"3"ĹĐRÞÇ",1PN,1LP1"K"4.6"7499")1"û5Ł¸3"4.7N | 5.8"6Ã02"3(4"51704833"3"ĹĐRÞÇ",1PN,1LP1"K"4.6"7499")1"û5Ł¸3"4.7N |
| 4 | 31 | 19 | 28 | 90.3 | 3P5UN5NP5UN5UN3AN3"JŁY"1.4"5"2U1LN | 3P5NU5NP4UN5NU4NA3"JŁY"1.4"5"2U1NL |
| 5 | 31 | 10 | 7 | 22.6 | 4LNP3.6AN4"×_"5"¤īĩĵ®ü"5AN3AN3LNP2UN3"ċ"5LN | 4NPL2.5NA4"×_"5"¤īĩĵ®ü"3.5NA3AN3PLN2UN3"ċ"5NL |
| 6 | 30 | 9 | 26 | 86.7 | 5.6"ŀ¬¦"5"!pp%"1UNP5"Ù6"4.8AP3(3N,4"8¸",5UNP)5.11P | 5.6"ŀ¬¦"5"!pp%"1PNU5"Ù6"4.7PA3(3N,3UP,4"8¸")4P4.5P |
| 7 | 25 | 12 | 5 | 20.0 | 2"AAHH"5"30"3(4.7UP,4L)5"ĥèßłĉ,,,"4.5N | 2"AAHH"5"30"5(3UP,4L,4UP,1PL)5"ĥèßłĉ,,,"4.5N |
| 8 | 31 | 14 | 30 | 96.8 | 5LP4")))¦¦§÷"4.10LN5.10UNP2"gx"1UN | 5PL4")))¦¦§÷"4.11NL5.10PUN2"gx"1UN |
Legend
Sample Size - The number records available for test sample. Sometimes a pattern describes less than 31 possible exact matches. Hence the lower values
Context Window - Maximum number of sample records that fit to generate a new pattern
Rows matched - This is the number of sample records that were matched by the generated pattern
Note that items 4, 6 and 8 achieve a match count greater than the context window.
When patterns are generated from values the context window used and proportion of matching is returned.
For challenging data samples it can be productive to retry the "Pattern from Values" button for improved results.
Showing the context window by row is useful to provide opportunity for specific records to be elevated into prompt for generative consideration.
.png)
Benchmark inferencing was conducted via HuggingFace spaces to scale up available GPU for timely execution.
Reporting is useful to analyse qualitative closeness of proposals by the LLM in terms of:
* Match type ( String, Number, Alphanumeric, Uppercase-Alpha)
* Choice of Quantity range ( 2-to-3 of ... )
* Option lists ( "123" or "456" )
* Structured elements using brackets
This opens up opportunities for training to be improved.
Please feel free to feedback on improvements for:
- Language translations
- Pattern match behaviors.
