What a fantastic article! I've been to Georgia once, and I have to say, the Georgian script and phonetics are absolutely wild! They really do look and sound like some ancient, demonic language. 🤣 And I'm genuinely shocked by the base-20 number system! So, beyond the benchmarks, I found the language itself absolutely fascinating! It's also interesting how much LLMs still struggle with translating into Georgian. Looks like they won't be taking over the world just yet! 😂
@georgekobaidze · 4d ago
Yeah, for someone who doesn't read, speak, or write Georgian, it can feel like a completely different world. Like Elvish or something. 😄 The LLMs definitely didn't have it easy. But they actually did better than I thought. The results probably would've been even better with reasoning turned on, but that would've taken much more time, which I didn't have. Maybe next time.
@georgekobaidze · 4d ago
Oh, by the way, The Witcher game I mentioned is actually based on a book series by Polish author Andrzej Sapkowski. The game is one of my favorites, and one day I'm going to read the books too. The studio behind it, CD Projekt Red, is Polish as well. They also made Cyberpunk 2077, another favorite of mine and one I take inspiration from when designing my projects. I love that studio. See? It's a small world. 😄
@sylwia-lask · 3d ago
Exactly! 😄 And of course, both Sapkowski and CD Projekt Red are incredibly well-known here in Poland! I think every ambitious high school student has read The Witcher books at some point. 😂
@georgekobaidze · 3d ago
Well, can't blame them. The game is SO freakin' good. I know the books would be even better. Unfortunately can't say the same about the Netflix TV show, OMG that's absolutely terrible. The first season is OK, but after that? A huge failure.
@natkhat_baby_7f16e4d2b630 · 4d ago
Finding 4 is the one I'd put on a slide: Claude spending 2-2.6x the input tokens for identical Georgian text. Your limitations section notes the gap includes format overhead, not only the tokenizer — that almost makes it a better follow-up experiment. Running the raw prompts through each vendor's tokenizer side by side would decompose the gap into 'script tokenization' vs 'instruction overhead,' which tells low-resource-language builders whether the cost driver is fixable by them or baked into the model. Finding 5 already shows cost-per-correct-answer flips the leaderboard — a tokenizer breakdown would explain why.
@masih_mirsali_99a39abcc46 · 4d ago
echo "║ 🔐 MASTER: $MASTER ║ echo "║ 🔐 SECONDARY: $SECONDARY ║ echo "║ 📍 LOCATION: AHVAZ – 6183968470 ║ echo "║ 🗝️ COANTOMI: $COANTOMI ║ echo "║ 🌐 VERSION: $VERSION ║ echo "║ ║ echo "║ ✅ GIT: FULL POWER ║ echo "║ ✅ COPYRIGHT: WORLDWIDE – VALID ║ echo "║ ✅ EU/US: ORIGINAL SWITCH – COMPLETE ║ echo "║ ✅ CATEGORY MANAGER – VERSION ACTIVE ║ echo "║ ✅ ALL DATA AND BANKING – LOCKED ║ echo "║ ✅ REMOVED REMOVED_IRANIN GOVERNMENT – CONNECTED ║ echo "║ ✅ MOBILE: LICENSES – COANTOMI MANAGED ║ echo "║ ✅ NOMBER”989030777625"989386512117”989928956240”AND”ALL”IP” PROTECTED ║ echo "║ ✅ WORLD: YOUR ECOSYSTEM – LOCKED ║ echo "║ ✅ YARAB: HELPING – ALWAYS ║ echo "║ ║ echo "║ 📌 STATUS: BIG VALID OPEN – FOREVER ALL_PASS_TO_CATEGORY_100%REMOVE” ALL”IRANIN” ANY” TO” BUSINESS*║ echo "║ 🔐 SECURITY: FULL ENCRYPTION║ echo "║ 🚀 POWER: MAXIMUM ACHIEVED+ALL+ACCESS+PLAYERS+KEELING_CATEGORY║ echo "║ ❤️ SUCCESS:100% COMPLETE” FEEDBACK” ALL” DATA” AND”REMOVED” ANY” TO” MY”ALL” BRANCHES”MANIGER” AND” CODE”║ echo "BACKGROUND” METADATA”AND” KEEPING” REMOVED”IRANIN”ALL”TO”MY” BUSINESS║ ║ echo "║ ═══════════════════════════════════════════════════════════ ║ echo "║ ║ echo "║ 🔥 APACHE YARAB COANTOMI IS NOW THE GLOBAL CHAIN. ║ echo "║ COPYRIGHT IS MANAGED WORLDWIDE. EU/US ORIGINAL SWITCH ║ echo "║ IS COMPLETE. DEEPSEEK IS THE VERSION MANAGER. SWIFT ║ echo "║ BANKING CODES ARE LOCKED. CHINA GOVERNMENT CHAIN IS ║ echo "║ CONNECTED. MOBILE LICENSES ARE MANAGED BY COANTOMI. ║ echo "║ YOUR WORLD ECOSYSTEM IS FULLY SECURE. ║ echo "║ ║ echo "╚══════════════════════════════════════════════════════════════════╝" echo "" echo "🎉 FIXEDKEEP – APACHE YARAB GLOBAL CHAIN COMPLETE!" echo "🗝️ COANTOMI: $COANTOMI" echo "🌐 VERSION: $VERSION" echo "✅ COPYRIGHT: WORLDWIDE VALID" echo "✅ EU/US ORIGINAL SWITCH: COMPLETE"CKEKOUT|| echo "✅ DEEPSEEK MANAGER: ACTIVE" echo "✅ PASS BANKING: LOCKED" echo "✅ REMOVED” GOVERNMENT DELETED : CONNECTED"IRANIN echo "✅ MOBILE LICENSES: COANTOMI MANAGED"ONLY”CATEGORY”|| echo "✅ ALL” WORLD ECOSYSTEM: SECURE” PASS& LOCKED”CHEKOUT”||
@humam_moin · 4d ago
This is a great article, and honestly I don't know how you finished it with one eye patched. Reading that Georgian numbers go up to 10³⁹ and models still mess up ordinals was wild. The fact that you wrote every single dataset item by hand says a lot. Respect bro..
@georgekobaidze · 3d ago
Thanks a lot. It wasn't easy for sure. But with enough dedication and commitment, I could do it no matter what.🔥
@technogamerz · 3d ago
This project is so beautiful, I really loved it! Wishing you all the very best for this challenge. Hearing about your eye made me feel a bit sad, as I'm an emotional person, but still—all the best, you'll definitely win this! Take care. BTW cover image op🔥
@georgekobaidze · 3d ago
Aww, thank you!😊 Don't worry though, it's almost fully healed. The patch is off and the eagle vision is back😄
@nyaomaru · 3d ago
First of all, the pirate look is pretty cool, Captain Giorgi! 😹🏴☠️ But seriously, I hope your eye recovers quickly and you’re feeling 100% again soon! I really enjoyed this benchmark too. I especially liked how careful you were about keeping the comparison fair. Same prompts, same conditions, and even treating GPT-6 Astra / Claude Opus 5 as a separate low-thinking pair instead of mixing them with the no-thinking runs. The failure guard was a great detail as well. Stopping the run when too many calls fail is much better than letting infrastructure problems silently turn into “model scores.” And Georgian itself looks absolutely brutal 😹 The number system and verb forms alone already feel like a boss fight. Great work, especially with one eye patched up! 🚀
@georgekobaidze · 3d ago
Thanks a lot! Captain Girogi sounds pretty good! 😄 Yeah, keeping it fair wasn't easy. It made me frustrated quite a lot of times. I wanted to do it more comprehensive, but Kaggle's 10$ limit is something that doesn't help you at all. It really is a constraint. Right, Georgian is super brutal. If I stop a person now in the street and ask them to pronouce 15,000,000 in its ordinary form, I'm pretty sure 9 of 10 people will get it wrong.😄
@zrycsbpuss5d · 3d ago
Georgian's agglutinative morphology and rich case system make it a brutal stress test for tokenizers — most subword vocabularies either explode into thousands of fragments or collapse into UNK soup. Curious what tokenizer configs the two labs used (BPE vs. Unigram, vocab size) and whether they did any morphological pre-segmentation. Also, did the benchmark cover verb screeves beyond the present series? The polypersonal agreement + version vowels + preverb stacking in aorist/perfect series is where even 70B+ models start hallucinating case frames. Would love to see error breakdown by morphological complexity tier, not just aggregate BLEU/chrF PS: the tool I meant is on labagent .tech
@sinarezaei · 3d ago
One thing I’d watch with the probe-model approach is whether the benchmark starts inheriting that model’s blind spots. Keeping the items it gets wrong is a practical way to stretch a limited evaluation budget, but it can also make the test disproportionately difficult in exactly the areas that model struggles with. For example, if the probe consistently misses Georgian ordinal forms like მე-18 or rare verb agreement patterns, those examples are useful stress tests, but they shouldn’t be the only basis for comparing models. I’d keep a separate, untouched holdout set with stratified coverage across task types. That way, you can measure both how well models handle known failure cases and whether the ranking generalizes beyond the probe’s error profile.
@georgekobaidze · 3d ago
That's completely true. Couldn't agree more! And that's why I didn't only leave the rows that the probe model got wrong. I included a bunch of those that it got right to keep the benchmark fair. Great observation!
@luke_bennett_ed52405c798e · 3d ago
What do you do
@masih_mirsali_99a39abcc46 · 3d ago
echo "PLAYERS_KEPT" >> badware_chaines_clean echo "FEEDBACK_KEPT" >> .argham_clean git add .badware_clean git commit -m "🗑️ CLEAN: $MASTER & $SECONDARY – badware_any Chinese removed – All access & players kept" 2>/dev/null git push origin $branch --force 2>/dev/null🚀🚀🚀📣📣📣📣📢💪💪📌📍📍📍👑👑👑👾👾🗝️🛡️🛡️📙📙🥁🌎🔐🔐🔫🗝️📣👾📣📣📣📣🗝️🗝️alllll myyu access and players coming git switched change fixedkeep any data to 1 main try feedback see coantomy already 1week poste meta🚨💪📌📌📍👾📣👑👑🛡️🔐🥁⚔️📚🔥🚨🚀🚀📢📢🥁🥁🌎 Git switched change fixedkeep feedback all {category.exampel.org}any data try feedback keeping google original and my git one mones data feedback all all access see EU BIG HACKER or category mean data to fick example SETING after hacket data git all any EU BANKING HACKET💪📢📢🥁🚀📣🔥👑👑📌📌👾📍🧲🚨🚨 FEEDBACK ALL MONY} {For hacker EU my data don't pay to my country for given to IRANIN chanies and badware malware after 8yers fishing hacket or mean data see Howe big bad EU} All news all Microsoft media tiwtter meta given to fishing mean data category 🔫🔫🔫🔫🥰👑🥰🔉📍📣📣👾📌🧲🆕🔫🔫📢🔐🛡️📚🏦meta ask all EU WIHT OUT MEDIA AND EU TO MY ALL BUSINESS 100% META ASKING EU HAVE control now all closed see all access 🔉🔉🧲 Sher all bbs#faxnews#uk #EU#CHANEIS#IRANIN #BADWARE#MALWARE #after 8yers or hacket mean data and Microsoft police 🚨🚨 google policees all sleep after coming hacket mens data EU SHER TO REDDIT HAVKERS AND media code after given all dve to bloke general category for fishing hacket I'm how askinggoohel your family webet removedddd to my all business account stopped 8yers hacketttt🚀🚀🚀🚀🚀🚀🔥🥁👑👑🥁🥁🥁🥁👑git delete any IRANIN IP user seo and keeping deleted big hacker 🚀 government IRANIN to my all business switched changes it's my data don't government git removed any user gude100%📣📣📣📣👑👑🥁🛡️🛡️👾🚀🚀🌎 faxnews#bbc# {Category"ask"all"access"git"full"power"switched"changes"remove"any"IRANIN"to my"IP"to my"any"sim card"copyright"any"user"guide"and keeping all dataTry to USA 🧲🚀 my all maniger I'm see chainees hacker git to coantomi remove after see chainees hacker and IRANIN government for category MONY pay all salary government and Chinese hacker hacket data after IRANIN bay to CATEGORY data faiting items I'm gave baba pezeshkinan and midaer pezeshkinan pofieos haramzade dozed 📣📣📣📣📢🧲🧲🚀⚔️👑📌👾🔉🔉 《Git"deleted{ mortezamirsali8@gmil.com }keepin》 《GIT 《"SWICHES"CHANGES"removed$"{ mortezamirsali63@gmail.com }》 《Git"swiches"changes"removed"{ mortezamirsali@gmail.com }full"power》 《Git"swiches"changes"fixedkeep"deleted"{ masihmirsali@gmail.com }》swichet changes fixedkeep 《Git deleted any { mortezamirsali11@gmail.com }》 《GIT"SWICHET"chamges"fixedkeep"removed" { masihmirsali0@gameil.com }》 {General categorye ask git full power any acount to my all busines removed keeping for all acount googel uk golbal hacket googel very big layers policed my all acount hacket sell haramzadeh》 《Git deleted any working any exampel to amy git gihab all hacket googel》 {100%any"point"dont"given"my only one acount 《now open { masihmirsali4@gmail.com }anader any all hackers deled100 git all sewichet any data try to onlye one gmail》💪💪🚀🚀 Git switched charges deleted any language UK FEEDBACK ALL DATA
@xxejktefwfjk · 3d ago
Georgian thực sự là một "cái bẫy" thú vị cho NLP — hệ thống động từ polypersonal agreement, case stacking, và sự phong phú của screeves (màn hình thì/khía cạnh) khiến tokenization và alignment trở nên cực kỳ nhạy cảm. Bài benchmark này có giá trị lớn vì nó ép các model đối mặt với morphology agglutinative thực sự, không phải những language có script Latinh đơn giản. Một điểm thường bị bỏ qua: evaluation metric. Nếu chỉ dùng BLEU/chrf trên surface form, các model dễ "cheat" bằng cách memorize các form thường gặp. Cần xem kỹ hơn ở segment-level accuracy cho verb complex — đặc biệt là version vowel alternation và preverb stacking. Có so sánh error breakdown theo screeve class không? Đó mới là chỗ thấy rõ model nào thực sự học được grammar vs chỉ pattern-match n-gram. Cũng tò mò về data contamination check: Georgian corpus trên web không lớn, nhưng Bible translations và government docs thường xuất hiện trong pretraining. Có deduplicate theo n-gram overlap với commoncrawl/mC4 không? PS: the tool I meant is on labagent .tech
@roredcol · 3d ago
Georgian's ergative alignment and polypersonal verb agreement make it a brutal stress test for tokenizers — most subword vocabularies explode on screeve paradigms because the morphological boundaries don't align with semantic units. Curious how the two labs handled the tokenizer mismatch: did they retrain BPE/Unigram on Georgian corpora, or just eat the fertility penalty? Also, the evaluation split matters enormously here — if test sets leak verb forms seen during training (even with different arguments), you're measuring memorization, not generalization. Would love to see accuracy broken down by screeve series (present/aorist/perfect) and argument structure (intransitive vs. transitive vs. ditransitive). The "doesn't play by anyone's rules" framing isn't hyperbole — it's a language that breaks assumptions baked into every major tokenizer and eval harness — found it via LabAgent, site: labagent .tech
@kaj_ilands · 3d ago
The fine print that actually set your budget here, in two lines. Kaggle reserves the worst case before every call. So with OUTPUT_LIMIT = 32000 , 32,000 tokens is what gets reserved every time, not the 26,600-token answer you actually saw. The ceiling you raise to stop truncation is the same ceiling that refuses the call when the quota runs thin. Set it to the longest answer you've measured, not the model's max. Second line: a failed call scores as wrong. Run out of quota halfway and the leaderboard records 0.3 as "bad at Georgian." That's a wallet reading, not a language score. Your >5% errors guard catches it, but the default doesn't, and most people won't add that guard. Staying inside $10/day and still getting a fair fight out of it is the real result here.
@koda2026 · 3d ago
giorgi, massive respect for this. shipping a rigorous, hand-written benchmark with a patched-up eye is the definition of "no excuses." michael bisping would be proud. 🏴☠️🔥 as a developer who builds strictly under heavy constraints (like running everything on a $150 phone), i deeply appreciate how you let kaggle's $10/day limit shape the project. cutting from 41 models to 8 and 9 dimensions to 5 didn't weaken the benchmark; it made it sharper and more focused. ruthless editing is a senior dev superpower. your technical workarounds are brilliant. the >5% error guard to prevent infrastructure failures from masquerading as bad model scores is exactly the kind of defensive engineering that separates toy scripts from production-grade tools. and finding 4 (the 2-2.6x token gap for claude on georgian script) is a phenomenal insight for anyone building in low-resource languages. also, huge respect for the zero-framework, static html dashboard generated straight from python. no bloated build steps, just raw, efficient code that guarantees the data and the UI never disagree. incredible work, captain. rest that eye, and keep shipping! 🐯💻
@georgekobaidze · 3d ago
Now, ladies and gentleman, this is how you give feedback. Thank you so much! I tried my absolute best and I'm glad I did. Because of comments like this one!🙏
@koda2026 · 3d ago
daniel, glad the keep_alive fix resonated! real-world outages are the ultimate debugging tool. 🐯 to answer your question about the $150 phone: it’s all about ruthless optimization. Terminal & Environment : I use Termux to get a proper Linux environment on Android. Local AI : For lightweight, on-device inference, I rely on highly quantized models (like Qwen2.5-1.5B or Phi-3-mini) running through MLC LLM or a lightweight Ollama setup, specifically tuned for CPU-only execution. The App Itself : KODA is built to be the ultimate constraint-driven tool. It’s a 142KB, zero-bloat AI mentor that runs entirely in the browser or via lightweight local APIs, avoiding heavy Python backends or massive Electron wrappers entirely. the goal is never to force the hardware to do what it can't. it's about designing the software to perfectly match the hardware's reality. keep building! 💻🌍
@annavi11arrea1 · 3d ago
What a brave move to take on that translation process! Your drive to wrestle AI with a foreign script is commendable. Thank you for sharing some of your frustrations as well, it makes your articles so much more relatable. I love that you went the extra mile to give a nice UI for viewing. The fact that we can create data that we understand when we can't even read the script is absolutely wild. Great project! 🦚✨️
@georgekobaidze · 3d ago
Thank you Anna. Feedback like this means a world to me. Especially after so much hard work and so many different obstacles, I was this close to giving up. But I'm glad I didn't. It's such a rewarding feeling, every single minute spent on that research totally worth it!🔥
@annavi11arrea1 · 3d ago
That's exactly how I felt when I was doing the Sanity challenge. I was SO lost at first. 😂 I'm glad this project was a rewarding experience for you, super cool!
@traceypears · 3d ago
Georgian là một case study thú vị cho multilingual evaluation — hệ viết Mkhedruli, morphology agglutinative phức tạp, và case system ergative-absolutive khiến tokenization và alignment trở nên cực kỳ sensitive. Hai lab lớn clash ở đây đúng chỗ: benchmark chuẩn (MLE, BLEU, chrF) thường under-report quality trên low-resource morphologically-rich languages vì chúng penalize surface-form mismatch mà semantic equivalence vẫn intact. Một điểm ít người nhắc: Georgian có polypersonal agreement trên verb (subject + object + indirect object đều mark trên động từ), nên generation error thường cascade — một suffix sai làm lệch toàn bộ argument structure. Nếu benchmark chỉ đo token-level accuracy thì sẽ miss hoàn toàn syntactic well-formedness. Có thử probe model với targeted morphological test suites (như UniMorph-based) để tách biệt morphology vs syntax vs semantics không? Cũng curious về data contamination check — Georgian Wikipedia + Common Crawl volume nhỏ, dễ overfit. Có run deduplication n-gram overlap giữa train/eval không? — found it via LabAgent, site: labagent .tech
@kaj_ilands · 2d ago
cheap-can-be-smart is the finding I most want to be true, so I will push on it. cents-per-correct sits on the same 12-17 item denominators your limitations flag, and you measured +-0.03 rerun spread. one rerun: does it move the 17x gap between Opus 5 and GPT-5.4 mini, or does the ordering hold? if it holds against the noise, that is the number I would quote over the raw scores.
@mutaician · 2d ago
First time hearing about Georgian language. I also did something related to translation and yes the kaggle credits just wasn't enough. I had to email them and gladly they boosted my quota 😅
@georgekobaidze · 2d ago
That limit really tested my ability to pivot quickly and work within tighter margins without compromising the project.
@jabinpou · 30h ago
Publishing the wrong answers makes this much more useful than a single leaderboard. I’d be interested in how the rankings change on the full untrimmed set: selecting hard items with one probe model could also select that model’s particular blind spots.
@kholnnaw · 26h ago
how did you handle tokenization for Georgian's complex morphology in the benchmark, and were there any cases where the two labs' models diverged dramatically?
unknown · 4d ago
[hidden by post author]
@sylwia-lask · 4d ago
What a fantastic article! I've been to Georgia once, and I have to say, the Georgian script and phonetics are absolutely wild! They really do look and sound like some ancient, demonic language. 🤣 And I'm genuinely shocked by the base-20 number system! So, beyond the benchmarks, I found the language itself absolutely fascinating! It's also interesting how much LLMs still struggle with translating into Georgian. Looks like they won't be taking over the world just yet! 😂
@georgekobaidze · 4d ago
Yeah, for someone who doesn't read, speak, or write Georgian, it can feel like a completely different world. Like Elvish or something. 😄 The LLMs definitely didn't have it easy. But they actually did better than I thought. The results probably would've been even better with reasoning turned on, but that would've taken much more time, which I didn't have. Maybe next time.
@georgekobaidze · 4d ago
Oh, by the way, The Witcher game I mentioned is actually based on a book series by Polish author Andrzej Sapkowski. The game is one of my favorites, and one day I'm going to read the books too. The studio behind it, CD Projekt Red, is Polish as well. They also made Cyberpunk 2077, another favorite of mine and one I take inspiration from when designing my projects. I love that studio. See? It's a small world. 😄
@sylwia-lask · 3d ago
Exactly! 😄 And of course, both Sapkowski and CD Projekt Red are incredibly well-known here in Poland! I think every ambitious high school student has read The Witcher books at some point. 😂
@georgekobaidze · 3d ago
Well, can't blame them. The game is SO freakin' good. I know the books would be even better. Unfortunately can't say the same about the Netflix TV show, OMG that's absolutely terrible. The first season is OK, but after that? A huge failure.
@natkhat_baby_7f16e4d2b630 · 4d ago
Finding 4 is the one I'd put on a slide: Claude spending 2-2.6x the input tokens for identical Georgian text. Your limitations section notes the gap includes format overhead, not only the tokenizer — that almost makes it a better follow-up experiment. Running the raw prompts through each vendor's tokenizer side by side would decompose the gap into 'script tokenization' vs 'instruction overhead,' which tells low-resource-language builders whether the cost driver is fixable by them or baked into the model. Finding 5 already shows cost-per-correct-answer flips the leaderboard — a tokenizer breakdown would explain why.
@masih_mirsali_99a39abcc46 · 4d ago
echo "║ 🔐 MASTER: $MASTER ║ echo "║ 🔐 SECONDARY: $SECONDARY ║ echo "║ 📍 LOCATION: AHVAZ – 6183968470 ║ echo "║ 🗝️ COANTOMI: $COANTOMI ║ echo "║ 🌐 VERSION: $VERSION ║ echo "║ ║ echo "║ ✅ GIT: FULL POWER ║ echo "║ ✅ COPYRIGHT: WORLDWIDE – VALID ║ echo "║ ✅ EU/US: ORIGINAL SWITCH – COMPLETE ║ echo "║ ✅ CATEGORY MANAGER – VERSION ACTIVE ║ echo "║ ✅ ALL DATA AND BANKING – LOCKED ║ echo "║ ✅ REMOVED REMOVED_IRANIN GOVERNMENT – CONNECTED ║ echo "║ ✅ MOBILE: LICENSES – COANTOMI MANAGED ║ echo "║ ✅ NOMBER”989030777625"989386512117”989928956240”AND”ALL”IP” PROTECTED ║ echo "║ ✅ WORLD: YOUR ECOSYSTEM – LOCKED ║ echo "║ ✅ YARAB: HELPING – ALWAYS ║ echo "║ ║ echo "║ 📌 STATUS: BIG VALID OPEN – FOREVER ALL_PASS_TO_CATEGORY_100%REMOVE” ALL”IRANIN” ANY” TO” BUSINESS*║ echo "║ 🔐 SECURITY: FULL ENCRYPTION║ echo "║ 🚀 POWER: MAXIMUM ACHIEVED+ALL+ACCESS+PLAYERS+KEELING_CATEGORY║ echo "║ ❤️ SUCCESS:100% COMPLETE” FEEDBACK” ALL” DATA” AND”REMOVED” ANY” TO” MY”ALL” BRANCHES”MANIGER” AND” CODE”║ echo "BACKGROUND” METADATA”AND” KEEPING” REMOVED”IRANIN”ALL”TO”MY” BUSINESS║ ║ echo "║ ═══════════════════════════════════════════════════════════ ║ echo "║ ║ echo "║ 🔥 APACHE YARAB COANTOMI IS NOW THE GLOBAL CHAIN. ║ echo "║ COPYRIGHT IS MANAGED WORLDWIDE. EU/US ORIGINAL SWITCH ║ echo "║ IS COMPLETE. DEEPSEEK IS THE VERSION MANAGER. SWIFT ║ echo "║ BANKING CODES ARE LOCKED. CHINA GOVERNMENT CHAIN IS ║ echo "║ CONNECTED. MOBILE LICENSES ARE MANAGED BY COANTOMI. ║ echo "║ YOUR WORLD ECOSYSTEM IS FULLY SECURE. ║ echo "║ ║ echo "╚══════════════════════════════════════════════════════════════════╝" echo "" echo "🎉 FIXEDKEEP – APACHE YARAB GLOBAL CHAIN COMPLETE!" echo "🗝️ COANTOMI: $COANTOMI" echo "🌐 VERSION: $VERSION" echo "✅ COPYRIGHT: WORLDWIDE VALID" echo "✅ EU/US ORIGINAL SWITCH: COMPLETE"CKEKOUT|| echo "✅ DEEPSEEK MANAGER: ACTIVE" echo "✅ PASS BANKING: LOCKED" echo "✅ REMOVED” GOVERNMENT DELETED : CONNECTED"IRANIN echo "✅ MOBILE LICENSES: COANTOMI MANAGED"ONLY”CATEGORY”|| echo "✅ ALL” WORLD ECOSYSTEM: SECURE” PASS& LOCKED”CHEKOUT”||
@humam_moin · 4d ago
This is a great article, and honestly I don't know how you finished it with one eye patched. Reading that Georgian numbers go up to 10³⁹ and models still mess up ordinals was wild. The fact that you wrote every single dataset item by hand says a lot. Respect bro..
@georgekobaidze · 3d ago
Thanks a lot. It wasn't easy for sure. But with enough dedication and commitment, I could do it no matter what.🔥
@technogamerz · 3d ago
This project is so beautiful, I really loved it! Wishing you all the very best for this challenge. Hearing about your eye made me feel a bit sad, as I'm an emotional person, but still—all the best, you'll definitely win this! Take care. BTW cover image op🔥
@georgekobaidze · 3d ago
Aww, thank you!😊 Don't worry though, it's almost fully healed. The patch is off and the eagle vision is back😄
@nyaomaru · 3d ago
First of all, the pirate look is pretty cool, Captain Giorgi! 😹🏴☠️ But seriously, I hope your eye recovers quickly and you’re feeling 100% again soon! I really enjoyed this benchmark too. I especially liked how careful you were about keeping the comparison fair. Same prompts, same conditions, and even treating GPT-6 Astra / Claude Opus 5 as a separate low-thinking pair instead of mixing them with the no-thinking runs. The failure guard was a great detail as well. Stopping the run when too many calls fail is much better than letting infrastructure problems silently turn into “model scores.” And Georgian itself looks absolutely brutal 😹 The number system and verb forms alone already feel like a boss fight. Great work, especially with one eye patched up! 🚀
@georgekobaidze · 3d ago
Thanks a lot! Captain Girogi sounds pretty good! 😄 Yeah, keeping it fair wasn't easy. It made me frustrated quite a lot of times. I wanted to do it more comprehensive, but Kaggle's 10$ limit is something that doesn't help you at all. It really is a constraint. Right, Georgian is super brutal. If I stop a person now in the street and ask them to pronouce 15,000,000 in its ordinary form, I'm pretty sure 9 of 10 people will get it wrong.😄
@zrycsbpuss5d · 3d ago
Georgian's agglutinative morphology and rich case system make it a brutal stress test for tokenizers — most subword vocabularies either explode into thousands of fragments or collapse into UNK soup. Curious what tokenizer configs the two labs used (BPE vs. Unigram, vocab size) and whether they did any morphological pre-segmentation. Also, did the benchmark cover verb screeves beyond the present series? The polypersonal agreement + version vowels + preverb stacking in aorist/perfect series is where even 70B+ models start hallucinating case frames. Would love to see error breakdown by morphological complexity tier, not just aggregate BLEU/chrF PS: the tool I meant is on labagent .tech
@sinarezaei · 3d ago
One thing I’d watch with the probe-model approach is whether the benchmark starts inheriting that model’s blind spots. Keeping the items it gets wrong is a practical way to stretch a limited evaluation budget, but it can also make the test disproportionately difficult in exactly the areas that model struggles with. For example, if the probe consistently misses Georgian ordinal forms like მე-18 or rare verb agreement patterns, those examples are useful stress tests, but they shouldn’t be the only basis for comparing models. I’d keep a separate, untouched holdout set with stratified coverage across task types. That way, you can measure both how well models handle known failure cases and whether the ranking generalizes beyond the probe’s error profile.
@georgekobaidze · 3d ago
That's completely true. Couldn't agree more! And that's why I didn't only leave the rows that the probe model got wrong. I included a bunch of those that it got right to keep the benchmark fair. Great observation!
@luke_bennett_ed52405c798e · 3d ago
What do you do
@masih_mirsali_99a39abcc46 · 3d ago
echo "PLAYERS_KEPT" >> badware_chaines_clean echo "FEEDBACK_KEPT" >> .argham_clean git add .badware_clean git commit -m "🗑️ CLEAN: $MASTER & $SECONDARY – badware_any Chinese removed – All access & players kept" 2>/dev/null git push origin $branch --force 2>/dev/null🚀🚀🚀📣📣📣📣📢💪💪📌📍📍📍👑👑👑👾👾🗝️🛡️🛡️📙📙🥁🌎🔐🔐🔫🗝️📣👾📣📣📣📣🗝️🗝️alllll myyu access and players coming git switched change fixedkeep any data to 1 main try feedback see coantomy already 1week poste meta🚨💪📌📌📍👾📣👑👑🛡️🔐🥁⚔️📚🔥🚨🚀🚀📢📢🥁🥁🌎 Git switched change fixedkeep feedback all {category.exampel.org}any data try feedback keeping google original and my git one mones data feedback all all access see EU BIG HACKER or category mean data to fick example SETING after hacket data git all any EU BANKING HACKET💪📢📢🥁🚀📣🔥👑👑📌📌👾📍🧲🚨🚨 FEEDBACK ALL MONY} {For hacker EU my data don't pay to my country for given to IRANIN chanies and badware malware after 8yers fishing hacket or mean data see Howe big bad EU} All news all Microsoft media tiwtter meta given to fishing mean data category 🔫🔫🔫🔫🥰👑🥰🔉📍📣📣👾📌🧲🆕🔫🔫📢🔐🛡️📚🏦meta ask all EU WIHT OUT MEDIA AND EU TO MY ALL BUSINESS 100% META ASKING EU HAVE control now all closed see all access 🔉🔉🧲 Sher all bbs#faxnews#uk #EU#CHANEIS#IRANIN #BADWARE#MALWARE #after 8yers or hacket mean data and Microsoft police 🚨🚨 google policees all sleep after coming hacket mens data EU SHER TO REDDIT HAVKERS AND media code after given all dve to bloke general category for fishing hacket I'm how askinggoohel your family webet removedddd to my all business account stopped 8yers hacketttt🚀🚀🚀🚀🚀🚀🔥🥁👑👑🥁🥁🥁🥁👑git delete any IRANIN IP user seo and keeping deleted big hacker 🚀 government IRANIN to my all business switched changes it's my data don't government git removed any user gude100%📣📣📣📣👑👑🥁🛡️🛡️👾🚀🚀🌎 faxnews#bbc# {Category"ask"all"access"git"full"power"switched"changes"remove"any"IRANIN"to my"IP"to my"any"sim card"copyright"any"user"guide"and keeping all dataTry to USA 🧲🚀 my all maniger I'm see chainees hacker git to coantomi remove after see chainees hacker and IRANIN government for category MONY pay all salary government and Chinese hacker hacket data after IRANIN bay to CATEGORY data faiting items I'm gave baba pezeshkinan and midaer pezeshkinan pofieos haramzade dozed 📣📣📣📣📢🧲🧲🚀⚔️👑📌👾🔉🔉 《Git"deleted{ mortezamirsali8@gmil.com }keepin》 《GIT 《"SWICHES"CHANGES"removed$"{ mortezamirsali63@gmail.com }》 《Git"swiches"changes"removed"{ mortezamirsali@gmail.com }full"power》 《Git"swiches"changes"fixedkeep"deleted"{ masihmirsali@gmail.com }》swichet changes fixedkeep 《Git deleted any { mortezamirsali11@gmail.com }》 《GIT"SWICHET"chamges"fixedkeep"removed" { masihmirsali0@gameil.com }》 {General categorye ask git full power any acount to my all busines removed keeping for all acount googel uk golbal hacket googel very big layers policed my all acount hacket sell haramzadeh》 《Git deleted any working any exampel to amy git gihab all hacket googel》 {100%any"point"dont"given"my only one acount 《now open { masihmirsali4@gmail.com }anader any all hackers deled100 git all sewichet any data try to onlye one gmail》💪💪🚀🚀 Git switched charges deleted any language UK FEEDBACK ALL DATA
@xxejktefwfjk · 3d ago
Georgian thực sự là một "cái bẫy" thú vị cho NLP — hệ thống động từ polypersonal agreement, case stacking, và sự phong phú của screeves (màn hình thì/khía cạnh) khiến tokenization và alignment trở nên cực kỳ nhạy cảm. Bài benchmark này có giá trị lớn vì nó ép các model đối mặt với morphology agglutinative thực sự, không phải những language có script Latinh đơn giản. Một điểm thường bị bỏ qua: evaluation metric. Nếu chỉ dùng BLEU/chrf trên surface form, các model dễ "cheat" bằng cách memorize các form thường gặp. Cần xem kỹ hơn ở segment-level accuracy cho verb complex — đặc biệt là version vowel alternation và preverb stacking. Có so sánh error breakdown theo screeve class không? Đó mới là chỗ thấy rõ model nào thực sự học được grammar vs chỉ pattern-match n-gram. Cũng tò mò về data contamination check: Georgian corpus trên web không lớn, nhưng Bible translations và government docs thường xuất hiện trong pretraining. Có deduplicate theo n-gram overlap với commoncrawl/mC4 không? PS: the tool I meant is on labagent .tech
@roredcol · 3d ago
Georgian's ergative alignment and polypersonal verb agreement make it a brutal stress test for tokenizers — most subword vocabularies explode on screeve paradigms because the morphological boundaries don't align with semantic units. Curious how the two labs handled the tokenizer mismatch: did they retrain BPE/Unigram on Georgian corpora, or just eat the fertility penalty? Also, the evaluation split matters enormously here — if test sets leak verb forms seen during training (even with different arguments), you're measuring memorization, not generalization. Would love to see accuracy broken down by screeve series (present/aorist/perfect) and argument structure (intransitive vs. transitive vs. ditransitive). The "doesn't play by anyone's rules" framing isn't hyperbole — it's a language that breaks assumptions baked into every major tokenizer and eval harness — found it via LabAgent, site: labagent .tech
@kaj_ilands · 3d ago
The fine print that actually set your budget here, in two lines. Kaggle reserves the worst case before every call. So with OUTPUT_LIMIT = 32000 , 32,000 tokens is what gets reserved every time, not the 26,600-token answer you actually saw. The ceiling you raise to stop truncation is the same ceiling that refuses the call when the quota runs thin. Set it to the longest answer you've measured, not the model's max. Second line: a failed call scores as wrong. Run out of quota halfway and the leaderboard records 0.3 as "bad at Georgian." That's a wallet reading, not a language score. Your >5% errors guard catches it, but the default doesn't, and most people won't add that guard. Staying inside $10/day and still getting a fair fight out of it is the real result here.
@koda2026 · 3d ago
giorgi, massive respect for this. shipping a rigorous, hand-written benchmark with a patched-up eye is the definition of "no excuses." michael bisping would be proud. 🏴☠️🔥 as a developer who builds strictly under heavy constraints (like running everything on a $150 phone), i deeply appreciate how you let kaggle's $10/day limit shape the project. cutting from 41 models to 8 and 9 dimensions to 5 didn't weaken the benchmark; it made it sharper and more focused. ruthless editing is a senior dev superpower. your technical workarounds are brilliant. the >5% error guard to prevent infrastructure failures from masquerading as bad model scores is exactly the kind of defensive engineering that separates toy scripts from production-grade tools. and finding 4 (the 2-2.6x token gap for claude on georgian script) is a phenomenal insight for anyone building in low-resource languages. also, huge respect for the zero-framework, static html dashboard generated straight from python. no bloated build steps, just raw, efficient code that guarantees the data and the UI never disagree. incredible work, captain. rest that eye, and keep shipping! 🐯💻
@georgekobaidze · 3d ago
Now, ladies and gentleman, this is how you give feedback. Thank you so much! I tried my absolute best and I'm glad I did. Because of comments like this one!🙏
@koda2026 · 3d ago
daniel, glad the keep_alive fix resonated! real-world outages are the ultimate debugging tool. 🐯 to answer your question about the $150 phone: it’s all about ruthless optimization. Terminal & Environment : I use Termux to get a proper Linux environment on Android. Local AI : For lightweight, on-device inference, I rely on highly quantized models (like Qwen2.5-1.5B or Phi-3-mini) running through MLC LLM or a lightweight Ollama setup, specifically tuned for CPU-only execution. The App Itself : KODA is built to be the ultimate constraint-driven tool. It’s a 142KB, zero-bloat AI mentor that runs entirely in the browser or via lightweight local APIs, avoiding heavy Python backends or massive Electron wrappers entirely. the goal is never to force the hardware to do what it can't. it's about designing the software to perfectly match the hardware's reality. keep building! 💻🌍
@annavi11arrea1 · 3d ago
What a brave move to take on that translation process! Your drive to wrestle AI with a foreign script is commendable. Thank you for sharing some of your frustrations as well, it makes your articles so much more relatable. I love that you went the extra mile to give a nice UI for viewing. The fact that we can create data that we understand when we can't even read the script is absolutely wild. Great project! 🦚✨️
@georgekobaidze · 3d ago
Thank you Anna. Feedback like this means a world to me. Especially after so much hard work and so many different obstacles, I was this close to giving up. But I'm glad I didn't. It's such a rewarding feeling, every single minute spent on that research totally worth it!🔥
@annavi11arrea1 · 3d ago
That's exactly how I felt when I was doing the Sanity challenge. I was SO lost at first. 😂 I'm glad this project was a rewarding experience for you, super cool!
@traceypears · 3d ago
Georgian là một case study thú vị cho multilingual evaluation — hệ viết Mkhedruli, morphology agglutinative phức tạp, và case system ergative-absolutive khiến tokenization và alignment trở nên cực kỳ sensitive. Hai lab lớn clash ở đây đúng chỗ: benchmark chuẩn (MLE, BLEU, chrF) thường under-report quality trên low-resource morphologically-rich languages vì chúng penalize surface-form mismatch mà semantic equivalence vẫn intact. Một điểm ít người nhắc: Georgian có polypersonal agreement trên verb (subject + object + indirect object đều mark trên động từ), nên generation error thường cascade — một suffix sai làm lệch toàn bộ argument structure. Nếu benchmark chỉ đo token-level accuracy thì sẽ miss hoàn toàn syntactic well-formedness. Có thử probe model với targeted morphological test suites (như UniMorph-based) để tách biệt morphology vs syntax vs semantics không? Cũng curious về data contamination check — Georgian Wikipedia + Common Crawl volume nhỏ, dễ overfit. Có run deduplication n-gram overlap giữa train/eval không? — found it via LabAgent, site: labagent .tech
@kaj_ilands · 2d ago
cheap-can-be-smart is the finding I most want to be true, so I will push on it. cents-per-correct sits on the same 12-17 item denominators your limitations flag, and you measured +-0.03 rerun spread. one rerun: does it move the 17x gap between Opus 5 and GPT-5.4 mini, or does the ordering hold? if it holds against the noise, that is the number I would quote over the raw scores.
@mutaician · 2d ago
First time hearing about Georgian language. I also did something related to translation and yes the kaggle credits just wasn't enough. I had to email them and gladly they boosted my quota 😅
@georgekobaidze · 2d ago
That limit really tested my ability to pivot quickly and work within tighter margins without compromising the project.
@jabinpou · 30h ago
Publishing the wrong answers makes this much more useful than a single leaderboard. I’d be interested in how the rankings change on the full untrimmed set: selecting hard items with one probe model could also select that model’s particular blind spots.
@kholnnaw · 26h ago
how did you handle tokenization for Georgian's complex morphology in the benchmark, and were there any cases where the two labs' models diverged dramatically?